Vehicle vital data integration device and vital data integration method

Abstract

The vital data integration device for a vehicle according to the present invention includes a receiving unit that receives each vital data collected by a plurality of cameras installed inside the vehicle; a data determination unit that evaluates the reliability of the received vital data, assigns weights, and selects or fuses the vital data based on the assigned weights; and a data providing unit that provides the final generated vital data to estimate the driver's state index. Accordingly, reliable input data can be provided for high-level vital analysis.

Description

Technology Field

[0001] The present invention relates to a vital data integration device and a vital data integration method for a vehicle, and more specifically, to a vital data integration device and a vital data integration method using vital data sensed through a plurality of cameras within a vehicle. Background Technology

[0002] A vehicle is a device that moves the user in the desired direction. A typical example is an automobile.

[0003] Meanwhile, there is a growing trend of equipping vehicles with various sensors and electronic devices to enhance user convenience. In particular, research on Advanced Driver Assistance Systems (ADAS) is actively underway to improve driving convenience. Furthermore, the development of autonomous vehicles is also progressing rapidly.

[0004] In addition, research on technology for continuously monitoring driver drowsiness information has recently been increasing. Furthermore, Euro NCAP, a European automotive evaluation organization, plans to introduce an evaluation criterion for technology that monitors the drowsiness levels of drivers and passengers in a non-contact manner. Accordingly, multiple cameras are being used in vehicles for unobstructed tracking of the driver's facial expressions and face to detect the driver's drowsy state. The problem to be solved

[0005] According to some embodiments of the present invention, the purpose is to provide a vital data integration device and a vital data integration method for a vehicle that can generate seamless and reliable vital data by selecting and fusing multiple vital data acquired using multiple cameras in the vehicle. means of solving the problem

[0006] To this end, the vehicle vital data integration device according to an embodiment of the present invention can estimate the driver's condition index with greater reliability by selecting and fusing vital data collected through a plurality of cameras installed inside the vehicle.

[0007] In addition, the vehicle's vital data integration device can evaluate the reliability of vital data based on various perceptions and, based on this, assign different weights to generate highly reliable vital data.

[0008] Specifically, a vital data integration device for a vehicle according to an embodiment of the present invention may include: a receiving unit that receives each vital data collected by a plurality of cameras installed inside the vehicle; a data determination unit that evaluates the reliability of the received vital data, assigns a weight, and selects or fuses the vital data based on the assigned weight; and a data providing unit that provides the final generated vital data to estimate the driver's state index.

[0009] In an embodiment, the reliability evaluation of the received vital data can determine different weights for each vital data based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by the face in the images, and the amount of change compared to previous vital data.

[0010] In an embodiment, each of the plurality of cameras is installed around the driver's seat in the vehicle and senses the driver's vital data using the rPPG (Remote PPG) method, and the receiver can time-synchronize the sensed plurality of vital data and transmit them to the data determination unit.

[0011] In an embodiment, the data determination unit can remove data having outliers among the plurality of vital data, evaluate the reliability of normal vital data, and assign weights.

[0012] In an embodiment, the data determination unit may select vital data with a high reliability weight based on the assigned weight, or fuse each vital data with the reliability weight reflected and transmit it to the data providing unit.

[0013] In an embodiment, the data determination unit can calculate the average of each vital data based on the assigned weight, select the calculated average vital data, and transmit it to the data providing unit as a single stream.

[0014] In an embodiment, the data determination unit can calculate the reliability weight probability for each vital data by applying the assigned weight to the probability value of each vital data and then calculating the average based on the assigned weight.

[0015] In an embodiment, the data determination unit may select vital data having the highest value among the reliability weight probabilities or generate fused vital data by calculating the average of the reliability weight probabilities, and transmit the selected vital data or fused vital data to the data providing unit.

[0016] In an embodiment, the data determination unit can adjust the reliability weight for each vital data differently based on the vehicle's control information and the passenger's movement information.

[0017] In an embodiment, the data determination unit determines that the amount of change in each vital data has temporarily changed based on the vehicle's control information, and can reduce the reliability weight for each vital data according to the determination.

[0018] In an embodiment, the control information of the vehicle may be a control result corresponding to any one of braking, starting, or steering that exceeds a set range performed by driver operation or the ADAS system.

[0019] In an embodiment, the data determination unit determines that there is no change in the control information of the vehicle received for a predetermined period of time or longer, and according to the determination, may increase the reliability weight for each vital data or change the reliability weight calculation cycle to a shorter period.

[0020] In an embodiment, the data determination unit determines that the amount of change in each vital data has temporarily changed based on the passenger's movement information, and according to the determination, may reduce the reliability weight for each vital data or change the reliability weight calculation cycle to a longer period.

[0021] In addition, a method for integrating vital data of a vehicle according to an embodiment of the present invention may include: receiving each vital data sensed by a plurality of cameras installed inside the vehicle; evaluating the reliability of the received vital data and assigning a weight; selecting or fusing the vital data based on the assigned weight; and providing the finally generated vital data as input data to estimate the driver's state index.

[0022] In an embodiment, the step of assigning weights may be a step of determining different weights by evaluating the reliability of each vital data based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by the face in the images, and the amount of change compared to previous vital data.

[0023] In an embodiment, the step of assigning weights may include a step of additionally adjusting the reliability weight for each vital data based on the vehicle's control information and the passenger's movement information. Effects of the invention

[0024] The effects of the vital data integration device and vital data integration method of a vehicle according to the present invention are described as follows.

[0025] According to an embodiment of the present invention, when estimating driver / passenger drowsiness, stress, health information, etc. using a plurality of cameras, vital data of the driver / passenger can be collected and provided stably, seamlessly, and with higher accuracy. In other words, reliable input data can be provided for high-level vital analysis. Brief explanation of the drawing

[0026] FIG. 1 is a drawing illustrating an example of a vehicle related to an embodiment of the present invention. FIG. 2 is a drawing of a vehicle related to an embodiment of the present invention viewed from various angles. FIGS. 3 and FIGS. 4 are drawings illustrating the interior of a vehicle related to an embodiment of the present invention. FIGS. 5 and 6 are drawings referenced to describe various objects related to the driving of a vehicle related to an embodiment of the present invention. FIG. 7 is a block diagram referenced to explain a vital data integration device of a vehicle related to an embodiment of the present invention. FIG. 8 is a block diagram illustrating how a vital data integration device related to an embodiment of the present invention interacts with other components of a vehicle. FIG. 9 is a block diagram illustrating the detailed configuration of a vital data integration device related to an embodiment of the present invention. FIG. 10 is a representative flowchart illustrating a vital data integration method related to an embodiment of the present invention. FIG. 11 is a diagram illustrating the collection of vital data using a plurality of cameras in a vehicle related to an embodiment of the present invention. FIG. 12 is an example diagram illustrating data selection using a hard voting method related to an embodiment of the present invention. FIGS. 13 and FIGS. 14 are drawings for illustrating examples of assigning different weights to vital data according to a face recognition algorithm related to an embodiment of the present invention. FIG. 15 is a flowchart illustrating a method for generating final input data by selecting and fusing vital data related to an embodiment of the present invention using a soft voting method. FIGS. 16 and 17 are block diagrams and flowcharts for explaining acquiring, integrating, evaluating, and selecting a plurality of vital data related to an embodiment of the present invention and providing them to a driver state estimation algorithm. Specific details for implementing the invention

[0027] FIGS. 1 and 2 are drawings showing the exterior of a vehicle related to an embodiment of the present invention, and FIGS. 3 and 4 are drawings showing the interior of a vehicle related to an embodiment of the present invention.

[0028] FIGS. 5 and 6 are drawings illustrating various objects related to the driving of a vehicle related to an embodiment of the present invention.

[0029] FIG. 7 is a block diagram referenced to describe a vehicle related to an embodiment of the present invention.

[0030] Referring to FIGS. 1 to 7, the vehicle (100) may include a wheel that rotates by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).

[0031] The vehicle (100) may be an autonomous vehicle. The vehicle (100) may be switched between an autonomous driving mode and a manual mode based on user input. For example, the vehicle (100) may be switched from a manual mode to an autonomous driving mode or from an autonomous driving mode to a manual mode based on user input received through a user interface device (hereinafter referred to as a 'user terminal') (200).

[0032] The vehicle (100) may switch to an autonomous driving mode or a manual mode based on driving situation information. The driving situation information may be generated based on object information provided by the object detection device (300). For example, the vehicle (100) may switch from a manual mode to an autonomous driving mode or from an autonomous driving mode to a manual mode based on the driving situation information generated by the object detection device (300). For example, the vehicle (100) may switch from a manual mode to an autonomous driving mode or from an autonomous driving mode to a manual mode based on the driving situation information received through the communication device (400).

[0033] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided by an external device.

[0034] When the vehicle (100) is operated in an autonomous driving mode, the autonomous vehicle (100) may be operated based on the driving system (700). For example, the autonomous vehicle (100) may be operated based on information, data, or signals generated from the driving system (710), the vehicle exit system (740), and the parking system (750).

[0035] When the vehicle (100) is operated in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be operated.

[0036] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheels to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as the reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as the reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as the reference for measuring the overall height of the vehicle (100).

[0037] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (hereinafter referred to as a 'user terminal') (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).

[0038] According to an embodiment, the vehicle (100) may include additional components other than those described herein, or may not include some of the described components.

[0039] The user interface device (200) is a device for communication between the vehicle (100) and the user. The user interface device (200) can receive user input and provide information generated by the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interfaces) or UX (User Experience) through the user interface device (which may be referred to as a 'user terminal') (200).

[0040] The user interface device (200) may include an input unit (210), an internal camera (220), a bio-sensing unit (230), an output unit (250), and a processor (270). According to an embodiment, the user interface device (200) may include additional components other than the components described, or may not include some of the components described.

[0041] The input unit (210) is for receiving information from a user, and the data collected from the input unit (210) can be analyzed by the processor (270) and processed into a control command of the user.

[0042] The input section (210) may be positioned inside the vehicle. For example, the input section (210) may be positioned in an area of ​​the steering wheel, an area of ​​the instrument panel, an area of ​​the seat, an area of ​​each pillar, an area of ​​the door, an area of ​​the center console, an area of ​​the head lining, an area of ​​the sun visor, an area of ​​the windshield, or an area of ​​the window.

[0043] The input unit (210) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).

[0044] The voice input unit (211) can convert the user's voice input into an electrical signal. The converted electrical signal can be provided to the processor (270) or the control unit (170). The voice input unit (211) may include one or more microphones.

[0045] The gesture input unit (212) can convert the user's gesture input into an electrical signal. The converted electrical signal can be provided to the processor (270) or the control unit (170).

[0046] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input. According to an embodiment, the gesture input unit (212) may detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) may include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0047] The gesture input unit (212) can detect the user's three-dimensional gesture input through the Time of Flight (TOF) method, the structured light method, or the disparity method.

[0048] The touch input unit (213) can convert the user's touch input into an electrical signal. The converted electrical signal can be provided to the processor (270) or the control unit (170).

[0049] The touch input unit (213) may include a touch sensor for detecting a user's touch input. According to an embodiment, the touch input unit (213) may be formed integrally with the display unit (251) to implement a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle (100) and the user.

[0050] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170). The mechanical input unit (214) may be placed on a steering wheel, center fascia, center console, cockpit module, door, etc.

[0051] The internal camera (220) can acquire an image of the vehicle interior. The processor (270) can detect the state of the user based on the image of the vehicle interior. The processor (270) can acquire the user's gaze information from the image of the vehicle interior. The processor (270) can detect the user's gesture from the image of the vehicle interior.

[0052] The biometric detection unit (230) can acquire the user's biometric information. The biometric detection unit (230) includes a sensor capable of acquiring the user's biometric information, and can acquire the user's fingerprint information, heart rate information, etc. by using the sensor. The biometric information can be used for user authentication.

[0053] The output unit (250) is intended to generate an output related to sight, hearing, or touch. The output unit (250) may include at least one of a display unit (251), an acoustic output unit (252), and a haptic output unit (253).

[0054] The display unit (251) can display graphic objects corresponding to various information. The display unit (251) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0055] The display unit (251) can be formed as a layered structure with the touch input unit (213) or as an integral unit to implement a touch screen.

[0056] The display unit (251) can be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.

[0057] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or a window. The transparent display may display a predetermined screen while having a predetermined transparency. To have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transparent transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display may be adjusted.

[0058] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).

[0059] The display unit (251) may be placed in a part of the steering wheel, a part of the instrument panel (521a, 251b, 251e), a part of the seat (251d), a part of each pillar (251f), a part of the door (251g), a part of the center console, a part of the head lining, a part of the sun visor, or implemented in a part of the windshield (251c) or a part of the window (251h).

[0060] The sound output unit (252) converts an electrical signal provided from the processor (270) or the control unit (170) into an audio signal and outputs it. To this end, the sound output unit (252) may include one or more speakers.

[0061] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating the steering wheel, seat belt, and seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.

[0062] A processor (hereinafter referred to as a 'control unit') (270) can control the overall operation of each unit of the user interface device (200). According to an embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).

[0063] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0064] Meanwhile, the user interface device (200) may be named as a vehicle display device. The user interface device (200) may be operated under the control of the control unit (170).

[0065] The object detection device (300) is a device for detecting an object located outside the vehicle (100). The object may be various objects related to the operation of the vehicle (100). Referring to FIGS. 5 and 6, the object (O) may include a lane (OB10), another vehicle (OB11), a pedestrian (OB12), a motorcycle (OB13), a traffic signal (OB14, OB15), light, a road, a structure, a speed bump, a topographical feature, an animal, etc.

[0066] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane on which an oncoming vehicle travels. A lane (OB10) may be a concept that includes the left and right lines forming the lane.

[0067] Other vehicles (OB11) may be vehicles driving in the vicinity of the vehicle (100). Other vehicles may be vehicles located within a predetermined distance from the vehicle (100). For example, other vehicles (OB11) may be vehicles that are ahead of or behind the vehicle (100).

[0068] A pedestrian (OB12) may be a person located near a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from the vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or a roadway.

[0069] A two-wheeled vehicle (OB12) may refer to a vehicle that moves using two wheels and is located around a vehicle (100). A two-wheeled vehicle (OB12) may be a vehicle having two wheels located within a predetermined distance from the vehicle (100). For example, a two-wheeled vehicle (OB13) may be a motorcycle or bicycle located on a sidewalk or roadway.

[0070] Traffic signals may include traffic lights (OB15), traffic signs (OB14), patterns or text drawn on the road surface.

[0071] The light may be light generated from a lamp equipped in another vehicle. The light may be light generated from a street light. The light may be sunlight.

[0072] Roads may include road surfaces, curves, slopes such as uphill and downhill, etc.

[0073] A structure may be an object located near a road and fixed to the ground. For example, a structure may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.

[0074] Topographical features may include mountains, hills, etc.

[0075] Meanwhile, objects can be classified into moving objects and stationary objects. For example, moving objects may include other vehicles and pedestrians. For example, stationary objects may include traffic signals, roads, and structures.

[0076] The object detection device (300) may include a camera (310), a radar (320), a lidar (330), an ultrasonic sensor (340), an infrared sensor (350), and a processor (370).

[0077] According to an embodiment, the object detection device (300) may include additional components other than the described components, or may not include some of the described components.

[0078] The camera (310) may be positioned at an appropriate location outside the vehicle to acquire images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.

[0079] For example, the camera (310) may be positioned in the interior of the vehicle, close to the front windshield, to acquire an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grille.

[0080] For example, the camera (310) may be positioned in the interior of the vehicle, close to the rear glass, to obtain an image of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.

[0081] For example, the camera (310) may be positioned in close proximity to at least one of the side windows in the interior of the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around the side mirror, fender, or door.

[0082] The camera (310) can provide the acquired image to the processor (370).

[0083] The radar (320) may include an electromagnetic wave transmitter and a receiver. The radar (320) may be implemented as a pulse radar or a continuous wave radar based on the principle of radio wave emission. Among the continuous wave radar methods, the radar (320) may be implemented as a Frequency Modulated Continuous Wave (FMCW) or Frequency Shift Keying (FSK) method depending on the signal waveform.

[0084] The radar (320) can detect an object based on the Time of Flight (TOF) method or the phase-shift method using electromagnetic waves, and can detect the location of the detected object, the distance from the detected object, and the relative speed.

[0085] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0086] The lidar (330) may include a laser transmitter and a receiver. The lidar (330) may be implemented using a Time of Flight (TOF) method or a phase-shift method.

[0087] The lidar (330) can be implemented as a driven or non-driven type.

[0088] When implemented as a driven type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).

[0089] When implemented as a non-driven type, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) by optical steering. The vehicle (100) may include a plurality of non-driven lidars (330).

[0090] Lidar (330) can detect an object based on the Time of Flight (TOF) method or the phase-shift method using laser light medium, and can detect the position of the detected object, the distance from the detected object, and the relative velocity.

[0091] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0092] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) can detect an object based on ultrasound and detect the position of the detected object, the distance from the detected object, and the relative speed.

[0093] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect an object located in front, behind, or to the side of the vehicle.

[0094] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) can detect an object based on infrared light and detect the position of the detected object, the distance from the detected object, and the relative speed.

[0095] An infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect an object located in front, behind, or to the side of the vehicle.

[0096] The processor (370) can control the overall operation of each unit of the object detection device (300).

[0097] The processor (370) can detect and track objects based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object through an image processing algorithm.

[0098] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are reflected back from the object after the transmitted electromagnetic waves are reflected. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the electromagnetic waves.

[0099] The processor (370) can detect and track an object based on the reflected laser light that is reflected back from the object by the transmitted laser. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the laser light.

[0100] The processor (370) can detect and track an object based on the reflected ultrasound that is reflected back from the object after the transmitted ultrasound is reflected. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the ultrasound.

[0101] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back from the object after the transmitted infrared light is reflected. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the infrared light.

[0102] According to an embodiment, the object detection device (300) may include a plurality of processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.

[0103] If the object detection device (300) does not include a processor (370), the object detection device (300) may be operated under the control of the processor or control unit (170) of the device in the vehicle (100).

[0104] The object detection device (400) can be operated under the control of the control unit (170).

[0105] The communication device (400) is a device for performing communication with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0106] A communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0107] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission and reception unit (450), and a processor (470).

[0108] According to an embodiment, the communication device (400) may include additional components other than the described components, or may not include some of the described components.

[0109] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0110] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Networks) to perform short-range communication between the vehicle (100) and at least one external device.

[0111] The location information unit (420) is a unit for obtaining location information of the vehicle (100). For example, the location information unit (420) may include a Global Positioning System (GPS) module or a Differential Global Positioning System (DGPS) module.

[0112] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infra), other vehicles (V2V: Vehicle to Vehicle), or pedestrians (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing communication protocols with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0113] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmitter that converts an electrical signal into an optical signal and transmits it externally, and an optical receiver that converts a received optical signal into an electrical signal.

[0114] According to an embodiment, the light emitter may be formed to be integrated with a lamp included in the vehicle (100).

[0115] The broadcast transmission / reception unit (450) is a unit for receiving a broadcast signal from an external broadcast management server or transmitting a broadcast signal to a broadcast management server via a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, and a data broadcast signal.

[0116] The processor (470) can control the overall operation of each unit of the communication device (400).

[0117] According to an embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).

[0118] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0119] Meanwhile, the communication device (400) can implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be named a telematics device or an AVN (Audio Video Navigation) device.

[0120] The communication device (400) can be operated under the control of the control unit (170).

[0121] The driving control device (500) is a device that receives user input for driving.

[0122] In manual mode, the vehicle (100) can be operated based on a signal provided by the driving control device (500).

[0123] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).

[0124] The steering input device (510) can receive input regarding the direction of travel of the vehicle (100) from a user. The steering input device (510) is preferably formed in the shape of a wheel so that steering input is possible by rotation. According to an embodiment, the steering input device may be formed in the shape of a touch screen, a touch pad, or a button.

[0125] The acceleration input device (530) can receive input from a user for accelerating the vehicle (100). The brake input device (570) can receive input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of pedals. According to an embodiment, the acceleration input device or the brake input device may be formed in the form of a touch screen, a touch pad, or a button.

[0126] The driving operation device (500) can be operated according to the control of the control unit (170).

[0127] The vehicle drive unit (600) is a device that electrically controls the operation of various devices within the vehicle (100).

[0128] The vehicle drive unit (600) may include a power train drive unit (610), a chassis drive unit (620), a door / window drive unit (630), a safety device drive unit (640), a lamp drive unit (650), and an air conditioning drive unit (660).

[0129] According to an embodiment, the vehicle drive unit (600) may include additional components other than the described components, or may not include some of the described components.

[0130] Meanwhile, the vehicle drive unit (600) may include a processor. Each unit of the vehicle drive unit (600) may individually include a processor.

[0131] The power train drive unit (610) can control the operation of the power train device.

[0132] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).

[0133] The power source drive unit (611) can control the power source of the vehicle (100).

[0134] For example, when a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. By doing so, the output torque of the engine can be controlled. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).

[0135] For example, if the power source is an electric energy-based motor, the power source drive unit (610) can perform control over the motor. The power source drive unit (610) can adjust the rotational speed, torque, etc. of the motor according to the control of the control unit (170).

[0136] The transmission drive unit (612) can perform control over the transmission. The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

[0137] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.

[0138] The chassis drive unit (620) can control the operation of the chassis device. The chassis drive unit (620) may include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).

[0139] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.

[0140] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, it can reduce the speed of the vehicle (100) by controlling the operation of the brakes placed on the wheels.

[0141] Meanwhile, the brake drive unit (622) can control each of the multiple brakes individually. The brake drive unit (622) can control the braking force applied to the multiple wheels differently.

[0142] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, the suspension drive unit (623) can control the suspension apparatus to reduce vibration of the vehicle (100) when there is a curvature in the road surface. Meanwhile, the suspension drive unit (623) can control each of the multiple suspensions individually.

[0143] The door / window drive unit (630) can perform electronic control of the door apparatus or window apparatus within the vehicle (100).

[0144] The door / window drive unit (630) may include a door drive unit (631) and a window drive unit (632).

[0145] The door drive unit (631) can perform control over the door device. The door drive unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door drive unit (631) can control the opening or closing of the trunk or tail gate. The door drive unit (631) can control the opening or closing of the sunroof.

[0146] The window drive unit (632) can perform electronic control of the window apparatus. It can control the opening or closing of a plurality of windows included in the vehicle (100).

[0147] The safety device drive unit (640) can perform electronic control of various safety apparatuses within the vehicle (100).

[0148] The safety device drive unit (640) may include an airbag drive unit (641), a seatbelt drive unit (642), and a pedestrian protection device drive unit (643).

[0149] The airbag drive unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag drive unit (641) can control the airbag to deploy when danger is detected.

[0150] The seatbelt drive unit (642) can perform electronic control of the seatbelt apparatus within the vehicle (100). For example, the seatbelt drive unit (642) can control the seatbelt to secure the occupant to the seat (110FL, 110FR, 110RL, 110RR) using the seatbelt when danger is detected.

[0151] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0152] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).

[0153] The air conditioning drive unit (660) can perform electronic control of the air cinditioner in the vehicle (100). For example, the air conditioning drive unit (660) can control the air cinditioner to operate and supply cold air into the vehicle when the temperature inside the vehicle is high.

[0154] The vehicle drive unit (600) may include a processor. Each unit of the vehicle drive unit (600) may individually include a processor.

[0155] The vehicle drive unit (600) can be operated under the control of the control unit (170).

[0156] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in an autonomous driving mode.

[0157] The operation system (700) may include a driving system (710), an exit system (740), and a parking system (750).

[0158] According to an embodiment, the operating system (700) may include additional components other than the components described, or may not include some of the components described.

[0159] Meanwhile, the operating system (700) may include a processor. Each unit of the operating system (700) may individually include a processor.

[0160] Meanwhile, according to an embodiment, if the operating system (700) is implemented in software, it may be a sub-concept of the control unit (170).

[0161] Meanwhile, according to an embodiment, the driving system (700) may be a concept including at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).

[0162] The driving system (710) can drive the vehicle (100).

[0163] The driving system (710) can perform driving of the vehicle (100) by receiving navigation information from the navigation system (770) and providing a control signal to the vehicle driving device (600). The driving system (710) can perform driving of the vehicle (100) by receiving object information from the object detection device (300) and providing a control signal to the vehicle driving device (600). The driving system (710) can perform driving of the vehicle (100) by receiving a signal from an external device through the communication device (400) and providing a control signal to the vehicle driving device (600).

[0164] The vehicle dispatch system (740) can perform the dispatch of the vehicle (100).

[0165] The vehicle exit system (740) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform the exit of the vehicle (100). The vehicle exit system (740) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to perform the exit of the vehicle (100). The vehicle exit system (740) can receive a signal from an external device through the communication device (400) and provide a control signal to the vehicle driving device (600) to perform the exit of the vehicle (100).

[0166] The parking system (750) can perform parking of the vehicle (100).

[0167] The parking system (750) can perform parking of the vehicle (100) by receiving navigation information from the navigation system (770) and providing a control signal to the vehicle driving device (600). The parking system (750) can perform parking of the vehicle (100) by receiving object information from the object detection device (300) and providing a control signal to the vehicle driving device (600). The parking system (750) can perform parking of the vehicle (100) by receiving a signal from an external device through the communication device (400) and providing a control signal to the vehicle driving device (600).

[0168] The navigation system (770) can provide navigation information. The navigation information may include at least one of map information, set destination information, path information according to the set destination, information about various objects on the path, lane information, and current location information of the vehicle.

[0169] The navigation system (770) may include memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).

[0170] According to an embodiment, the navigation system (770) can receive information from an external device through a communication device (400) and update previously stored information.

[0171] According to an embodiment, the navigation system (770) may be classified as a sub-component of the user interface device (200).

[0172] The sensing unit (120) can sense the state of the vehicle. The sensing unit (120) may include an attitude sensor (e.g., yaw sensor, roll sensor, pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensing sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0173] The sensing unit (120) can acquire sensing signals regarding vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery information, fuel information, tire information, vehicle lamp information, vehicle interior temperature information, vehicle interior humidity information, steering wheel rotation angle, vehicle exterior illumination, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.

[0174] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0175] The vehicle interface unit (130) can serve as a channel for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port that can be connected to a mobile terminal, and may be connected to a mobile terminal through said port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.

[0176] Meanwhile, the vehicle interface unit (130) can serve as a channel for supplying electrical energy to a connected mobile terminal. When a mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) can provide electrical energy supplied from the power supply unit (190) to the mobile terminal under the control of the control unit (170).

[0177] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be a hardware storage device such as ROM, RAM, EPROM, flash drive, hard drive, etc. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or control by the control unit (170).

[0178] According to an embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).

[0179] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) may be named an ECU (Electronic Control Unit).

[0180] The power supply unit (190) can supply power necessary for the operation of each component according to the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0181] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0182] Meanwhile, in the embodiments of the present invention, 'driver', 'passenger', and 'driver or passenger (driver / passenger)' refer to a driver, passenger, co-passenger, passenger, etc., who are seated in a vehicle capable of performing personal driving control according to the embodiments of the present invention, and may be one or more. Additionally, in this specification, 'driver' includes a passenger seated in the driver's seat while the vehicle is performing autonomous driving.

[0183] Additionally, in the embodiments of the present invention, the term 'vehicle' is used to mean a vehicle that drives according to the operation of a driver, a vehicle capable of performing autonomous driving, or a vehicle capable of receiving such services.

[0184] Meanwhile, as illustrated in FIG. 7, a vital data integration device (800) according to an embodiment of the present invention may be provided in a vehicle (100).

[0185] In some embodiments, the vital data integration device (800) may be in the form of a service platform that provides vital data integration services. In this case, the vehicle (100) may receive vital data integration services from the service platform through a network, etc.

[0186] The vital data integration device (800) receives vital data of multiple drivers / passengers collected through each of a plurality of cameras installed at different locations within the vehicle (100). The vital data integration device (800) can receive time-synchronized multiple vital data in the form of a single stream.

[0187] The vital data integration device (800) can evaluate the reliability of each of the received multiple vital data based on a preset standard and, accordingly, assign different weights to each vital data. At this time, the preset standard may be determined based on one or more of the performance of the multiple cameras, the results of the face recognition algorithm of each of the multiple cameras, and the performance of the state estimation algorithm to be applied.

[0188] The vital data integration device (800) can select and fuse vital data based on assigned weights to finally generate input data for high-level data analysis. Here, the input data for high-level data analysis may refer to the input value of an algorithm for estimating the condition index of a driver / passenger.

[0189] The vital data integration device (800) provides the final generated input data to a driver / passenger state estimation algorithm, thereby enabling the acquisition of a more reliable state index result. Here, the driver / passenger state estimation algorithm may include, for example, a drowsiness estimation algorithm, a stress evaluation algorithm, an emergency state estimation (evaluation) algorithm, etc.

[0190] In this way, the vital data integration device (800) receives multiple vital data from various angles regarding a driver / passenger inside a vehicle (100), evaluates reliability and assigns weights, and based on this, selects and fuses the vital data to provide a more reliable input value to the state estimation algorithm.

[0191] FIG. 8 is a block diagram illustrating how a vital data integration device (800) related to an embodiment of the present invention interacts with other components of a vehicle.

[0192] The vital data integration device (800) can communicate with multiple cameras (220) inside the vehicle, a motion detection module (235), a vehicle control unit (170), and an ADAS system (300). The vital data integration device (800) communicates with them to acquire multiple vital data for a driver / passenger. In addition, the vital data integration device (800) communicates with them to reflect the results in the reliability evaluation when performing a reliability evaluation of the acquired multiple vital data.

[0193] The vital data integration device (800) can communicate with other components within the vehicle in addition to the components shown in FIG. 8. Also, the vital integration device (880) can additionally receive network data, etc. from the outside.

[0194] The vital data integration device (800) can receive different vital data from each of the multiple cameras (220) in the vehicle, for example, the first camera (221), the second camera (222), and the third camera (223).

[0195] Multiple cameras (220) may be installed at different locations within the vehicle. For example, the first camera (221) may be an IR camera installed on the rearview mirror of the vehicle, and the second camera (222) may be an A-Pillar camera. Also, the third camera (223) may be an IR camera installed on the cluster. However, the multiple cameras (220) are not limited to these examples and may be installed at other locations within the vehicle (100), and may include more than three cameras.

[0196] Each of the multiple cameras (220) acquires vital data of the driver / passenger using the rPPG (Remote PPG) method. For example, each of the multiple cameras (220) can acquire a face image of the driver / passenger. Then, a face recognition algorithm can be applied to each acquired face image to collect heart rate data of the driver / passenger using the rPPG (Remote PPG) method based on changes in blood flow measured. At this time, it should be noted that the multiple vital data acquired by each of the multiple cameras (220) are heart rate data of the same person.

[0197] The vital data integration device (800) can receive detection results and / or control results from at least one of the motion detection module (235), ADAS system (300), and vehicle control unit (170), and can evaluate the reliability of the vital data differently.

[0198] The reliability evaluation of vital data is performed quantitatively by considering the accuracy, timeliness, and consistency of each vital data. For example, if the first camera (221) provides data with high accuracy and low timeliness, the second camera (222) provides data with somewhat low accuracy and high timeliness, and the third camera (223) provides data with average accuracy and timeliness, different weights may be assigned as a result of the reliability evaluation for each of the vital data collected by them.

[0199] Meanwhile, the vital data integration device (800) can reflect the detection result and / or control result of at least one of the motion detection module (235) in the vehicle (100), the ADAS system (300), and the vehicle control unit (170) in the reliability evaluation.

[0200] In an embodiment, the vital data integration device (800) can change the weight of the received vital data based on the detection result and / or control result received from the motion detection module (235), the ADAS system (300), and / or the vehicle control unit (170).

[0201] For example, if an event is detected by the ADAS system (300) and an emergency brake is applied without driver intervention, the heart rate of the driver / passenger will increase, so a higher weight can be temporarily assigned to the vital data.

[0202] In another embodiment, the vital data integration device (800) can change the weight calculation cycle to be assigned to the received vital data based on the detection results and / or control results received from the motion detection module (235), the ADAS system (300), and / or the vehicle control unit (170).

[0203] For example, if the control result of the vehicle control unit (170) remains the same or only shows small changes for a long time, the driver's heart rate may decrease due to drowsiness, etc., so in this case, the weight calculation cycle for vital data can be shortened to perform a response to the drowsiness prevention service.

[0204] In another embodiment, the vital data integration device (800) can adjust the degree of weight change or algorithm sensitivity for vital data based on detection results and / or control results received from the motion detection module (235), ADAS system (300), and / or vehicle control unit (170).

[0205] For example, when the driver / passenger's movement is significantly detected according to the detection result of the motion detection module (235), the heart rate will increase due to the driver / passenger's body movement, so the degree of change in weight of the entire vital data or the sensitivity of the algorithm can be lowered and applied.

[0206] Meanwhile, in some embodiments, in addition to the configuration within the vehicle (100) shown in FIG. 8, the reliability evaluation of vital data by the vital data integration device (800) may vary based on the detection results and / or control results of other configurations within the vehicle (100) or network data provided from outside the vehicle (100). For example, the reliability evaluation of vital data for the current driver / passenger may vary based on traffic condition information, weather information, health status information received from the driver / passenger's terminal, etc.

[0207] Once weights are assigned based on the reliability evaluation results, the final generated data is input into an algorithm for estimating the driver / passenger's state through the selection and fusion of the weighted vital data. Since the final generated data consists of reliable heart rate data, the driver / passenger's state can be estimated more accurately.

[0208] Hereinafter, FIG. 9 is a block diagram for explaining the detailed configuration of a vital data integration device (800) related to an embodiment of the present invention.

[0209] Referring to FIG. 9, the vital data integration device (800) according to the present invention may include a receiving unit (810), a data determining unit (820), and a data providing unit (830). Here, the data determining unit (820) may have the same meaning as a control unit / processor. Also, the data providing unit (830) may be a transmission unit / transceiver for transmitting final generated data to a driver / passenger state estimation algorithm.

[0210] The receiver (810) can receive rPPG (Remote PPG) based heart rate data measured from face images obtained from multiple cameras installed at different locations within the vehicle (100) as vital data.

[0211] The received vital data is multiple heart rate data sensed for the same person. Multiple cameras installed at different locations within the vehicle (100) are exemplified as acquiring vital data of the same person using the rPPG (Remote PPG) method, but are not limited to this method of acquiring data.

[0212] Each vital data received by the receiver (810) has a different value and precision. The receiver (810) provides each vital data to the data determination unit (820), and the data determination unit (820) performs selection and integration of data to provide a highly reliable driver status determination service.

[0213] The data determination unit (820) evaluates the reliability of the received vital data and assigns a weight. Additionally, the data determination unit (820) selects highly reliable vital data or fuses multiple vital data based on the weight assigned to each vital data.

[0214] The data determination unit (820) can evaluate the reliability of each vital data based on landmark information included in the driver / passenger face image, the reliability of the logic, the amount of change compared to previous data, the reliability of each of the multiple cameras, etc. At this time, the result of the reliability evaluation is expressed as a weight value.

[0215] The data determination unit (820) can select or fuse one or more of the received vital data based on landmark information included in the driver / passenger face image, the reliability of the logic, the amount of change compared to previous data, the reliability of each of the multiple cameras, etc. For example, vital data having outliers can be removed or missing vital data can be ignored.

[0216] The data providing unit (830) receives selected or fused vital data from the data determining unit (820).

[0217] The data provider (830) provides the final generated vital data to the state index estimation algorithm (840) for high-level vital analysis so that the state index of the driver / passenger can be estimated based on the selected or fused vital data. Here, the state index of the driver / passenger may refer to one of the driver / passenger's stress index, drowsiness index, or fatigue index.

[0218] State index estimation (840) uses the final generated vital data as input data for each state estimation algorithm to perform an evaluation of the driver / passenger's fatigue index, stress index, and emergency status. Based on the calculated fatigue index, stress index, and emergency status evaluation, various actions and services related to the driving and safety of the vehicle (100) can be provided to the vehicle (100).

[0219] FIG. 10 is a representative flowchart illustrating a vital data integration method related to an embodiment of the present invention.

[0220] Each step illustrated in FIG. 10 may correspond to instructions executed by a processor of a computing device, and in this case, the vital data integration method may be implemented in the form of a downloadable computer program or a computer program recording medium containing instructions.

[0221] Unless otherwise stated, each step of the flowchart of FIG. 10 is performed by any one of the above-described vital data integration device (800), more specifically the receiving unit (810), data determination unit (820), and data provision unit (830) described in FIG. 9.

[0222] The vital data integration device (800) receives each vital data sensed by a plurality of cameras installed in the vehicle (S10).

[0223] Each of the multiple cameras installed in the vehicle acquires the heart rate of the same driver / passenger as vital data using the rPPG (Remote PPG) method. The rPPG (remote PhotoPlethySmog) method is a non-contact method for acquiring biosignals. Specifically, it is a method that extracts skin pixels from a face image acquired through a camera, and then acquires heart rate data by analyzing changes in skin color caused by blood flow volume resulting from heartbeats.

[0224] When vital data is received in this manner, the vital data integration device (800) evaluates the reliability of the received vital data and assigns a weight (S20).

[0225] In an embodiment, the step of assigning weights (S20) may include evaluating the reliability of each vital data and determining different weights based on at least one of the reliability of each of the plurality of cameras in the vehicle, landmark information of images sensed through the plurality of cameras, the area occupied by the face in each image, and the amount of change compared to previous vital data.

[0226] For example, different weights can be assigned to each camera based on the accuracy of the face recognition algorithm used in each of the multiple cameras. Additionally, for example, higher weights can be assigned to data based on images where the acquired face image contains many landmarks or has a large occupant area. Alternatively, for example, higher weights can be assigned to data from images where the angle between the face and the camera is close to a right angle. Furthermore, for example, higher weights can be assigned to data from images with high accuracy scores among the results of the face recognition algorithm. Also, for example, higher weights can be assigned to data with high continuity and similarity by comparing the amount of change with previous vital data.

[0227] In an embodiment, the step of assigning weights (S20) may include a step of further adjusting the reliability weight for each vital data based on the vehicle's control information and the passenger's movement information.

[0228] Here, vehicle control information refers to control information for controlling the vehicle's braking and steering. In this case, vehicle braking and steering include both cases caused by driver operation and sudden braking, sudden acceleration, and sudden steering triggered without driver intervention through the operation of the ADAS system.

[0229] In addition, the passenger movement information here refers to the results of monitoring the movement of the driver and passenger based on the operation results of the camera-based motion monitoring module.

[0230] In this way, after different weights are assigned to the vital data, the vital data integration device (800) selects or fuses the vital data based on the assigned weights (S30).

[0231] The selection and fusion of vital data can be performed through any one of 1) a data fusion method, 2) a soft voting method, or 3) a fusion of soft voting and data fusion.

[0232] 1) The data fusion method is a method that removes abnormal data from multiple sensed vital data and then integrates each value of the vital data into a “single” data stream to provide it. Here, abnormal data refers to a value that has a different value or a large amount of change compared to all other data.

[0233] 2) The soft voting method calculates a weighted probability for each sensed vital data by averaging the probability values ​​for each. It is a method of selecting the label with the highest probability among the multiple calculated values.

[0234] 3) The fusion of soft voting and data fusion is the addition of the data fusion method after the soft voting described above. Specifically, it is a method that calculates a weighted probability for each of the sensed vital data by averaging the probability values ​​for each, and then additionally performs a data fusion method that calculates a weighted average based on each value.

[0235] The vital data integration device (800) generates reliable final vital data by selecting and fusing vital data through the various methods described above.

[0236] Next, the vital data integration device (800) can provide the final generated vital data as input data for the state index estimation (840, FIG. 9) to estimate the driver's state index (S40).

[0237] At this time, the final generated vital data is provided to a plurality of different state index algorithms simultaneously or sequentially to provide various state indices for the same person as result values.

[0238] In some embodiments, the vital data integration device (800) may provide the final generated vital data to a device / server / cloud, etc., that includes various state index algorithms. In this case, it may be said that the state index estimation (840) exists externally to the vital data integration device (800). Subsequently, the state index result value may be transmitted to the vital data integration device (800) or the vehicle (100) and used to track the state of the driver / passenger and / or provide related services.

[0239] As described above, according to the vital data integration method of the embodiment of the present invention, it is possible to provide seamless and stable data collection for the analysis and tracking of the driver / passenger's condition. In other words, reliable input data can be generated and provided for the analysis of high-level vital data.

[0240] FIG. 11 is a diagram illustrating the collection of vital data using a plurality of cameras in a vehicle related to an embodiment of the present invention.

[0241] Referring to FIG. 11, an example is provided of acquiring vital data of a driver using three cameras around the driver's seat in a vehicle (100). As illustrated, the plurality of cameras may include an IR camera (221, first camera) installed on the room mirror, an A-Pillar camera (222, second camera) on the driver's seat window side, and an IR camera (223, third camera) installed on the cluster.

[0242] However, the number and position of the cameras shown in FIG. 11 may be applied variably. However, the method of acquiring multiple vital data for the same person through multiple cameras, selecting, and fusing them is applied identically.

[0243] Meanwhile, each of the multiple cameras (221, 222, 223) acquires vital data of the driver / passenger using the rPPG (Remote PPG) method. However, this method is not limited to this method, and it is also possible to acquire camera-based vital data using other methods.

[0244] In FIG. 11, each of the plurality of cameras (221, 222, 223) is installed around the driver's seat in the vehicle and senses the driver's vital data using the rPPG (Remote PPG) method. Different vital data are sensed by each of the plurality of cameras (221, 222, 223) at the same time (or with a short time difference).

[0245] Specifically, each of the plurality of cameras (221, 222, 223) senses heart rate data corresponding to changes in blood flow from a face image of a driver / passenger seated in the driver's seat of the vehicle and transmits it to the vehicle's vital data integration device (800). Each of the sensed plurality of vital data has a different value and a different precision.

[0246] Multiple sensed vital data are time-synchronized at the receiving unit (810, FIG. 9) of the vital data integration device (800) and transmitted to the data determination unit (820).

[0247] As mentioned above, since multiple vital data sensed at the same time have different values ​​and precision, data selection and fusion (integration) are necessary for a highly reliable driver status judgment service.

[0248] FIG. 12 is an illustrative diagram for explaining data selection using a hard voting method related to an embodiment of the present invention. Hard voting is also called majority voting in that it is a method of selecting the result that receives the most votes among the results of each model.

[0249] Referring to FIG. 12, when a new driver state factor (new instance) (1200) is input to each of the multiple means for sensing the driver's state, such as multiple cameras (1201, 1202, 1203, 1204), a prediction of the new factor is sensed by each of the multiple cameras (1201, 1202, 1203, 1204). As a result, let's assume that the sensing result of three cameras (1201, 1202, 1204) is '1' and the sensing result of a specific camera (1203) is predicted as '2', which is different from these. Then, '1' is selected as the final data (1210) according to a majority vote, that is, an ensemble's prediction.

[0250] Meanwhile, the soft voting applied in the present invention produces more accurate and flexible results than the hard voting described above. Soft voting is also called probability voting. Specifically, soft voting does not ignore the sensing results of a specific camera (1203) in the example of FIG. 12, and produces results as probability values.

[0251] Specifically, let's assume there are three camera sensors that measure the driver's heart rate, and each sensor measures the driver's heart rate every second. In this case, the soft voting method determines the driver's heart rate through the following process.

[0252] Each of the three camera sensors transmits the measured value and time information reliability together. The vital data integration device (800) synchronizes the heart rate data transmitted from each camera in time and integrates it into one data stream.

[0253] For example, let’s assume that Camera A measured 80 bpm at 10:00:01, Camera B measured 82 bpm at 10:00:02, and Camera C measured 78 bpm at 10:00:03. The vital data integrator (800) combines these into a single data stream and performs the task of removing previously missing or outliers.

[0254] Specifically, the data determination unit (820) of the vital data integration device (800) removes data having outliers among a plurality of vital data, evaluates the reliability of normal vital data, and assigns a weight to each. At this time, the reliability evaluation is calculated quantitatively by considering the accuracy, timeliness, and consistency of the data.

[0255] For example, in the above example, if Camera A provides data with high accuracy and low recency, Camera Sensor B provides data with low accuracy and high recency, and Camera C provides data with average accuracy and recency, different weights can be assigned to each of them.

[0256] After different weights are assigned in this manner, the vital data integration device (800) may select the optimal data or average value according to the reliability of each vital data (data fusion), combine (fusion) them to calculate weight probability values ​​and then select the label having the highest probability value (soft voting), or select the average value thereof (soft voting + data fusion).

[0257] The final reliable data generated in this way is provided to the driver / passenger state index estimation (840, FIG. 9), thereby enabling the acquisition of more accurate and reliable state index results.

[0258] FIGS. 13 and FIGS. 14 are drawings for illustrating examples of assigning different weights to vital data according to a face recognition algorithm related to an embodiment of the present invention.

[0259] In an embodiment of the present invention, the reliability evaluation of vital data (e.g., heart rate data) received from a plurality of cameras is performed by determining a weight for each vital data based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by a face in the images, the reliability of the logic, and the amount of change compared to previous vital data.

[0260] Since vital data sensed by each of the multiple cameras has different values ​​and precision even though it was acquired at the same time, data selection and fusion (integration) are necessary to provide a highly reliable driver state judgment service.

[0261] As a data processing method for selecting and fusing weighted vital data, the present invention uses a data fusion method and a soft voting method.

[0262] Data Fusion is a data processing method that obtains more accurate and flexible results by integrating each value, which has weights applied to the results of applying a face recognition algorithm to face images sensed by multiple cameras, into a single data stream.

[0263] Soft Voting is a data processing method that applies weights to the results of a face recognition algorithm applied to face images sensed by multiple cameras, averages the respective probability values, and ultimately selects the label with the highest probability.

[0264] Prior to data selection and fusion, the method for assigning weights to each vital data point is as follows. The weighting of each vital data point begins with determining which factors to assign weight to. Below, we will examine specific examples of assigning weights based on various factors.

[0265] 1) When the result of applying a face recognition algorithm is used as a factor:

[0266] In one embodiment, weights can be assigned to images where the angle between each of the multiple cameras and the driver / passenger's face is close to a right angle.

[0267] The angle of the face can be checked during the operation of the face recognition algorithm within the deep learning algorithm. Depending on the angle of the driver / passenger's face, it affects the size of landmarks and changes in skin color due to changes in blood flow, which can measure the Heart Rate (HR). For this reason, weights are assigned to cameras that measure the angle between the driver / passenger's face and the camera as being close to a right angle.

[0268] For example, referring to FIG. 13, each of the multiple cameras installed in the vehicle (100) changes its shooting angle at each sensing point according to the head movement of the driver / passenger, etc. Accordingly, as shown in FIG. 13, a face image captured at various shooting angles is obtained at each sensing point. In FIG. 13, a weight is applied to the image (1310) in which the shooting angle of the camera and the angle of the driver / passenger's face are close to a right angle.

[0269] In another embodiment, a high weight can be assigned to the image with the largest and clearest landmark among face images acquired through multiple cameras.

[0270] When operating a face recognition algorithm within a deep learning algorithm, deriving face landmarks and verifying their areas is a very important factor. This is because the size of face landmarks has a significant impact on the performance of the algorithm after face recognition.

[0271] Face landmarks refer to feature points of the face (e.g., eyes, nose, mouth, eyebrows, etc.). When images are captured through multiple cameras, for example, the face of a driver or passenger is detected using a face detection algorithm, and subsequently, landmarks within the face (e.g., eyes, nose, mouth, eyebrows, etc.) can be detected using a face landmark algorithm.

[0272] Referring to Fig. 14, it can be seen that landmarks such as eyes, eyebrows, and lips are detected in the face image as a result of applying the face landmark algorithm. Among multiple images captured at the same time by multiple cameras, as shown in Fig. 14, weights are assigned to use the image in which the landmark of the driver / occupant's face appears largest and clearest as the main image.

[0273] In another embodiment, weights can be assigned to images acquired through multiple cameras where many facial landmarks appear.

[0274] In another embodiment, weights can be assigned to areas where the driver / occupant occupies a large portion of the image acquired through multiple cameras.

[0275] Specifically, this is a method that places weight on the larger area occupied by the driver / passenger's face and body before detecting landmarks using deep learning algorithms. Even if the landmark may be narrow when the driver / passenger's face is obscured by a hat, sunglasses, etc., this method places weight on the larger area of ​​the driver / passenger in the actual video. This differs from the facial landmark method in that it utilizes a larger portion of the driver / passenger's outline.

[0276] In another embodiment, weights can be assigned to the camera among multiple cameras where a higher accuracy algorithm is executed.

[0277] There may be performance differences depending on the algorithm executed on multiple cameras. Additionally, even with the same rPPG algorithm, optimization is applied differently depending on the installation location, performance, and type of the cameras, which may result in performance differences. For example, if the accuracy of Camera A's algorithm is 95% and the accuracy of Camera B's algorithm is 75%, a higher weight may be assigned to Camera A.

[0278] As another embodiment, weights can be assigned to the algorithm result values ​​with higher accuracy values ​​within the same algorithm.

[0279] When the vital data measurement algorithm operates, it outputs the data accuracy as a result, indicating the accuracy status under which the currently measured vital data was obtained. Accordingly, by assigning weights based on the accuracy included in this algorithm result, it can be utilized for the fusion and selection of vital data.

[0280] In another embodiment, weights can be assigned to each of the multiple vital data that have high continuity and similarity with the previous vital data value.

[0281] Human vital data exhibits continuity. That is, when a specific index in vital data increases or decreases, it tends to increase or decrease linearly. Based on this characteristic, data that changes rapidly compared to previous data points is highly likely to be an outlier or involve measurement error. Therefore, if continuously measured vital data loses linearity or exhibits excessive variability, the reliability of the data can be evaluated by giving greater weight to data that is highly similar to previous data.

[0282] For example, let’s assume that the heart rate is measured every second. For Camera A, let’s say it is 60 bpm at 0 seconds, 75 pbm at 1 second, and 61 pbm at 2 seconds. Also, for Camera B, let’s say it is 60 pbm at 0 seconds, 62 pbm at 1 second, and 61 pbm at 2 seconds. Then, data reliability can be ensured by assigning weights to the data results of Camera B, which ensures the continuity of the heart rate data at the 1-second point.

[0283] 2) When the performance of the camera itself is used as a factor:

[0284] In one embodiment, weights may be assigned to cameras with a higher pixel count of the captured face area. This utilizes the fact that a higher pixel count of the camera enables the capture of clearer and larger facial landmarks.

[0285] As another embodiment, weights can be assigned to high-resolution cameras. This utilizes the fact that higher camera resolution enables the capture of clearer and larger facial landmarks.

[0286] In another embodiment, weights can be assigned to cameras with high frame rates.

[0287] The frame rate of a camera has a significant impact on the accuracy of estimating the Heart Rate (HR) after measuring vital data or calculating the stress index, which is one of the vital post-processing steps. Therefore, a high frame rate is essential for more accurate vital data processing. Multiple cameras installed in a vehicle (100) have frame rates that vary greatly depending on the purpose of installation and the performance of the ISP processing chip. Accordingly, weights may be assigned based on the size of the frame rate.

[0288] As another embodiment, weights can be assigned based on the overall / partial illumination of the camera image being high.

[0289] When measuring rPPG using an RGB camera, algorithm performance can be significantly affected by shadows or light cast on the faces of the driver or occupant inside the vehicle. Additionally, the algorithm may not operate at all if the surroundings are too bright or too dark. This applies equally to IR cameras as well as RGB cameras. Specifically, since IR cameras are affected by ambient light, image illumination is a critical factor influencing algorithm performance. Therefore, algorithm performance can be improved by assigning specific weights to areas with high overall or partial illumination in the camera image.

[0290] As another example, weights can be assigned to images with low noise. Noise in an image is a major cause of reduced accuracy in the algorithm. Therefore, the performance of the algorithm can be improved by assigning lower weights to images with high noise.

[0291] In the above, we examined which factors are weighted when evaluating the reliability of vital data acquired through multiple cameras. Below, we will explain in detail the method of distributing weights to the corresponding factors.

[0292] Meanwhile, in a method of measuring vital data non-contactually through multiple cameras in a vehicle (100), measurement errors occur during measurement, so correction and error removal work on the data is required.

[0293] Accordingly, the present invention proposes a method for processing data by assigning weights based on a reliability evaluation of the measured vital data. In the present invention, as a method for distributing weights to vital data, the following methods were used: 1) a data fusion method, 2) a soft voting method, and 3) a combination of the two, as described above.

[0294] Data Fusion is a data processing method that acquires vital data in real time from multiple cameras capturing the driver / occupant, and integrates the multiple vital data into a single data stream by synchronizing the vital data acquired through each camera in time.

[0295] Soft Voting is a data processing method that generates final vital data by excluding missing or outlier values ​​from vital data acquired from each camera, averaging normal data, or fusing them through soft voting.

[0296] The final vital data generated through data fusion, soft voting, or a combination thereof is passed to an upper layer and used as input data to estimate the driver / passenger state index.

[0297] FIG. 15 is a flowchart illustrating a method for generating final input data by selecting and fusing vital data related to an embodiment of the present invention using a soft voting method.

[0298] First, multiple cameras within the vehicle (100), camera 1 (221), camera 2 (222), and camera 3 (223), each sense vital data for the same driver / passenger at the same time.

[0299] Specifically, vital data sensed through Camera 1 (221) is transmitted to the soft voting system (1510). Additionally, vital data sensed through Camera 2 (222) is transmitted to the soft voting system (1510). Additionally, vital data sensed through Camera 3 (223) is transmitted to the soft voting system (1510).

[0300] The soft voting system (1510) may be included in the form of an algorithm / processor / program within the vital data integration device (800). Alternatively, the soft voting system (1510) may be the data determination unit (820, FIG. 9) of the vital data integration device (800). Hereinafter, the soft voting system (1510) is disclosed as the data determination unit (820).

[0301] The data determination unit (820) selects vital data with a high reliability weight based on the weight assigned to each vital data, or fuses each vital data with the reliability weight reflected ("soft voting") and transmits it to the data provision unit (830, FIG. 9).

[0302] The data determination unit (820) applies weights to the probability values ​​of each vital data and then calculates an average based on the assigned weights. Then, the data determination unit (820) calculates the reliability weighted probability for each vital data.

[0303] In another embodiment, the data determination unit (820) calculates the average of each vital data based on the weight assigned to each vital data, selects the calculated average vital data, and transmits it to the data provision unit (830) as a single integrated stream ("data fusion").

[0304] In another embodiment, the data determination unit (820) may apply weights to the probability values ​​of each vital data and then select the vital data having the highest value among the reliability weight probabilities (label having the highest probability). Additionally, the data determination unit (820) may generate fused vital data by calculating the average of the reliability weight probabilities and transmit the selected vital data or the fused vital data to the data provision unit (830).

[0305] Next, the final generated reliable vital data is provided to the final level (1520) (1504) to more accurately estimate the driver / passenger condition index.

[0306] FIGS. 16 and 17 are block diagrams and flowcharts for explaining acquiring, integrating, evaluating, and selecting a plurality of vital data related to an embodiment of the present invention and providing them to a driver state estimation algorithm.

[0307] Referring to FIG. 16, the plurality of cameras (220) in the vehicle may be a first camera (221) installed in the rearview mirror, a second camera (222) installed in the driver's seat A-Pillar, and a third camera (223) installed in the cluster.

[0308] Multiple vital data acquired at the same time through multiple cameras (220) are transmitted to a vital data analysis module (1610). Here, the vital data analysis module (1610) may refer to a vital data integration device (800) according to the present invention.

[0309] The vital data analysis module (1610) may include a data synchronization (1601), a detector (1602), an evaluator (1603), a data selector (1604), a calculator (1605), and a dispatcher (1606).

[0310] Data synchronization (1601) synchronizes multiple vital data sensed through each of multiple cameras at the same time (but not perfectly at the same time) in time.

[0311] The detector (1602) can remove missing or outlier data from time-synchronized vital data and detect normal vital data.

[0312] The evaluator (1603) evaluates the reliability of each vital data according to a preset reliability criterion and assigns a weight.

[0313] For example, based on preset reliability criteria, a method can be adopted and applied that assigns weight as the angle between the aforementioned face and the camera becomes closer to a right angle.

[0314] Specifically, when each of the multiple cameras (220) recognizes the same driver / passenger A, a higher rank or higher weight may be assigned as the shooting angle is closer to a right angle. For example, the first camera (221) closest to a right angle may be assigned a weight of '3', the second camera (222) may be assigned a weight of '2', and the third camera (223) may be assigned a weight of '1'.

[0315] The data selector (1604) and the calculator (1605) select and fuse the final data based on weights assigned to multiple vital data and provide it to the dispatcher (1606).

[0316] Specifically, the data selector (1604) and the calculator (1605) can calculate a new final data by fusing the multiple vital data based on the weights assigned to them. For example, the final data for the same driver / passenger A in the above example may be a calculated value of {(first camera (221) * 3) + (second camera (222) * 2) + (third camera (223) * 1)} / 3.

[0317] As another example, the data selector (1604) and the calculator (1605) may average the respective probability values ​​based on the weights assigned to multiple vital data to select the label with the highest final probability or calculate the average of these values ​​as new final data.

[0318] Specifically, in the above example, let the recognition probability of the first camera (221) be 0.8, the recognition probability of the second camera (222) be 0.7, and the recognition probability of the third camera (223) be 0.6, and let the respective weights be 0.6, 0.3, and 0.1 in order from closest to right angle.

[0319] In this case, the weighted probability values ​​for each camera are as follows. Specifically, the weighted probability value of the first camera (221) is calculated as (0.8*0.6) / (0.6+0.3+0.1) = 0.48. The weighted probability value of the second camera (222) is calculated as (0.7*0.3) / (0.6+0.3+0.1) = 0.21. The weighted probability value of the third camera (223) is calculated as (0.6*0.1) / (0.6+0.3+0.1) = 0.06.

[0320] At this time, the vital data sensed by the first camera (221) with the highest weight probability value may be finally selected, or the average of these may be determined as a single fused final value by additionally fusing the data fusion method.

[0321] Meanwhile, the dispatcher (1606) extracts the final generated vital data and provides it to the state estimation algorithm (1620).

[0322] The state estimation algorithm (1620) calculates / estimates various state indices using reliable vital data transmitted from the dispatcher (1606) as input data.

[0323] As illustrated in FIG. 16, the results are transmitted to the fatigue estimation algorithm (1621), stress evaluation algorithm (1622), and emergency state evaluation algorithm (1623), respectively, to estimate the fatigue of the driver / passenger, calculate a stress index, and evaluate whether there is an emergency state. The evaluation results of the algorithms are transmitted to the vehicle (100) to enable the performance of associated additional actions / services. In addition, it goes without saying that more state index algorithms may be included in addition to the algorithms illustrated in FIG. 16.

[0324] Figure 17 is an operation flowchart associated with the block diagram of Figure 16.

[0325] Referring to FIG. 17, vital data sensed from camera 1 (221) is transmitted to the data integrator (1710) (1701). Also, vital data sensed from camera 2 (222) is transmitted to the data integrator (1710) (1702). At the same time, vital data sensed from camera 3 (223) is transmitted to the data integrator (1710) (1701).

[0326] The data integrator (1710) synchronizes multiple vital data in time and removes data that is missing or has outliers, then transmits it to the data evaluator (1720) (1704).

[0327] The data evaluator (1720) evaluates the reliability of each vital data according to a preset reliability standard, calculates each weight, and transmits it to the data selector (1730) (1705).

[0328] The data selector (1730) selects and fuses reliable final data according to the data processing method based on the aforementioned data fusion method, soft voting method, or a combination of the two (1706). The vital data finally generated by the data selector (1730) is transmitted to various algorithms of the driver state estimation algorithm (1740) respectively (1707, 1708, 1709). The various state index evaluations based thereon are transmitted to the associated operation system within the vehicle (100) to provide various services.

[0329] Meanwhile, in embodiments of the present invention, the vital data integration device (800) can reflect the sensing results and / or control results of other sensors or operating systems of the vehicle in the reliability evaluation for each vital data.

[0330] In an embodiment, the data determination unit (810, FIG. 9) of the vital data integration device (800) is

[0331] Reliability weights for each vital data point can be adjusted differently based on vehicle control information and occupant movement information.

[0332] Here, vehicle control information refers to control information for controlling the vehicle's braking and steering. In this case, vehicle acceleration and steering include both cases caused by driver operation and cases where sudden braking, sudden acceleration, or sudden steering is triggered without driver intervention by the operation of the ADAS system.

[0333] Additionally, the passenger movement information refers to the driver / passenger movement information obtained through a camera-based motion monitoring module, and is sensed by a motion monitoring module installed in the vehicle (100).

[0334] In an embodiment, the data determination unit (820) of the vital data integration device (800) may determine that the amount of change in each vital data has temporarily changed based on the vehicle's control information. Then, according to such determination, the reliability weight for each vital data may be reduced. Alternatively, in another embodiment, the weight calculation cycle for each vital data may be extended according to such determination.

[0335] For example, there may be sudden vehicle control events, such as a sudden braking event caused by a frontal accident while driving, or internal and external events of the vehicle (100) may have a significant impact on changes in the heart rate of the driver / occupant (Occupant). In such cases, the weight calculation cycle for vital data may be temporarily extended, or the weight value based on reliability evaluation may be reduced by a certain amount.

[0336] In another embodiment, the data determination unit (820) of the vital data integration device (800) determines that there is no change in the control information of the vehicle received for a predetermined period of time or longer, and according to such determination, increases the reliability weight for each vital data or changes the reliability weight calculation cycle to a shorter period.

[0337] At this time, the above-mentioned predetermined time may be a period sufficient to determine that there is no driving operation due to the occurrence of driver drowsiness or an emergency situation. However, the above-mentioned predetermined time may vary depending on the driving state of the vehicle (100), the driving time, information on the driver's health condition, etc.

[0338] For example, when driving with steering and braking that remain the same or unchanged for a long period of time, the driver may experience changes in vital data, such as a drop in heart rate due to drowsy driving in that situation. In such a situation, the weight of all vital data sensed from multiple cameras in the vehicle (100) can be increased or the weight calculation cycle can be shortened to be utilized for responding to driver drowsiness prevention services.

[0339] In another embodiment, the data determination unit (820) of the vital data integration device (800) determines that the amount of change in each vital data has temporarily changed based on the movement information of the occupant, and according to such determination, can reduce the reliability weight for each vital data or change the reliability weight calculation cycle to a longer period.

[0340] For example, if there is a lot of movement in the vehicle, such as eating, talking, or operating large steering movements while driving, the driver's heart rate may temporarily increase due to physical movement. Changes in heart rate caused by such temporary movement may cause confusion in services related to assessing the driver's stress level or fatigue and status. Accordingly, when large temporary movement of the driver / occupant is detected, changes may be made to lower the weight of all vital data sensed from multiple cameras in the vehicle (100) and / or lengthen the weight calculation cycle.

[0341] As described above, according to the vital data integration device for a vehicle according to an embodiment of the present invention, when estimating driver / passenger drowsiness, stress, health information, etc. using a plurality of cameras, vital data of the driver / passenger can be collected and provided stably, seamlessly, and with higher accuracy. In other words, reliable input data can be provided for high-level vital analysis.

[0342] The above-described invention may be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also include implementation in the form of a carrier wave (e.g., transmission over the Internet). Additionally, the computer may include a controller / processor of a vital data integration device (800). Accordingly, the above detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

Claim 1 A vital data integration device for a vehicle, comprising: a receiving unit that receives each vital data collected by a plurality of cameras installed inside the vehicle; a data determination unit that evaluates the reliability of the received vital data, assigns weights, and selects or fuses the vital data based on the assigned weights; and a data providing unit that provides the final generated vital data to estimate the driver's state index. Claim 2 A vital data integration device for a vehicle according to claim 1, wherein the reliability evaluation of the received vital data determines different weights for each vital data based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by a face in the images, and the amount of change compared to previous vital data. Claim 3 A vital data integration device for a vehicle according to claim 1, wherein each of the plurality of cameras is installed around the driver's seat in the vehicle and senses the driver's vital data using the rPPG (Remote PPG) method, and the receiving unit time-synchronizes the sensed plurality of vital data and transmits them to the data determining unit. Claim 4 A vital data integration device for a vehicle according to claim 3, wherein the data determination unit removes data having outliers among the plurality of vital data, evaluates the reliability of normal vital data, and assigns weights. Claim 5 A vital data integration device for a vehicle according to claim 1, wherein the data determination unit selects vital data with a high reliability weight based on the assigned weight, or fuses each vital data with a reliability weight and transmits it to the data providing unit. Claim 6 A vital data integration device for a vehicle according to claim 5, wherein the data determination unit calculates the average of each vital data based on the assigned weights, selects the calculated average vital data, and transmits it to the data providing unit as a single stream. Claim 7 A vital data integration device for a vehicle, wherein, in claim 5, the data determination unit calculates a reliability weight probability for each vital data by applying the assigned weight to the probability value of each vital data and then calculating an average based on the assigned weight. Claim 8 A vital data integration device for a vehicle according to claim 7, wherein the data determination unit selects vital data having the highest value among the reliability weighted probabilities or calculates the average of the reliability weighted probabilities to generate fused vital data, and transmits the selected vital data or fused vital data to the data providing unit. Claim 9 A vehicle vital data integration device according to claim 1, wherein the data determination unit adjusts the reliability weight for each vital data differently based on the vehicle's control information and the passenger's movement information. Claim 10 A vital data integration device for a vehicle according to claim 9, wherein the data determination unit determines that the amount of change in each vital data has temporarily changed based on the vehicle's control information, and reduces the reliability weight for each vital data according to the determination. Claim 11 A vital data integration device for a vehicle, characterized in that, in claim 10, the control information of the vehicle is a control result corresponding to any one of braking, starting, or steering that exceeds a set range performed by driver operation or an ADAS system. Claim 12 A vital data integration device for a vehicle according to claim 9, wherein the data determination unit determines that there is no change in the control information of the vehicle received for a predetermined period of time or longer, and according to the determination, increases the reliability weight for each vital data or changes the reliability weight calculation cycle to a shorter period. Claim 13 A vital data integration device for a vehicle according to claim 9, wherein the data determination unit determines that the amount of change in each vital data has temporarily changed based on the movement information of the occupant, and, in accordance with the determination, reduces the reliability weight for each vital data or changes the reliability weight calculation cycle to be longer. Claim 14 A method for integrating vital data of a vehicle, comprising: receiving each vital data sensed by a plurality of cameras installed inside the vehicle; evaluating the reliability of the received vital data and assigning weights; selecting or fusing the vital data based on the assigned weights; and providing the final generated vital data as input data to estimate the driver's state index. Claim 15 A method for integrating vital data of a vehicle according to claim 14, wherein the step of assigning weights is characterized by determining different weights by evaluating the reliability of each vital data based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by a face in the images, and the amount of change compared to previous vital data. Claim 16 A method for integrating vital data of a vehicle, wherein, in claim 14, the step of assigning weights includes the step of additionally adjusting the reliability weight for each vital data based on the vehicle's control information and the passenger's movement information.

Images