Vital sign data integration apparatus and vital sign data integration method for a vehicle

Abstract

The vital sign data integration device of the vehicle of the present application includes: a reception section that receives each vital sign data collected by a plurality of cameras provided in the interior of the vehicle; a data determination section that evaluates the reliability of the received vital sign data and assigns a weight, and filters or fuses the vital sign data based on the assigned weight; and a data provision section that provides the finally generated vital sign data to estimate the state index of the driver. Thus, input data with reliability can be provided to perform high-level vital sign analysis.

Description

Technical Field

[0001] This invention relates to a vehicle vital signs data integration device and a vital signs data integration method, and more specifically, to a vital signs data integration device and a vital signs data integration method utilizing vital signs data sensed by a plurality of cameras inside the vehicle. Background Technology

[0002] A vehicle is a device that enables its users to move in the desired direction. A car can be taken as a representative example.

[0003] On the other hand, there is a trend towards installing various sensors and electronic devices to enhance user convenience. In particular, research is actively underway on Advanced Driver Assistance Systems (ADAS) to improve driving convenience for users. Furthermore, autonomous vehicles are being actively developed.

[0004] Furthermore, research on technologies for continuously monitoring driver drowsiness is increasing recently. Additionally, Euro NCAP, the European car assessment body, has incorporated technologies for non-contact monitoring of driver and passenger drowsiness levels into its assessment programs. Consequently, multiple cameras are used inside the vehicle to detect driver drowsiness in order to achieve unobstructed tracking of driver expressions and facial features. Summary of the Invention

[0005] The problem that the invention aims to solve

[0006] According to some embodiments of the present invention, the objective is to provide a vehicle vital sign data integration device and a vital sign data integration method that can continuously generate reliable vital sign data by filtering and fusing multiple vital sign data obtained from multiple cameras inside the vehicle.

[0007] Technical solutions to the problem

[0008] Therefore, the vehicle vital signs data integration device of the present invention can more reliably estimate the driver's condition index by filtering, fusing and providing vital signs data collected by a plurality of cameras installed inside the vehicle.

[0009] In addition, the vehicle's vital signs data integration device can evaluate the reliability of vital signs data based on various identification methods and assign different weights to each data point accordingly, thereby generating highly reliable vital signs data.

[0010] Specifically, the vehicle vital sign data integration device according to an embodiment of the present invention may include: a receiving unit for receiving various vital sign data collected by a plurality of cameras installed inside the vehicle; a data determining unit for evaluating the reliability of the received vital sign data and assigning weights, and filtering or fusing vital sign data based on the assigned weights; and a data providing unit for providing the finally generated vital sign data to estimate the driver's condition index.

[0011] In an embodiment, the reliability of the received vital sign data can be evaluated based on at least one of the following: the reliability of each of the plurality of cameras, key point information of the images sensed by the plurality of cameras, the area occupied by the face in the image, and the amount of change compared to previous vital sign data, to determine the different weights of each vital sign data.

[0012] In one embodiment, each of the plurality of cameras may be positioned around the driver's seat inside the vehicle to sense the driver's vital signs data in a remote PPG (rPPG) manner. The receiving unit may synchronize the sensed plurality of vital signs data in time and transmit them to the data determining unit.

[0013] In an embodiment, the data determination unit may remove data with outliers from a plurality of vital sign data, and then evaluate and assign weights to the reliability of normal vital sign data.

[0014] In an embodiment, the data determination unit may filter vital sign data with high reliability weights based on the assigned weights, or fuse various vital sign data reflecting reliability weights, and transmit them to the data providing unit.

[0015] In an embodiment, the data determination unit may calculate the average value of each vital sign data based on the assigned weights, filter the calculated average vital sign data, and transmit it to the data providing unit in a single stream.

[0016] In an embodiment, the data determination unit may apply the assigned weights to the probability values ​​of each vital sign data, calculate the average value based on the assigned weights, and calculate the reliability weight probability for each vital sign data.

[0017] In an embodiment, the data determination unit may filter the vital sign data with the highest value among the reliability weight probabilities, or calculate the average value of the reliability weight probabilities to generate fused vital sign data, and transmit the filtered vital sign data or fused vital sign data to the data providing unit.

[0018] In an embodiment, the data determination unit can adjust the reliability weights of each vital sign data differently based on the vehicle's control information and the passenger's motion information.

[0019] In an embodiment, the data determination unit can determine, based on the vehicle's control information, that the change in each vital sign data has temporarily changed, and reduce the reliability weight of each vital sign data based on the determination.

[0020] In an embodiment, the vehicle control information may be a control result corresponding to any of the braking, starting, or steering actions that exceed a set range and are performed by the driver or the ADAS system.

[0021] In an embodiment, the data determination unit can determine that the received vehicle control information has not changed for a specified period of time, and increase the reliability weight of each vital sign data or shorten the reliability weight calculation cycle according to the determination.

[0022] In an embodiment, the data determination unit can determine that the change in the amount of each vital sign data has temporarily changed based on the rider's motion information, and reduce the reliability weight of each vital sign data or extend the reliability weight calculation period based on the determination.

[0023] Additionally, the vehicle vital sign data integration method of the present invention may include: receiving vital sign data sensed by a plurality of cameras installed inside the vehicle; evaluating the reliability of the received vital sign data and assigning weights; filtering or fusing the vital sign data based on the assigned weights; and providing the finally generated vital sign data as input data to estimate the driver's condition index.

[0024] In an embodiment, the step of assigning the weights may be a step of evaluating the reliability of each vital sign data based on at least one of the following: the reliability of each of the plurality of cameras, key point information of the images sensed by the plurality of cameras, the area occupied by the face in the image, and the amount of change compared to previous vital sign data, and determining different weights for each.

[0025] In an embodiment, the step of assigning the weights may include: further adjusting the reliability weights of each vital sign data based on the vehicle's control information and the passenger's motion information.

[0026] Invention Effects

[0027] The effects of the vehicle vital signs data integration device and vital signs data integration method of the present invention will be explained below.

[0028] According to embodiments of the present invention, when multiple cameras are used to determine the driver's / passenger's drowsiness, stress, health information, etc., the driver's / passenger's vital sign data can be stably, continuously, and with higher accuracy summarized and provided. That is, reliable input data can be provided for high-level vital sign analysis. Attached Figure Description

[0029] Figure 1 This is a diagram illustrating an example of a vehicle related to an embodiment of the present invention.

[0030] Figure 2 These are diagrams showing vehicles related to embodiments of the present invention viewed from various angles.

[0031] Figure 3 and Figure 4 This is a diagram showing the interior of a vehicle in relation to an embodiment of the present invention.

[0032] Figure 5 and Figure 6 The diagram is a reference to various objects related to the driving of a vehicle in relation to embodiments of the present invention.

[0033] Figure 7 This is a diagram illustrating a vehicle vital signs data integration device related to an embodiment of the present invention.

[0034] Figure 8 This is a block diagram illustrating the state of interaction between the vital signs data integration device and other components of the vehicle in relation to embodiments of the present invention.

[0035] Figure 9 This is a block diagram illustrating the detailed configuration of a vital signs data integration device related to embodiments of the present invention.

[0036] Figure 10 This is a representative flowchart illustrating a vital signs data integration method related to embodiments of the present invention.

[0037] Figure 11 This is a diagram illustrating the collection of vital signs data using a plurality of cameras inside a vehicle, in relation to embodiments of the present invention.

[0038] Figure 12 This is an example diagram illustrating the data filtering method of hard voting in relation to embodiments of the present invention.

[0039] Figure 13 and Figure 14 This is a diagram illustrating an example of assigning different weights to vital sign data according to a face recognition algorithm, in relation to embodiments of the present invention.

[0040] Figure 15 This is a flowchart illustrating a method for filtering and fusing vital sign data and generating final input data in a soft voting manner, in relation to embodiments of the present invention.

[0041] Figure 16 and Figure 17 These are block diagrams and flowcharts illustrating the acquisition, integration, evaluation, and screening of multiple vital sign data in relation to embodiments of the present invention and their provision to a driver state estimation algorithm. Detailed Implementation

[0042] Figure 1 and Figure 2 This refers to the appearance of the vehicle in relation to embodiments of the present invention. Figure 3 and Figure 4 This is a diagram showing the interior of a vehicle in relation to an embodiment of the present invention.

[0043] Figures 5 to 6 This is a diagram illustrating various objects related to the driving of a vehicle in relation to embodiments of the present invention.

[0044] Figure 7 This is a block diagram illustrating a vehicle in relation to an embodiment of the present invention.

[0045] Reference Figures 1 to 7 The vehicle 100 may include wheels that are rotated by a power source and a steering input device 510 for adjusting the direction of travel of the vehicle 100.

[0046] Vehicle 100 may be an autonomous vehicle. Vehicle 100 may switch between autonomous driving mode and manual mode based on user input. For example, vehicle 100 may switch from manual mode to autonomous driving mode, or from autonomous driving mode to manual mode, based on user input received through a user interface device (hereinafter, which may be referred to as "user terminal") 200.

[0047] Vehicle 100 can switch between autonomous driving mode and manual mode based on driving condition information. The driving condition information can be generated based on object information provided by object detection device 300. For example, vehicle 100 can switch from manual mode to autonomous driving mode, or vice versa, based on driving condition information generated by object detection device 300. Alternatively, vehicle 100 can switch from manual mode to autonomous driving mode, or vice versa, based on driving condition information received via communication device 400.

[0048] Vehicle 100 can switch from manual mode to autonomous driving mode, or from autonomous driving mode to manual mode, based on information, data, and signals provided by external devices.

[0049] When vehicle 100 is operating in autonomous driving mode, autonomous vehicle 100 can operate based on operating system 700. For example, autonomous vehicle 100 can operate based on information, data or signals generated in driving system 710, vehicle dispatch system 740, and parking system 750.

[0050] When the vehicle 100 is operating in manual mode, the autonomous vehicle 100 can receive user input for driving via the driving control device 500. The vehicle 100 can operate based on the user input received via the driving control device 500.

[0051] Overall length refers to the length from the front to the rear of vehicle 100, overall width refers to the width of vehicle 100, and overall height refers to the length from the bottom of the wheels to the roof. In the following description, the overall length direction L can refer to the direction based on the overall length measurement of vehicle 100, the overall width direction W can refer to the direction based on the overall width measurement of vehicle 100, and the overall height direction H can refer to the direction based on the overall height measurement of vehicle 100.

[0052] like Figure 7 As exemplified, vehicle 100 may include a user interface device (hereinafter, which may be referred to as "user terminal") 200, an object detection device 300, a communication device 400, a driving operation device 500, a vehicle drive device 600, an operating system 700, a navigation system 770, a sensing unit 120, a vehicle interface unit 130, a memory 140, a control unit 170, and a power supply unit 190.

[0053] According to the embodiments, the vehicle 100 may include other constituent elements in addition to those described in this specification, or may exclude some of the constituent elements described.

[0054] The user interface device 200 is a means for communication between the vehicle 100 and the user. The user interface device 200 can receive user input and provide the user with information generated in the vehicle 100. The vehicle 100 can implement UI (User Interfaces) or UX (User Experience) through the user interface device (hereinafter, may be referred to as "user terminal") 200.

[0055] The user interface device 200 may include an input unit 210, an internal camera 220, a biometric detection unit 230, an output unit 250, and a processor 270. According to an embodiment, the user interface device 200 may include other components besides those described, or may exclude some of the described components.

[0056] The input unit 210 is used to receive information from the user. The data collected in the input unit 210 can be analyzed by the processor 270 and processed into control commands for the user.

[0057] The input unit 210 can be configured inside the vehicle. For example, the input unit 210 can be configured in an area of ​​the steering wheel, an area of ​​the instrument panel, an area of ​​the seat, an area of ​​each pillar, an area of ​​the door, an area of ​​the center console, an area of ​​the head lining, an area of ​​the sun visor, an area of ​​the windshield, or an area of ​​the window, etc.

[0058] The input unit 210 may include a voice input unit 211, a gesture input unit 212, a touch input unit 213, and a mechanical input unit 214.

[0059] The voice input unit 211 can convert the user's voice input into an electrical signal. The converted electrical signal can be provided to the processor 270 or the control unit 170. The voice input unit 211 may include one or more microphones.

[0060] The gesture input unit 212 can convert the user's gesture input into an electrical signal. The converted electrical signal can be provided to the processor 270 or the control unit 170.

[0061] The gesture input unit 212 may include at least one of an infrared sensor and an image sensor for detecting user gesture input. According to an embodiment, the gesture input unit 212 can detect three-dimensional gesture input from the user. For this purpose, the gesture input unit 212 may include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0062] The gesture input unit 212 can detect the user's three-dimensional gesture input through TOF (Time of Flight), structured light, or disparity methods.

[0063] The touch input unit 213 can convert the user's touch input into an electrical signal. The converted electrical signal can be provided to the processor 270 or the control unit 170.

[0064] The touch input unit 213 may include a touch sensor for detecting user touch input. According to an embodiment, the touch input unit 213 is integrated with the display unit 251, thereby realizing a touchscreen. This touchscreen can together provide both an input interface and an output interface between the vehicle 100 and the user.

[0065] The mechanical input unit 214 may include at least one of a button, a dome switch, a rotary knob, and a rotary switch. The electrical signal generated by the mechanical input unit 214 can be provided to the processor 270 or the control unit 170. The mechanical input unit 214 may be configured in the steering wheel, central instrument panel, center console, cockpit module, door, etc.

[0066] The interior camera 220 can acquire images of the vehicle's interior. The processor 270 can detect the user's state based on the images of the vehicle's interior. The processor 270 can acquire the user's gaze information from the images of the vehicle's interior. The processor 270 can detect the user's gestures from the images of the vehicle's interior.

[0067] The biometric detection unit 230 can acquire a user's biometric information. The biometric detection unit 230 may include sensors capable of acquiring the user's biometric information, such as fingerprints and heart rate data. This biometric information can be used for user authentication.

[0068] The output unit 250 is used to generate outputs related to vision, hearing, or touch. The output unit 250 may include at least one of a display unit 251, a sound output unit 252, and a tactile output unit 253.

[0069] Display unit 251 can display graphic objects corresponding to various information. Display unit 251 may include at least one of liquid crystal display (LCD), thin film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), flexible display, 3D display, and e-ink display.

[0070] The display unit 251 and the touch input unit 213 form a layered structure or are integrated into each other, thereby enabling a touch screen.

[0071] The display unit 251 can be implemented by a HUD (Head-Up Display). When the display unit 251 is implemented by a HUD, the display unit 251 can be provided with a projection module to output information by projecting an image onto a windshield or window.

[0072] Display unit 251 may include a transparent display. The transparent display may be attached to a windshield or window. The transparent display may have a specified transparency and display a specified image. To achieve transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent) display, a transparent OLED (Organic Light-Emitting Diode) display, a transparent LCD (Liquid Crystal Display) display, a transmissive transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display is adjustable.

[0073] On the other hand, the user interface device 200 may include a plurality of display units 251a to 251g.

[0074] The display unit 251 can be configured in an area of ​​the steering wheel, an area 521a, 251b, 251e of the instrument panel, an area 251d of the seat, an area 251f of each pillar, an area 251g of the door, an area of ​​the center console, an area of ​​the headliner, an area of ​​the sun visor, or it can be implemented in an area 251c of the windshield or an area 251h of the window.

[0075] The sound output unit 252 converts the electrical signals provided by the processor 270 or the control unit 170 into audio signals and outputs them. For this purpose, the sound output unit 252 may include more than one speaker.

[0076] The tactile output unit 253 generates tactile output. For example, the tactile output unit 253 can make the user recognize the output by causing the steering wheel, seat belt, seat 110FL, 110FR, 110RL, 110RR to vibrate.

[0077] The processor (hereinafter, may be referred to as the "control unit") 270 can control the overall operation of each unit of the user interface device 200. According to an embodiment, the user interface device 200 may include a plurality of processors 270, or may not include any processors 270.

[0078] If the user interface device 200 does not include the processor 270, the user interface device 200 can operate according to the control of the processor or control unit 170 of other devices in the vehicle 100.

[0079] On the other hand, the user interface device 200 can be named a vehicle display device. The user interface device 200 can operate according to the control of the control unit 170.

[0080] The object detection device 300 is a device for detecting objects located outside the vehicle 100. The objects can be various objects related to the operation of the vehicle 100. (See reference...) Figures 5 to 6 Object O can include lane OB10, other vehicles OB11, pedestrians OB12, two-wheeled vehicles OB13, traffic signals OB14, OB15, light, roads, structures, speed bumps, terrain features, animals, etc.

[0081] Lane OB10 can be a driving lane, a lane adjacent to a driving lane, or a lane for oncoming vehicles. Lane OB10 can also include the concept of the left and right side lines that form the lane.

[0082] Other vehicles OB11 can be vehicles traveling around vehicle 100. Other vehicles can be vehicles located within a specified distance of vehicle 100. For example, other vehicles OB11 can be vehicles that travel before or after vehicle 100.

[0083] Pedestrian OB12 can be a person located around vehicle 100. Pedestrian OB12 can be a person located within a specified distance from vehicle 100. For example, pedestrian OB12 can be a person located on a sidewalk or driveway.

[0084] Two-wheeled vehicles OB12 may be located around vehicle 100 and may refer to passenger vehicles that move using two wheels. Two-wheeled vehicles OB12 may be passenger vehicles with two wheels located within a specified distance from vehicle 100. For example, two-wheeled vehicles OB13 may be motorcycles or bicycles located on sidewalks or driveways.

[0085] Traffic signals may include traffic lights OB15, traffic signs OB14, and patterns or text painted on the road surface.

[0086] Light can be generated by lights installed on other vehicles. Light can be generated by streetlights. Light can be sunlight.

[0087] Roads can include road surfaces, curves, uphill slopes, downhill slopes, and other ramps.

[0088] Structures can be located around roads or can be objects fixed to the ground. For example, structures can include streetlights, trees, buildings, utility poles, traffic lights, and bridges.

[0089] Topographic features can include mountains, hills, etc.

[0090] On the other hand, objects can be classified into moving objects and stationary objects. For example, moving objects can include concepts such as other vehicles and pedestrians. Stationary objects can include concepts such as traffic signals, roads, and structures.

[0091] The object detection device 300 may include a camera 310, a radar 320, a lidar 330, an ultrasonic sensor 340, an infrared sensor 350, and a processor 370.

[0092] According to the embodiments, the object detection device 300 may include other components in addition to the components described, or may exclude some of the components described.

[0093] Camera 310 can be located at an appropriate position outside the vehicle to acquire images of the vehicle's exterior. Camera 310 can be a monocular camera, a binocular camera 310a, an Around View Monitoring (AVM) camera 310b, or a 360-degree camera.

[0094] For example, camera 310 can be positioned inside the vehicle near the windshield to capture an image of the area in front of the vehicle. Alternatively, camera 310 can be positioned around the front bumper or radiator grille.

[0095] For example, camera 310 can be positioned inside the vehicle near the rear window to capture images of the area behind the vehicle. Alternatively, camera 310 can be positioned around the rear bumper, trunk, or tailgate.

[0096] For example, camera 310 may be configured in at least one of the side windows inside the vehicle to acquire images of the side of the vehicle. Alternatively, camera 310 may be configured around a side mirror, fender, or door.

[0097] The camera 310 can provide the acquired images to the processor 370.

[0098] Radar 320 may include an electromagnetic wave transmitter and a receiver. Radar 320 can be implemented using either pulse radar or continuous wave radar, based on the principle of electromagnetic wave transmission. In continuous wave radar, radar 320 can be implemented using either frequency modulated continuous wave (FMCW) or frequency shift keying (FSK) methods, depending on the signal waveform.

[0099] Radar 320 can detect objects using electromagnetic waves as a medium, based on TOF (Time of Flight) or phase-shift methods. It can detect the position of the detected object, the distance between the detected objects, and the relative speed.

[0100] The radar 320 can be configured at appropriate locations on the exterior of a vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0101] The lidar 330 may include a laser emitter and a receiver. The lidar 330 may be implemented in a TOF (Time of Flight) mode or a phase-shift mode.

[0102] The LiDAR 330 can be implemented in either driven or non-driven mode.

[0103] In a driven configuration, the lidar 330 can be rotated by a motor to detect objects around the vehicle 100.

[0104] In a non-driven implementation, the lidar 330 can detect objects within a specified range relative to the vehicle 100 via optical steering. The vehicle 100 may include a plurality of non-driven lidars 330.

[0105] The LiDAR 330 can detect objects using laser as a medium, based on TOF (Time of Flight) or phase-shift methods. It can detect the position of the detected object, the distance between the detected objects, and the relative speed.

[0106] The LiDAR 330 can be configured at appropriate locations on the exterior of a vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0107] The ultrasonic sensor 340 may include an ultrasonic transmitter and a receiver. The ultrasonic sensor 340 can detect objects based on ultrasonic waves, and can detect the position of the detected object, the distance between the detected objects, and the relative speed.

[0108] The ultrasonic sensor 340 can be configured at an appropriate location on the exterior of the vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0109] Infrared sensor 350 may include an infrared emitter and an infrared receiver. Infrared sensor 340 can detect objects based on infrared light, and can detect the position of the detected object, the distance between the detected objects, and the relative speed.

[0110] The infrared sensor 350 can be configured at an appropriate location on the exterior of the vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0111] The processor 370 can control the overall operation of each unit of the object detection device 300.

[0112] Processor 370 can detect and track objects based on the acquired images. Processor 370 can perform actions such as distance calculation and relative velocity calculation between itself and the object using image processing algorithms.

[0113] Processor 370 can detect and track objects based on the electromagnetic waves reflected back from them. Processor 370 can also perform actions such as distance calculation and relative velocity calculation based on these electromagnetic waves.

[0114] Processor 370 can detect and track objects based on the reflected laser light from the emitted laser. Processor 370 can also perform actions such as distance calculation and relative velocity calculation based on the laser light.

[0115] Processor 370 can detect and track objects based on reflected ultrasonic waves from emitted ultrasonic waves. Processor 370 can also perform actions such as distance calculation and relative speed calculation based on these ultrasonic waves.

[0116] Processor 370 can detect and track objects based on the reflected infrared light from the emitted infrared light. Processor 370 can also perform actions such as distance calculation and relative speed calculation based on the infrared light.

[0117] According to an embodiment, the object detection device 300 may include a plurality of processors 370, or may not include processors 370. For example, the camera 310, radar 320, lidar 330, ultrasonic sensor 340, and infrared sensor 350 may each include a processor individually.

[0118] If the object detection device 300 does not include the processor 370, the object detection device 300 can operate according to the control of the processor or the control unit 170 of the device inside the vehicle 100.

[0119] The object detection device 400 can operate under the control of the control unit 170.

[0120] The communication device 400 is a device for performing communication with external devices. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0121] The communication device 400 may include at least one of a transmitting antenna, a receiving antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0122] The communication device 400 may include a short-range communication unit 410, a location information unit 420, a V2X communication unit 430, an optical communication unit 440, a broadcast transceiver unit 450, and a processor 470.

[0123] According to the embodiments, the communication device 400 may include other components in addition to the components described, or may exclude some of the components described.

[0124] The short-range communication unit 410 is a unit for short-range communication. The short-range communication unit 410 can support short-range communication using at least one of the following technologies: Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi, Wi-Fi Direct, and Wireless Universal Serial Bus.

[0125] The short-range communication unit 410 can form a short-range wireless communication network to perform short-range communication between the vehicle 100 and at least one external device.

[0126] The location information unit 420 is a unit used to acquire the location information of the vehicle 100. For example, the location information unit 420 may include a Global Positioning System (GPS) module or a Differential Global Positioning System (DGPS) module.

[0127] The V2X communication unit 430 is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), other vehicles (V2V: Vehicle to Vehicle), or pedestrians (V2P: Vehicle to Pedestrian). The V2X communication unit 430 may include RF circuitry capable of implementing communication protocols with infrastructure (V2I), vehicle-to-vehicle (V2V), and pedestrian-to-pedestrian (V2P).

[0128] The optical communication unit 440 is a unit for performing communication with external devices using light as a medium. The optical communication unit 440 may include: an optical transmitter that converts electrical signals into optical signals and transmits them to the outside; and an optical receiver that converts received optical signals into electrical signals.

[0129] According to an embodiment, the light emitting part can be integrated with the lamps included in the vehicle 100.

[0130] The broadcast transceiver unit 450 is a unit used to receive broadcast signals from an external broadcast management server or to send broadcast signals to the broadcast management server via a broadcast channel. The broadcast channel may include a satellite channel or a terrestrial channel. The broadcast signal may include TV broadcast signals, radio broadcast signals, or data broadcast signals.

[0131] The processor 470 can control the overall operation of each unit of the communication device 400.

[0132] According to an embodiment, the communication device 400 may include a plurality of processors 470, or may not include processors 470.

[0133] In the absence of a processor 470, the communication device 400 may operate under the control of a processor or control unit 170 of other devices within the vehicle 100.

[0134] On the other hand, the communication device 400 can be used together with the user interface device 200 to implement a vehicle display device. In this case, the vehicle display device can be named a telematics device or an audio-visual navigation (AVN) device.

[0135] The communication device 400 can operate under the control of the control unit 170.

[0136] The driving control device 500 is a device that receives user input for driving.

[0137] In manual mode, vehicle 100 can operate based on signals provided by driving control device 500.

[0138] The driving control device 500 may include a steering input device 510, an acceleration input device 530, and a braking input device 570.

[0139] The steering input device 510 can receive the driving direction input of the vehicle 100 from the user. The steering input device 510 is preferably configured in the form of a steering wheel so that steering input can be achieved by rotation. According to an embodiment, the steering input device may also be configured as a touch screen, touchpad, or button.

[0140] The accelerator input device 530 can receive input from the user for accelerating the vehicle 100. The brake input device 570 can receive input from the user for decelerating the vehicle 100. The accelerator input device 530 and the brake input device 570 are preferably configured as pedals. According to an embodiment, the accelerator input device or the brake input device may also be configured as a touchscreen, touchpad, or button.

[0141] The driving control device 500 can operate according to the control unit 170.

[0142] The vehicle drive unit 600 is a device that drives various devices within the electrically controlled vehicle 100.

[0143] The vehicle drive unit 600 may include a powertrain drive unit 610, a chassis drive unit 620, a door / window drive unit 630, a safety device drive unit 640, a light drive unit 650, and an air conditioning drive unit 660.

[0144] According to the embodiments, the vehicle drive unit 600 may include other components in addition to the components described, or may exclude some of the components described.

[0145] On the other hand, the vehicle drive unit 600 may include a processor. Each unit of the vehicle drive unit 600 may individually include a processor.

[0146] The powertrain drive unit 610 can control the operation of the powertrain unit.

[0147] The powertrain drive unit 610 may include a power source drive unit 611 and a transmission drive unit 612.

[0148] The power source drive unit 611 can perform control of the power source of the vehicle 100.

[0149] For example, when a fossil fuel-based engine is used as the power source, the power source drive unit 610 can perform electronic control of the engine. This allows for control of the engine's output torque, etc. The power source drive unit 611 can adjust the engine's output torque according to the control unit 170.

[0150] For example, when an electric motor is used as the power source, the power source drive unit 610 can control the motor. The power source drive unit 610 can adjust the motor's rotational speed, torque, etc., according to the control unit 170.

[0151] The transmission drive unit 612 can perform control of the transmission. The transmission drive unit 612 can adjust the state of the transmission. The transmission drive unit 612 can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or park (P).

[0152] On the other hand, when the engine is the power source, the transmission drive unit 612 can adjust the gear engagement state in the forward (D) state.

[0153] The chassis drive unit 620 can control the movement of the chassis assembly. The chassis drive unit 620 may include a steering drive unit 621, a braking drive unit 622, and a suspension drive unit 623.

[0154] The steering drive unit 621 can perform electronic control of the steering apparatus within the vehicle 100. The steering drive unit 621 can change the direction of travel of the vehicle.

[0155] The brake drive unit 622 can perform electronic control of the brake apparatus within the vehicle 100. For example, the speed of the vehicle 100 can be reduced by controlling the braking action of the brakes configured on the wheels.

[0156] On the other hand, the brake drive unit 622 can individually control each of the plurality of brakes. The brake drive unit 622 can control the braking force applied to the plurality of wheels differently.

[0157] The suspension drive unit 623 can perform electronic control of the suspension apparatus within the vehicle 100. For example, the suspension drive unit 623 can control the suspension apparatus to reduce vibration of the vehicle 100 when the road surface is uneven. On the other hand, the suspension drive unit 623 can individually control each of the plurality of suspensions.

[0158] The door / window drive unit 630 can perform electronic control of the door apparatus or window apparatus inside the vehicle 100.

[0159] The door / window drive unit 630 may include a door drive unit 631 and a window drive unit 632.

[0160] The door drive unit 631 can control the door mechanism. The door drive unit 631 can control the opening and closing of a plurality of doors included in the vehicle 100. The door drive unit 631 can control the opening and closing of the trunk or tailgate. The door drive unit 631 can control the opening and closing of the sunroof.

[0161] The window drive unit 632 can perform electronic control of the window apparatus. It can control the opening or closing of a plurality of windows included in the vehicle 100.

[0162] The safety device drive unit 640 can perform electronic control of various safety devices within the vehicle 100.

[0163] The safety device drive unit 640 may include an airbag drive unit 641, a seat belt drive unit 642, and a pedestrian protection device drive unit 643.

[0164] The airbag actuator 641 can perform electronic control of the airbag apparatus within the vehicle 100. For example, the airbag actuator 641 can control the deployment of the airbag when a hazard is detected.

[0165] The seatbelt drive unit 642 can perform electronic control of the seatbelt apparatus within the vehicle 100. For example, the seatbelt drive unit 642 can control the use of seatbelts to secure passengers to seats 110FL, 110FR, 110RL, and 110RR when a hazard is detected.

[0166] The pedestrian protection device drive unit 643 can perform electronic control of the hood lifting mechanism and the pedestrian airbag. For example, the pedestrian protection device drive unit 643 can control the hood lifting mechanism to lift and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0167] The lamp drive unit 650 can perform electronic control of various lamp apparatuses within the vehicle 100.

[0168] The air conditioning drive unit 660 can perform electronic control of the air conditioning unit within the vehicle 100. For example, when the temperature inside the vehicle is high, the air conditioning drive unit 660 can control the air conditioning unit to operate and supply cool air to the vehicle interior.

[0169] The vehicle drive unit 600 may include a processor. Each unit of the vehicle drive unit 600 may individually include a processor.

[0170] The vehicle drive unit 600 can operate according to the control of the control unit 170.

[0171] The operating system 700 is a system that controls various operations of the vehicle 100. The operating system 700 can operate in automatic driving mode.

[0172] The operating system 700 may include a driving system 710, a vehicle dispatching system 740, and a parking system 750.

[0173] According to the embodiments, the operating system 700 may include other components in addition to the components described, or may exclude some of the components described.

[0174] On the other hand, the operating system 700 may include a processor. Each unit of the operating system 700 may individually include a processor.

[0175] On the other hand, according to the embodiment, when the running system 700 is implemented as software, it can also be a subordinate concept of the control unit 170.

[0176] On the other hand, according to the embodiment, the operating system 700 may be a concept including at least one of the following: user interface device 200, object detection device 300, communication device 400, vehicle drive device 600, and control unit 170.

[0177] The driving system 710 can drive the vehicle 100.

[0178] The driving system 710 can receive navigation information from the navigation system 770 and provide control signals to the vehicle drive unit 600, thereby driving the vehicle 100. The driving system 710 can also receive object information from the object detection device 300 and provide control signals to the vehicle drive unit 600, thereby driving the vehicle 100. Furthermore, the driving system 710 can receive signals from external devices via the communication device 400 and provide control signals to the vehicle drive unit 600, thereby driving the vehicle 100.

[0179] The vehicle dispatch system 740 can dispatch vehicle 100.

[0180] The vehicle dispatch system 740 can receive navigation information from the navigation system 770 and provide control signals to the vehicle drive unit 600, thereby dispatching the vehicle 100. The vehicle dispatch system 740 can also receive object information from the object detection device 300 and provide control signals to the vehicle drive unit 600, thereby dispatching the vehicle 100. Furthermore, the vehicle dispatch system 740 can receive signals from external devices via the communication device 400 and provide control signals to the vehicle drive unit 600, thereby dispatching the vehicle 100.

[0181] Parking system 750 can park 100 vehicles.

[0182] The parking system 750 can receive navigation information from the navigation system 770 and provide control signals to the vehicle drive unit 600 to park the vehicle 100. The parking system 750 can also receive object information from the object detection device 300 and provide control signals to the vehicle drive unit 600 to park the vehicle 100. Furthermore, the parking system 750 can receive signals from external devices via the communication device 400 and provide control signals to the vehicle drive unit 600 to park the vehicle 100.

[0183] The navigation system 770 can provide navigation information. The navigation information may include at least one of the following: map information, set destination information, route information set according to the destination, information about various objects on the route, lane information, and the vehicle's current location information.

[0184] The navigation system 770 may include a memory and a processor. The memory can store navigation information. The processor can control the operation of the navigation system 770.

[0185] According to an embodiment, the navigation system 770 can receive information from an external device via the communication device 400, thereby updating the stored information.

[0186] According to the embodiments, the navigation system 770 can also be classified as a subordinate component of the user interface device 200.

[0187] The sensing unit 120 can sense the state of the vehicle. The sensing unit 120 may include posture sensors (e.g., yaw sensor, roll sensor, pitch sensor), collision sensors, wheel sensors, speed sensors, tilt sensors, weight detection sensors, heading sensors, yaw sensors, gyroscope sensors, position modules, vehicle forward / reverse sensors, battery sensors, fuel sensors, tire sensors, steering sensors based on steering wheel rotation, vehicle interior temperature sensors, vehicle interior humidity sensors, ultrasonic sensors, illuminance sensors, accelerator pedal position sensors, brake pedal position sensors, etc.

[0188] The sensing unit 120 can acquire sensing signals regarding vehicle posture information, vehicle collision information, vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery information, fuel information, tire information, vehicle light information, vehicle interior temperature information, vehicle interior humidity information, steering wheel rotation angle, vehicle exterior illuminance, pressure applied to the accelerator pedal, and pressure applied to the brake pedal.

[0189] In addition, the sensing unit 120 may also include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a coolant temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crankshaft angle sensor (CAS), etc.

[0190] The vehicle interface unit 130 can function as a conduit for various types of external devices connected to the vehicle 100. For example, the vehicle interface unit 130 may be provided with a port that can connect to a mobile terminal, through which the vehicle interface unit 130 can exchange data with the mobile terminal.

[0191] On the other hand, the vehicle interface unit 130 can function as a pathway to supply power to the connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit 130, the vehicle interface unit 130 can supply power from the power supply unit 190 to the mobile terminal under the control of the control unit 170.

[0192] The memory 140 is electrically connected to the control unit 170. The memory 140 can store basic data about the unit, control data for controlling the unit's operation, and input and output data. The memory 140 can be a various storage device such as ROM, RAM, EPROM, flash memory drive, or hard disk. The memory 140 can store programs for processing or controlling the control unit 170, and various data related to the overall operation of the vehicle 100.

[0193] According to the embodiment, the memory 140 may be integrated with the control unit 170, or it may be implemented as a subordinate component of the control unit 170.

[0194] The control unit 170 can control the overall operation of various units within the vehicle 100. The control unit 170 can be named an electronic control unit (ECU).

[0195] The power supply unit 190 can supply the power required for the operation of each component according to the control of the control unit 170. In particular, the power supply unit 190 can receive power from the battery or the like inside the vehicle.

[0196] The vehicle 100 includes one or more processors and control units 170 that can be implemented using at least one of application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0197] On the other hand, in embodiments of the present invention, "driver," "passenger," or "driver or passenger (driver / passenger)" refers to a driver, passenger, co-passenger, or other person riding in a vehicle capable of performing personalized driving control according to embodiments of the present invention, and may be one or more. Additionally, in this specification, "driver" includes a passenger sitting in the driver's seat when the vehicle is performing autonomous driving.

[0198] In addition, in embodiments of the present invention, "vehicle" is used as a vehicle that operates according to the driver's operation, a vehicle capable of performing autonomous driving, or a vehicle capable of providing such a service.

[0199] On the other hand, such as Figure 7 As shown, the vital signs data integration device 800 of an embodiment of the present invention can be provided within a vehicle 100.

[0200] In some embodiments, the vital signs data integration device 800 may be in the form of a service platform providing vital signs data integration services. In this case, the vehicle 100 can receive vital signs data integration services from the corresponding service platform via a network or the like.

[0201] The vital signs data integration device 800 receives vital signs data of a plurality of drivers / passengers collected by each of a plurality of cameras located at different positions within the vehicle 100. The vital signs data integration device 800 can receive a plurality of time-synchronized vital signs data in a single stream.

[0202] The vital sign data integration device 800 can evaluate the reliability of each of the received multiple vital sign data points according to a preset benchmark, thereby assigning different weights to each vital sign data point. The preset benchmark can be determined based on one or more of the following: the performance of the multiple cameras, the results of the facial recognition algorithm for each of the multiple cameras, and the performance of the state estimation algorithm to be applied.

[0203] The vital signs data integration device 800 can filter and fuse vital signs data based on assigned weights to ultimately generate input data for high-level data analysis. Here, the input data for high-level data analysis may refer to the input values ​​of an algorithm that estimates the state index of the driver / passenger.

[0204] The vital signs data integration device 800 can provide the final generated input data to the driver / passenger state estimation algorithm, thereby obtaining a more reliable state index result. Here, the driver / passenger state estimation algorithm may include, for example, a drowsiness estimation algorithm, a stress assessment algorithm, an emergency state estimation (assessment) algorithm, etc.

[0205] As described above, the vital signs data integration device 800 can receive multiple vital signs data from various angles for the driver / passenger riding in the vehicle 100, evaluate the reliability and assign weights, and filter and fuse the vital signs data based on this, thereby providing a more reliable input value to the state estimation algorithm.

[0206] Figure 8 This is a block diagram illustrating the state in which the vital signs data integration device 800 interacts with other components of the vehicle in relation to embodiments of the present invention.

[0207] The vital signs data integration device 800 can communicate with multiple cameras 220, motion detection module 235, vehicle control unit 170, and ADAS system 300 within the vehicle. The vital signs data integration device 800 communicates with them and acquires multiple vital signs data for the driver / passenger. Furthermore, when performing a reliability evaluation of the acquired multiple vital signs data, the vital signs data integration device 800 communicates with them and reflects this evaluation in the reliability assessment.

[0208] Apart from Figure 8In addition to the configuration shown, the vital signs data integration device 800 can also communicate with other components within the vehicle. Furthermore, the vital signs integration device 880 can receive network data from external sources.

[0209] The vital signs data integration device 800 can receive different vital signs data from multiple cameras 220 inside the vehicle, such as the first camera 221, the second camera 222, and the third camera 223.

[0210] The plurality of cameras 220 can be positioned at different locations within the vehicle. For example, the first camera 221 could be an IR camera positioned on the rearview mirror, and the second camera 222 could be an A-pillar camera. Additionally, the third camera 223 could be an IR camera positioned on the dashboard. However, the plurality of cameras 220 is not limited to this example and can be positioned at other locations within the vehicle 100, potentially including more than three cameras.

[0211] Each of the plurality of cameras 220 acquires vital sign data of the driver / passenger in a remote PPG (rPPG) manner. For example, each of the plurality of cameras 220 can acquire a facial image of the driver / passenger. Furthermore, heart rate data of the driver / passenger can be collected in a remote PPG manner based on changes in blood flow measured by applying a facial recognition algorithm to each of the acquired facial images. It should be noted that the plurality of vital sign data acquired by each of the plurality of cameras 220 are heart rate data for the same person.

[0212] The vital signs data integration device 800 can receive detection results and / or control results from at least one of the motion detection module 235, ADAS system 300 and vehicle control unit 170, and perform different reliability evaluations on the vital signs data based on these results.

[0213] The reliability evaluation of vital sign data is performed quantitatively, taking into account factors such as accuracy, timeliness, and consistency of each vital sign data point. For example, if the first camera 221 provides data with high accuracy but low timeliness, the second camera 222 provides data with slightly lower accuracy but high timeliness, and the third camera 223 provides data with average accuracy and timeliness, then the reliability evaluation results for each vital sign data point collected by them can be assigned different weights.

[0214] On the other hand, the vital signs data integration device 800 can reflect the detection results and / or control results of at least one of the motion detection module 235, ADAS system 300 and vehicle control unit 170 in the vehicle 100 to the reliability evaluation.

[0215] In this embodiment, the vital signs data integration device 800 can change the weight of the received vital signs data based on the detection results and / or control results received from the motion detection module 235, the ADAS system 300, and / or the vehicle control unit 170.

[0216] For example, if an event is detected by the ADAS system 300 and emergency braking is initiated without driver intervention, the driver / passenger's heart rate may increase, and therefore, the vital signs data can be temporarily given higher weight.

[0217] As another embodiment, the vital signs data integration device 800 can change the weight calculation cycle to be assigned to the received vital signs data based on the detection results and / or control results received from the motion detection module 235, the ADAS system 300 and / or the vehicle control unit 170.

[0218] For example, if the control results of the vehicle control unit 170 remain the same or only change slightly over a long period of time, the driver's heart rate may decrease due to drowsiness. Therefore, the weight calculation cycle for vital sign data can be shortened and an anti-drowsiness service response can be executed.

[0219] As another embodiment, the vital signs data integration device 800 can adjust the degree of weight change or algorithm sensitivity of vital signs data based on the detection results and / or control results received from the motion detection module 235, the ADAS system 300 and / or the vehicle control unit 170.

[0220] For example, based on the detection results of the motion detection module 235, during periods when a large amount of movement by the driver / passenger is detected, the heart rate may increase due to the driver / passenger's physical movement. Therefore, at this time, the overall weight change of the vital signs data or the sensitivity of the algorithm can be reduced and applied.

[0221] On the other hand, in some embodiments, based on the vehicle 100, in addition to Figure 8 The reliability evaluation of the vital signs data from the vital signs data integration device 800 may differ depending on the detection results and / or control results from components other than those within the vehicle 100, or network data provided from outside the vehicle 100. For example, the reliability evaluation of the current driver / passenger's vital signs data may differ based on traffic information, weather information, health status information received from the driver / passenger's terminal, etc.

[0222] As described above, if a reliability evaluation result is assigned a weight, the final data generated by filtering and fusing the weighted vital sign data is then input into an algorithm for estimating the driver / passenger's condition. In this case, the final generated data is reliable heart rate data, thus enabling a more accurate estimation of the driver / passenger's condition.

[0223] the following, Figure 9 This is a block diagram illustrating the detailed configuration of the vital signs data integration device 800 related to embodiments of the present invention.

[0224] Reference Figure 9 The vital signs data integration device 800 of the present invention may include a receiving unit 810, a data determining unit 820, and a data providing unit 830. Here, the data determining unit 820 may refer to the same meaning as the control unit / processor. In addition, the data providing unit 830 may be a transmitting unit / transceiver for transmitting the final generated data to the driver / passenger's state estimation algorithm.

[0225] The receiving unit 810 can receive heart rate data based on rPPG (Remote PPG) measured from facial images acquired by a plurality of cameras located at different positions within the vehicle 100 as vital sign data.

[0226] The received vital signs data consists of multiple heart rate data sensed from the same person. For example, multiple cameras positioned at different locations within the vehicle 100 may be used to acquire vital signs data of the same person using rPPG (Remote PPG), but this method of data acquisition is not the only one.

[0227] The vital sign data received by the receiving unit 810 have different values ​​and precision. The receiving unit 810 provides each vital sign data to the data determination unit 820, which performs data filtering and integration to provide a highly reliable driver status assessment service.

[0228] The data determination unit 820 evaluates the reliability of the received vital sign data and assigns weights accordingly. Furthermore, the data determination unit 820 filters for highly reliable vital sign data or merges multiple vital sign data based on the weights assigned to each vital sign data point.

[0229] The data determination unit 820 can evaluate the reliability of each vital sign data based on landmark information contained in the driver / passenger's facial image, logical reliability, changes compared to previous data, and the reliability of each of the multiple cameras. The reliability evaluation result is represented by a weighted value.

[0230] The data determination unit 820 can filter or fuse one or more of the received vital sign data based on landmark information contained in the driver / passenger's facial image, logical reliability, the amount of change compared to previous data, and the reliability of each of the multiple cameras. For example, it can remove vital sign data with outliers or ignore missing vital sign data.

[0231] The data providing unit 830 receives screened or fused vital sign data from the data determining unit 820.

[0232] The data providing unit 830 provides the final generated vital sign data to the state index estimation algorithm 840 for high-level vital sign analysis, thereby enabling the estimation of the driver / passenger's state index based on the filtered or fused vital sign data. Here, the driver / passenger's state index can refer to one of the driver / passenger's stress index, drowsiness index, or fatigue index.

[0233] The state index estimation (840) uses the final generated vital sign data as input data for each state estimation algorithm to perform evaluations of driver / passenger fatigue index, stress index, and whether an emergency situation is underway. Based on the calculated fatigue index, stress index, and emergency situation evaluation, various actions and services related to the driving and safety of the vehicle 100 can be provided to the vehicle 100.

[0234] Figure 10 This is a representative flowchart illustrating a vital signs data integration method related to embodiments of the present invention.

[0235] Figure 10 The steps shown can respond to instructions executed by a processor of a computing device, in which case the vital signs data integration method can be implemented in the form of a computer program recording medium including a downloadable computer program or instructions.

[0236] Unless otherwise stated, Figure 10 The steps of the flowchart are executed by the aforementioned vital sign data integration device 800, more specifically, by... Figure 9 The receiving unit 810, the data determining unit 820, and the data providing unit 830 described herein are executed.

[0237] The vital signs data integration device 800 receives vital signs data sensed by a plurality of cameras installed in the vehicle (S10).

[0238] Multiple cameras installed inside the vehicle each acquire the driver's / passenger's heart rate as vital sign data using remote photoplethysmography (rPPG). rPPG is a non-contact method for acquiring biological signals. Specifically, it extracts skin pixels from facial images captured by cameras and then analyzes changes in skin color caused by blood flow during heartbeats to obtain heart rate data.

[0239] As described above, if vital sign data is received, the vital sign data integration device 800 evaluates the reliability of the received vital sign data and assigns a weight (S20).

[0240] In an embodiment, step S20 of assigning the weights may include: evaluating the reliability of each vital sign data based on at least one of the following: the reliability of each of the plurality of cameras in the vehicle, key point information of the images sensed by the plurality of cameras, the area occupied by the face in each image, and the amount of change compared to previous vital sign data, and determining different weights for each of them.

[0241] For example, each camera can be assigned a different weight based on the accuracy of the facial recognition algorithm used in each of the multiple cameras. Alternatively, for example, higher weight can be assigned to data based on the presence of more key points or a larger occupant area in the acquired facial image. Or, for example, higher weight can be assigned to data where the angle between the face and the camera is close to a right angle. Additionally, for example, higher weight can be assigned to data with high accuracy values ​​in the facial recognition algorithm's results. Furthermore, for example, higher weight can be assigned to data with high continuity and similarity by comparing the amount of change with previous vital sign data.

[0242] In an embodiment, step S20 of assigning the weights may include: further adjusting the reliability weights of each vital sign data based on the vehicle's control information and the passenger's motion information.

[0243] Here, vehicle control information refers to the control information used to control the vehicle's braking and steering. In this context, vehicle braking and steering include all actions performed by the driver, including emergency braking / accelerated acceleration / sharp steering, without driver intervention, both through driver input and through the actions of the ADAS system.

[0244] In addition, the passenger motion information here refers to the motion monitoring results of the driver and passengers obtained based on the motion monitoring results of the camera-based motion monitoring module.

[0245] As described above, after assigning different weights to the vital signs data, the vital signs data integration device 800 filters or merges the vital signs data based on the assigned weights (S30).

[0246] The screening and fusion of vital sign data can be carried out through one of the following methods: 1) data fusion, 2) soft voting, or 3) a combination of soft voting and data fusion.

[0247] 1) The data fusion method involves removing outlier data from a plurality of sensed vital sign data points, then integrating the values ​​of each vital sign data point into a single data stream and providing it. Here, outlier data refers to values ​​that differ from all other data or have a large amount of variation.

[0248] 2) Soft voting averages the probability values ​​of each sensed vital sign and calculates a weighted probability for each. This method ultimately selects the label with the highest probability from the calculated multiple values.

[0249] 3) The fusion of soft voting and data fusion is a method that adds data fusion after the aforementioned soft voting. Specifically, it involves averaging the probability values ​​of each sensed vital sign data, calculating their respective weighted probabilities, and then further performing a data fusion method that uses each value as a basis to obtain a weighted average.

[0250] The vital signs data integration device 800 filters and merges vital signs data through the various methods described above, and generates reliable final vital signs data.

[0251] Then, the vital signs data integration device 800 can provide the final generated vital signs data as a state index estimate (840, Figure 9 The input data is used to estimate the driver's state index (S40).

[0252] At this point, the final generated vital sign data can be provided simultaneously or sequentially to a plurality of different state index algorithms, thereby providing various state indices for the same person as result values.

[0253] In some embodiments, the vital signs data integration device 800 can provide the final generated vital signs data to devices / servers / clouds, including various state index algorithms. In this case, the state index estimation (840) can be said to indicate that the vital signs data integration device 800 exists externally. Subsequently, the state index result value can be sent to the vital signs data integration device 800 or the vehicle 100 for tracking the driver / passenger's status and / or providing related services.

[0254] As described above, the vital sign data integration method according to embodiments of the present invention can provide uninterrupted and stable data aggregation for driver / passenger status analysis and tracking. In other words, it can generate and provide reliable input data for high-level vital sign analysis.

[0255] Figure 11 This is a diagram illustrating the collection of vital signs data using multiple cameras inside a vehicle, in relation to embodiments of the present invention.

[0256] Reference Figure 11 Taking the acquisition of driver's vital signs data using three cameras around the driver's seat inside the vehicle 100 as an example. As shown in the figure, the plurality of cameras may include an IR camera (first camera) 221 set in the rearview mirror, an A-pillar camera (second camera) 222 on the side of the driver's seat window, and an IR camera (third camera) 223 set in the dashboard.

[0257] but, Figure 11 The number and location of the cameras shown can be varied. However, acquiring multiple vital sign data points for the same person using multiple cameras is equally applicable for filtering and fusion.

[0258] On the other hand, each of the plurality of cameras 221, 222, 223 acquires vital sign data for the driver / passenger in the form of rPPG (Remote PPG). However, it is not limited to this method; other methods can also be used to acquire camera-based vital sign data.

[0259] exist Figure 11 In this system, each of a plurality of cameras 221, 222, and 223 is positioned around the driver's seat inside the vehicle to sense the driver's vital signs data using a remote PPG (rPPG) method. Each of the plurality of cameras 221, 222, and 223 senses different vital signs data from each other at the same time point (or with short time intervals).

[0260] Specifically, each of the plurality of cameras 221, 222, and 223 senses heart rate data based on changes in blood flow from an image of the driver's / passenger's face in the driver's / passenger's seat inside the vehicle and sends it to the vehicle's vital signs data integration unit 800. Each of the sensed vital signs data has a different value and a different level of precision than the others.

[0261] The sensed multiple vital sign data are received in the receiving unit 810 of the vital sign data integration device 800. Figure 9Time synchronization is performed and transmitted to the data determination unit 820.

[0262] As mentioned above, multiple vital sign data sensed at the same time point have different values ​​and precision. Therefore, in order to provide a highly reliable driver condition assessment service, data filtering and fusion (integration) are required.

[0263] Figure 12 This is an example diagram illustrating a hard voting method for data filtering related to embodiments of the present invention. Hard voting is a method of selecting the result that receives the most votes among the various models, and is therefore also known as majority voting.

[0264] Reference Figure 12 If a new driver state factor (new instance) 1200 is input to each of a plurality of devices sensing the driver's state, such as a plurality of cameras 1201, 1202, 1203, 1204, then the prediction of the new factor is made by the senses of each of the plurality of cameras 1201, 1202, 1203, 1204. As a result, assuming the sensing result of three cameras 1201, 1202, 1204 is "1", and the sensing result of a particular camera 1203 is predicted to be "2", which is different from it, then, according to majority voting, i.e., ensemble prediction, "1" is selected as the final data 1210.

[0265] On the other hand, the soft voting method used in this invention calculates more accurate and flexible results than the hard voting method described above. Soft voting is also known as probability voting. Specifically, soft voting does not ignore... Figure 12 The sensing results of a specific camera 1203 in the example are given as probability values.

[0266] Specifically, suppose there are three camera sensors measuring the driver's heart rate, each measuring the driver's heart rate once per second. In this case, the soft voting method determines the driver's heart rate using the process described above.

[0267] Each of the three camera sensors transmits measured values ​​and time information together to ensure reliability. The vital signs data integration device 800 synchronizes the heart rate data transmitted from each camera in time and integrates them into a single data stream.

[0268] For example, suppose camera A measures 80 bpm at 10:00:01, camera B measures 82 bpm at 10:00:02, and camera C measures 78 bpm at 10:00:03. The vital signs data integration device 800 converts these into a single data stream and performs the work of removing previously missed or outlier values.

[0269] Specifically, the data determination unit 820 of the vital signs data integration device 800 evaluates the reliability of the normal vital signs data after removing outliers from a plurality of vital signs data, and assigns weights to each. At this time, the reliability evaluation is quantitatively calculated considering factors such as the accuracy, timeliness, and consistency of the data.

[0270] For example, in the example above, if camera A provides data with high accuracy but low timeliness, camera sensor B provides data with low accuracy but high timeliness, and camera C provides data with average accuracy and timeliness, then they can be assigned different weights.

[0271] As described above, after assigning different weights to each other, the vital sign data integration device 800 can also select the best data or average value (data fusion) based on the reliability of each vital sign data, combine them (fusion) and calculate the weighted probability value, and then select the label with the highest probability value (soft voting), or it can filter its average value (soft voting + data fusion).

[0272] As mentioned above, the final, reliable data is provided to the driver / passenger state index estimate (840, Figure 9 This allows us to obtain more accurate and reliable state index results.

[0273] Figure 13 and Figure 14 This is a diagram illustrating an example of assigning different weights to vital sign data based on a facial recognition algorithm, in relation to embodiments of the present invention.

[0274] In embodiments of the present invention, the reliability evaluation of vital sign data (e.g., heart rate data) received from a plurality of cameras is performed to determine the weight of each vital sign data based on at least one of the following: the reliability of each of the plurality of cameras, key point information of the image sensed by the plurality of cameras, the area occupied by the face in the image, the reliability of the logic, and the amount of change compared to previous vital sign data.

[0275] Although the vital signs data sensed by each of the multiple cameras are acquired at the same point in time, they have different values ​​and precision. Therefore, in order to provide a highly reliable driver condition assessment service, data filtering and fusion (integration) are required.

[0276] As a data processing method for filtering and fusing weighted vital sign data, this invention uses data fusion and soft voting methods.

[0277] Data fusion is a data processing method that integrates the values ​​obtained by applying weights to the results of facial recognition algorithms applied to facial images sensed by multiple cameras into a single data stream, thereby obtaining more accurate and flexible results.

[0278] The soft voting method involves applying weights to the results of facial recognition algorithms on facial images sensed by multiple cameras, averaging the probability values, and finally selecting the data processing method with the highest probability label.

[0279] Before data filtering and fusion, the weights for each vital sign data point are assigned as follows. Assigning weights to each vital sign data point begins with determining which factor to assign the weight to. The following will illustrate specific examples of assigning weights based on various factors.

[0280] 1) Cases where the application results of facial recognition algorithms are the key factors: As one embodiment, weights can be assigned to images from each of a plurality of cameras that are at approximately a right angle to the driver's / passenger's face.

[0281] In deep learning algorithms, facial recognition algorithms can determine the angle of the face during operation. Based on the angle of the driver / passenger's face, the size of the landmark and the skin color changes caused by blood flow variations, which can be used to measure heart rate (HR), are influenced. For this reason, weights are assigned to cameras that measure the driver / passenger's face at near-right angles.

[0282] For example, refer to Figure 13 Each of the multiple cameras installed inside the vehicle 100 has a different shooting angle at each sensing time point based on the driver's / passenger's head movements, etc. Therefore, as... Figure 13 As shown, facial images captured from various shooting angles are obtained at each sensing time point. Figure 13 In the image 1310, the angle between the camera's shooting angle and the driver's / passenger's face is close to a right angle.

[0283] As another embodiment, images with the largest and most vivid landmarks in facial images acquired through multiple cameras can be given high weights.

[0284] In deep learning algorithms, deriving facial landmarks and determining their area are crucial for face recognition. This is because the size of these landmarks significantly impacts the performance of the algorithm after face recognition.

[0285] Facial landmarks refer to the feature points of the face (e.g., eyes, nose, mouth, eyebrows, etc.). If images are captured by multiple cameras, face detection algorithms can be used to detect the driver's / passenger's face, and then facial landmark algorithms can be used to detect the key points within the face (e.g., eyes, nose, mouth, eyebrows, etc.).

[0286] Reference Figure 14 The application of facial landmark algorithms confirms the detection of key points such as eyes, eyebrows, and lips in facial images. In multiple images captured by multiple cameras at the same time point, such as... Figure 14 As shown, weights are assigned to use the image with the largest and most prominent landmarks on the driver's / occupant's face as the primary image.

[0287] As another embodiment, weights can be assigned to images in which facial landmarks appear most frequently in images acquired by multiple cameras.

[0288] As another embodiment, weights can be assigned to images where the driver / occupant occupies a larger area in images acquired by multiple cameras.

[0289] Specifically, this is a method of assigning weights to the larger areas of the driver / occupant's face and body before applying deep learning algorithms and before detecting keypoints. While the keypoints become narrower when the driver / occupant's face is obscured by hats, sunglasses, etc., this method still assigns weights to the larger areas of the driver / occupant in the actual image. The difference is that it doesn't use facial keypoints, but rather utilizes most of the driver / occupant's outline.

[0290] As another embodiment, weights can be assigned to cameras running algorithms with higher accuracy among a plurality of cameras.

[0291] Each algorithm running across multiple cameras may exhibit performance differences. Furthermore, even the same rPPG algorithm can be optimized differently depending on the camera's location, performance, and type, leading to performance variations. For example, if camera A's algorithm has 95% accuracy while camera B's algorithm has 75% accuracy, camera A can be given a higher weight.

[0292] As another embodiment, weights can be assigned to result values ​​with higher accuracy values ​​among the result values ​​of the same algorithm.

[0293] When the algorithm for measuring vital signs data operates, it outputs the accuracy of the data at which the currently measured vital signs data was obtained, as a result value. Therefore, by assigning weights based on the accuracy of this algorithmic result value, it can be applied to the fusion and filtering of vital signs data.

[0294] As another embodiment, among a plurality of vital sign data, data with high continuity and similarity to previous vital sign data values ​​can be assigned weights.

[0295] Human vital sign data exhibits continuity. That is, if a specific index in the vital sign data increases or decreases, it tends to increase or decrease linearly. Based on this characteristic, data that shows drastic changes compared to previous time points is highly likely to have outliers or measurement errors. Therefore, if continuously measured vital sign data loses linearity or fluctuates excessively, the reliability of the data can be evaluated by assigning greater weight to data that is highly similar to previous data.

[0296] For example, when measuring heart rate every second, suppose camera A measures 60 bpm at 0 seconds, 75 bpm at 1 second, and 61 bpm at 2 seconds. Conversely, suppose camera B measures 60 bpm at 0 seconds, 62 bpm at 1 second, and 61 bpm at 2 seconds. Then, data reliability can be ensured by assigning weights to the data from camera B, which guarantees the continuity of heart rate data at each one-second interval.

[0297] 2) Situation where the performance of the camera itself is the key factor: As one example, cameras with higher pixel counts in captured facial areas can be weighted. This leverages the fact that higher pixel counts in cameras ensure clearer and larger facial landmarks.

[0298] As another embodiment, higher-resolution cameras can be assigned weights. This leverages the fact that higher camera resolution ensures clearer and larger facial landmarks.

[0299] As another embodiment, high frame rate cameras can be assigned weights.

[0300] The frame rate of a camera significantly impacts the accuracy of estimating heart rate (HR) after measuring vital signs data or calculating stress indices as part of vital signs post-processing. Therefore, a high frame rate is essential for more accurate vital signs data processing. Depending on the intended use and the performance of the ISP processing chip, the frame rates of the multiple cameras installed in vehicle 100 vary considerably. Consequently, weights can be assigned based on the frame rate.

[0301] As another embodiment, weights can be assigned based on the overall / partial high illumination of the camera image.

[0302] When measuring rPPG using an RGB camera, algorithm performance is significantly affected by shadows or lighting cast on the driver's / occupant's face inside the vehicle. Furthermore, if the surrounding environment is too bright or too dark, the algorithm may not function at all. This applies not only to RGB cameras but also to IR cameras. Specifically, IR cameras are affected by ambient light, making image illumination a crucial factor influencing algorithm performance. Therefore, assigning specific weights to areas with high overall / partial illumination in the camera image can improve algorithm performance.

[0303] As another embodiment, weights can be assigned to areas with relatively low image noise. Image noise is a major cause of reduced algorithm accuracy. Therefore, algorithm performance can be improved by assigning lower weights to images with high image noise.

[0304] The above explains which factors need to be weighted when evaluating the reliability of vital sign data acquired through multiple cameras. The following section will explain in detail how to assign weights to these factors.

[0305] On the other hand, when measuring vital signs data non-contactly using multiple cameras within the vehicle 100, measurement errors can occur, thus requiring data correction and error removal.

[0306] Therefore, this invention proposes a method for processing data by assigning weights based on a reliability evaluation of the measured vital sign data. In this invention, as methods for assigning weights to vital sign data, as described above, 1) data fusion, 2) soft voting, and 3) a combination of the above two methods are used.

[0307] Data fusion is a data processing method that obtains vital sign data in real time from multiple cameras that capture the driver / occupant, synchronizes the vital sign data obtained from each camera in time, and integrates the multiple vital sign data into a single data stream.

[0308] Soft voting is a data processing method that removes missing or abnormal values ​​from the vital signs data obtained from each camera, averages the normal data, or merges them using soft voting to generate the final vital signs data.

[0309] Through data fusion, soft voting, or a combination of both, the resulting vital signs data is transmitted to higher levels as input data for estimating the driver's / passenger's condition index.

[0310] Figure 15 This is a flowchart illustrating a method for filtering and fusing vital sign data and generating final input data in a soft voting manner, in relation to embodiments of the present invention.

[0311] First, each of the multiple cameras (camera 1 (221), camera 2 (222), camera 3 (223)) inside the vehicle 100 senses vital signs data of the same driver / passenger at the same time.

[0312] Specifically, vital sign data sensed by camera 1 (221) is transmitted to soft voting system 1510. Additionally, vital sign data sensed by camera 2 (222) is transmitted to soft voting system 1510. Furthermore, vital sign data sensed by camera 3 (223) is transmitted to soft voting system 1510.

[0313] The soft voting system 1510 may be included in the vital signs data integration device 800 in the form of an algorithm / processor / program. Alternatively, the soft voting system 1510 may be the data determination unit 820 of the vital signs data integration device 800. Figure 9 Hereinafter, the soft voting system 1510 will be disclosed as the data determination unit 820.

[0314] The data determination unit 820, based on the weights assigned to each vital sign data, filters vital sign data with high reliability weights or merges vital sign data reflecting reliability weights ("soft voting"), and then provides the data to the data provision unit 830. Figure 9 )transmission.

[0315] After applying weights to the probability values ​​of each vital sign data, the data determination unit 820 calculates the average value based on the assigned weights. Then, the data determination unit 820 calculates the reliability weighted probability for each vital sign data.

[0316] In another embodiment, the data determination unit 820 calculates the average value of each vital sign data based on the weights assigned to each vital sign data, filters the calculated average vital sign data, and transmits it to the data providing unit 830 as an integrated single stream ("data fusion").

[0317] In another embodiment, the data determination unit 820 can apply weights to the probability values ​​of each vital sign data and then filter out the vital sign data with the highest reliability weight probability (the label with the highest probability). Furthermore, the data determination unit 820 can calculate the average of the reliability weight probabilities, generate fused vital sign data, and transmit the filtered or fused vital sign data to the data providing unit 830.

[0318] The resulting reliable vital signs data are then provided to the final level 1520 (1504) to more accurately estimate the driver / passenger's condition index.

[0319] Figure 16 and Figure 17 These are block diagrams and flowcharts used to illustrate the acquisition, integration, evaluation, and screening of multiple vital sign data in relation to embodiments of the present invention, and to provide them to a driver state estimation algorithm.

[0320] Reference Figure 16 The multiple cameras 220 inside the vehicle may be a first camera 221 installed in the rearview mirror, a second camera 222 installed in the driver's seat A-pillar, and a third camera 223 installed in the instrument panel.

[0321] Multiple vital sign data points acquired by multiple cameras 220 at the same time point are transmitted to the vital sign data analysis module 1610. Here, the vital sign data analysis module 1610 may refer to the vital sign data integration device 800 of the present invention.

[0322] The vital signs data analysis module 1610 may include a data synchronizer 1601, a detector 1602, an evaluator 1603, a data selector 1604, a calculator 1605, and a scheduler 1606.

[0323] The data synchronizer 1601 enables the synchronization of multiple vital sign data sensed by each of multiple cameras at the same point in time (however, not exactly the same point in time).

[0324] Detector 1602 can remove missing or outlier data from time-synchronized vital sign data and detect vital sign data with normal values.

[0325] The evaluator 1603 evaluates the reliability and assigns weights based on a preset reliability benchmark for each vital sign data.

[0326] For example, based on a preset reliability, the above-mentioned method of assigning more weight to faces and camera angles that are closer to a right angle can be adopted and applied.

[0327] Specifically, when each of the plurality of cameras 220 identifies the same driver / passenger A, the closer the shooting angle is to a right angle, the higher the priority or weight can be assigned. As an example, the first camera 221, which is closest to a right angle, can be assigned a weight of "3", the second camera 222 a weight of "2", and the third camera 223 a weight of "1".

[0328] Data selector 1604 and calculator 1605 filter and merge the final data according to the weights assigned to the multiple vital sign data, and provide it to scheduler 1606.

[0329] Specifically, the data selector 1604 and the calculator 1605 can calculate a new final data by merging the values ​​of multiple vital sign data based on the weights assigned to them. For example, in the example above, the final data for the same driver / passenger A could be the calculated value of {(first camera 221×3) + (second camera 222×2) + (third camera 223×1)} / 3.

[0330] As another example, the data selector 1604 and the calculator 1605 can average each probability value based on the weights assigned to multiple vital sign data, and ultimately select the label with the highest probability or calculate their average as new final data.

[0331] Specifically, in the example above, assume that the recognition probability of the first camera 221 is 0.8, the recognition probability of the second camera 222 is 0.7, and the recognition probability of the third camera 223 is 0.6, and their respective weights are 0.6, 0.3, and 0.1, respectively, according to the shooting angle being close to a right angle.

[0332] In this scenario, the weighted probability values ​​for each camera are as follows. Specifically, the weighted probability value for the first camera 221 is calculated as (0.8 × 0.6) / (0.6 + 0.3 + 0.1) = 0.48. The weighted probability value for the second camera 222 is calculated as (0.7 × 0.3) / (0.6 + 0.3 + 0.1) = 0.21. The weighted probability value for the third camera 223 is calculated as (0.6 × 0.1) / (0.6 + 0.3 + 0.1) = 0.06.

[0333] At this point, the vital signs data sensed by the first camera 221 with the highest weight probability value can be finally selected, or the data fusion method can be further used to fuse them to determine their average value as a final fusion value.

[0334] On the other hand, scheduler 1606 extracts the final generated vital signs data and provides it to state estimation algorithm 1620.

[0335] The state estimation algorithm 1620 uses reliable vital sign data transmitted from the scheduler 1606 as input data to calculate / estimate various state indices.

[0336] like Figure 16 As shown, data can be transmitted to each of the fatigue estimation algorithm 1621, stress assessment algorithm 1622, and emergency status assessment algorithm 1623 to estimate the driver / passenger's fatigue level, calculate the stress index, and assess whether an emergency situation has occurred. The algorithm's evaluation results can be transmitted to vehicle 100 to execute relevant additional actions / services. In addition, besides... Figure 16 Besides the algorithm shown, there are obviously many more state exponential algorithms that can be included.

[0337] Figure 17 Is with Figure 16 The flowchart of the related actions.

[0338] Reference Figure 17 The vital signs data sensed from camera 1 (221) are transmitted to data integrator 1710 (1701). In addition, the vital signs data sensed from camera 2 (222) are transmitted to data integrator 1710 (1702). At the same time, the vital signs data sensed from camera 3 (223) are transmitted to data integrator 1710 (1701).

[0339] The data integrator 1710 synchronizes multiple vital sign data in time, removes missing or outlier data, and then transmits the data to the data evaluator 1720 (1704).

[0340] The data evaluator 1720 evaluates the reliability of each vital sign data according to a preset reliability benchmark, calculates each weight, and transmits it to the data selector 1730 (1705).

[0341] Data selector 1730 filters and merges reliable final data (1706) according to the data fusion method, soft voting method, or a combination of the two methods described above. The vital sign data finally generated by data selector 1730 is transmitted to various algorithms (1707, 1708, 1709) of driver state estimation algorithm 1740. Based on this, various state index evaluations can be transmitted to the associated action system within vehicle 100, thereby providing various services.

[0342] On the other hand, in embodiments of the present invention, the vital signs data integration device 800 can reflect the sensing results and / or control results of other sensors or motion systems of the vehicle in the reliability evaluation of each vital signs data.

[0343] In an embodiment, the data determination unit 810 of the vital signs data integration device 800 ( Figure 9 The reliability weights of each vital sign data can be adjusted differently based on the vehicle's control information and the passenger's movement information.

[0344] Here, vehicle control information refers to the control information used to control the vehicle's braking and steering. In this context, vehicle braking and steering include all situations where emergency braking / accelerated acceleration / sharp steering is initiated based on driver input and by the ADAS system without driver intervention.

[0345] In addition, the occupant's motion information refers to the motion information of the driver / occupant obtained through a camera-based motion monitoring module, which is sensed by the motion monitoring module installed in the vehicle 100.

[0346] In one embodiment, the data determination unit 820 of the vital signs data integration device 800 can determine, based on the vehicle's control information, that the change in the amount of change of each vital signs data has temporarily changed. Furthermore, based on this determination, the reliability weight of each vital signs data can be reduced. Alternatively, as another embodiment, based on this determination, the weight calculation cycle for each vital signs data can be extended.

[0347] For example, during vehicle operation, sudden events involving rapid vehicle control or internal / external factors, such as emergency braking due to a forward accident, can significantly impact the driver's / occupant's heart rate. In such cases, the weighting calculation period for vital sign data can be temporarily extended, or the weights assigned based on reliability assessment can be reduced by a certain value.

[0348] In another embodiment, the data determination unit 820 of the vital signs data integration device 800 can determine that the received vehicle control information has not changed for more than a specified time, and based on this determination, increase the reliability weight of each vital signs data or shorten the reliability weight calculation cycle.

[0349] In this case, the specified time may be the period during which no driving operation can be determined due to driver drowsiness or an emergency. However, the specified time may be changed based on the vehicle 100's driving status, driving time, driver's health status, etc.

[0350] For example, when driving with the same or largely unchanged steering and braking over a prolonged period, a driver may experience changes in vital signs such as decreased heart rate due to drowsy driving. In such cases, the weight of all vital sign data sensed by multiple cameras within the vehicle 100 can be increased or the weighting calculation period can be shortened to prevent driver drowsiness from affecting the driver's response.

[0351] In another embodiment, the data determination unit 820 of the vital signs data integration device 800 can determine that the amount of change in each vital signs data has temporarily changed based on the rider's motion information, and reduce the reliability weight of each vital signs data or extend the reliability weight calculation period based on this determination.

[0352] For example, if a driver engages in strenuous activity such as eating, talking, or making large steering maneuvers while driving, their heart rate may temporarily increase due to this physical movement. Such temporary heart rate fluctuations can confuse services related to driver stress levels, fatigue assessments, and status evaluations. Therefore, if strenuous activity by the driver / occupant is temporarily detected, the weight of all vital signs data sensed by multiple cameras within the vehicle can be reduced and / or the weighting calculation period extended.

[0353] As described above, the vehicle vital sign data integration device according to an embodiment of the present invention can stably, continuously, and with higher accuracy summarize and provide the vital sign data of the driver / passenger when using multiple cameras to estimate the driver's / passenger's drowsiness, stress, health information, etc. That is, it can provide reliable input data for high-level vital sign analysis.

[0354] The foregoing invention can be implemented by computer-readable code in a storage medium. Computer-readable media include all types of storage devices storing data readable by a computer system. Examples of computer-readable media include HDDs (Hard Disk Drives), SSDs (Solid State Disks), SDDs (Silicon Disk Drives), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include those implemented as carrier waves (e.g., transmitted via the Internet). Additionally, the computer may include a processor / control unit of the vital signs data integration device 800. Therefore, the detailed description above should not be construed as limiting in all respects, but rather as exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

1. A vehicle vital signs data integration device, wherein, include: The receiving unit receives vital sign data collected by multiple cameras installed inside the vehicle. The data determination unit evaluates the reliability of the received vital sign data and assigns weights to it, and filters or fuses the vital sign data based on the assigned weights. as well as The data provider provides the final generated vital sign data to estimate the driver's condition index.

2. The vehicle vital signs data integration device according to claim 1, characterized in that, The reliability of the received vital sign data is evaluated by determining the different weights of each vital sign data based on at least one of the following: the reliability of each of the plurality of cameras, key point information of the images sensed by the plurality of cameras, the area occupied by the face in the image, and the amount of change compared to previous vital sign data.

3. The vehicle vital signs data integration device according to claim 1, characterized in that, Each of the plurality of cameras is positioned around the driver's seat inside the vehicle to remotely sense the driver's vital signs data via PPG. The receiving unit synchronizes the sensed plurality of vital sign data in time and transmits them to the data determining unit.

4. The vehicle vital signs data integration device according to claim 3, characterized in that, The data determination unit removes outliers from a plurality of vital sign data, and then evaluates and assigns weights to the reliability of the normal vital sign data.

5. The vehicle vital signs data integration device according to claim 1, characterized in that, The data determination unit filters vital sign data with high reliability weights based on the assigned weights, or merges vital sign data that reflect the reliability weights, and transmits the data to the data providing unit.

6. The vehicle vital signs data integration device according to claim 5, characterized in that, The data determination unit calculates the average value of each vital sign data based on the assigned weights, filters the calculated average vital sign data, and transmits it to the data providing unit in a single stream.

7. The vehicle vital signs data integration device according to claim 5, characterized in that, After applying the assigned weights to the probability values ​​of each vital sign data, the data determination unit calculates the average value based on the assigned weights and calculates the reliability weight probability for each vital sign data.

8. The vehicle vital signs data integration device according to claim 7, characterized in that, The data determination unit filters the vital sign data with the highest reliability weight probability, or calculates the average value of the reliability weight probability to generate fused vital sign data, and transmits the filtered vital sign data or fused vital sign data to the data providing unit.

9. The vehicle vital signs data integration device according to claim 1, characterized in that, The data determination unit adjusts the reliability weights of each vital sign data differently based on the vehicle's control information and the passenger's movement information.

10. The vehicle vital signs data integration device according to claim 9, characterized in that, The data determination unit determines, based on the vehicle's control information, that the change in each vital sign data has temporarily changed, and reduces the reliability weight of each vital sign data accordingly.

11. The vehicle vital signs data integration device according to claim 10, characterized in that, The vehicle control information is the control result corresponding to any of the braking, starting, or steering actions that exceed the set range and are performed by the driver or the ADAS system.

12. The vehicle vital signs data integration device according to claim 9, characterized in that, The data determination unit determines that the received vehicle control information has not changed for more than a specified time, and increases the reliability weight of each vital sign data or shortens the reliability weight calculation cycle based on the determination.

13. The vehicle vital signs data integration device according to claim 9, characterized in that, The data determination unit determines that the change in each vital sign data has temporarily changed based on the passenger's movement information, and reduces the reliability weight of each vital sign data or extends the reliability weight calculation period based on the determination.

14. A method for integrating vital sign data of a vehicle, wherein, include: The steps of receiving vital sign data sensed by a plurality of cameras located inside the vehicle; The step of evaluating the reliability of the received vital sign data and assigning weights to it; The steps of filtering or fusing vital sign data based on assigned weights; as well as The step of providing the final generated vital signs data as input data to estimate the driver's condition index.

15. The method for integrating vital sign data of a vehicle according to claim 14, characterized in that, The step of assigning the weights is to evaluate the reliability of each vital sign data based on at least one of the following: the reliability of each of the plurality of cameras, key point information of the images sensed by the plurality of cameras, the area occupied by the face in the image, and the amount of change compared to previous vital sign data, and to determine the different weights for each.

16. The method for integrating vital sign data of a vehicle according to claim 14, characterized in that, The steps of assigning the weights include: The step involves further adjusting the reliability weights of various vital sign data based on vehicle control information and passenger movement information.

Images