Apparatus for assisting driving of a vehicle and method thereof

Abstract

An apparatus for assisting driving of a vehicle and a method thereof. An apparatus for assisting driving of a vehicle is provided, the apparatus including a camera mounted on the vehicle and configured to have a front field of view of the vehicle and acquire image data, a lidar mounted on the vehicle and configured to have a front side field of view of the vehicle and acquire lidar data, and a controller configured to process at least one of the image data or the lidar data, wherein the controller includes a calibration matrix for correcting the image data and the lidar data, and the calibration matrix is configured to: obtain a feature map of the image data and a feature map of the lidar data using machine learning, correct the calibration matrix based on a difference between the feature map of the image data and the feature map of the lidar data, and correct the image data and the lidar data using the corrected calibration matrix.

Description

Technical Field

[0001] This disclosure relates to devices and methods for assisting driving of vehicles, and more particularly, to devices and methods for assisting driving of vehicles including cameras and lidar. Background Technology

[0002] As the most common means of transportation in modern society, the number of people using vehicles is increasing. Although the development of vehicle technology has the advantages of making long-distance travel easier and life more convenient, traffic congestion often occurs in densely populated areas (such as South Korea).

[0003] Recent studies have focused on vehicles equipped with Advanced Driver Assistance Systems (ADAS) that proactively provide information about vehicle status, driver status, and traffic conditions, aiming to reduce driver workload and improve convenience.

[0004] For example, ADAS can provide functions such as Lane Departure Warning (LDW), Lane Keeping Assist (LKA), High Beam Assist (HBA), Automatic Emergency Braking (AEB), Traffic Sign Recognition (TSR), Adaptive Cruise Control (ACC), and Blind Spot Detection (BSD).

[0005] ADAS can collect information about the surrounding environment and process that information. Furthermore, ADAS can identify objects and determine the vehicle's route based on the results of processing the collected information.

[0006] ADAS can use various sensor modules (such as cameras, radar, lidar, ultrasonic sensors, etc.) to collect information about the surrounding environment and fuse the data collected by each of the cameras, radar, lidar, or ultrasonic sensors.

[0007] In this scenario, ADAS requires calibration of each of the cameras, radars, lidar, or ultrasonic sensors to synchronize the data collected by each of them. Summary of the Invention

[0008] One aspect of this disclosure provides an apparatus and method for assisting driving of a vehicle, which can perform calibration of a camera and a lidar for synchronizing image data from the camera with lidar data from the lidar.

[0009] One aspect of this disclosure provides an apparatus and method for assisting driving of a vehicle, which can use machine learning to perform calibration of cameras and lidar.

[0010] According to one aspect of this disclosure, an apparatus for assisting driving of a vehicle is provided, the apparatus comprising: a camera mounted on the vehicle and configured to have a forward field of view of the vehicle and acquire image data; a lidar mounted on the vehicle and configured to have a forward side field of view of the vehicle and acquire lidar data; and a controller configured to process at least one of the image data or lidar data, wherein the controller includes a calibration matrix for correcting the image data and lidar data, and the calibration matrix is ​​configured to: use machine learning to obtain feature maps of the image data and the lidar data; correct the calibration matrix based on the difference between the feature maps of the image data and the lidar data; and correct the image data and lidar data using the corrected calibration matrix.

[0011] The controller is configured to convert LiDAR data into two-dimensional (2D) LiDAR data and then convert the 2D LiDAR data into pixel coordinates corresponding to image data.

[0012] The controller is configured to synchronize the size of the image data with the size of the 2D LiDAR data.

[0013] The controller is configured to use a convolutional neural network algorithm to obtain feature maps of synchronized image data, and to use a convolutional neural network algorithm to obtain feature maps of synchronized 2D LiDAR data.

[0014] The controller is configured to obtain feature maps of synchronized image data based on convolution between synchronized image data and filters, and to obtain feature maps of synchronized 2D LiDAR data based on convolution between synchronized 2D LiDAR data and filters.

[0015] The controller is configured to obtain a feature matrix in which feature maps of synchronized image data and synchronized 2D LiDAR data are integrated.

[0016] The controller is configured to obtain the feature matrix based on the difference between the feature map of the synchronized image data and the feature map of the synchronized 2D LiDAR data.

[0017] The controller is configured to use a convolutional neural network algorithm to obtain correction parameters from the feature matrix for correcting the calibration matrix.

[0018] The controller is configured to correct image data and LiDAR data using a calibration matrix corrected by calibration parameters.

[0019] According to one aspect of this disclosure, a method for assisted driving of a vehicle is provided, the method comprising the steps of: acquiring image data via a camera mounted on the vehicle and configured to have a forward field of view of the vehicle; acquiring lidar data via a lidar mounted on the vehicle and configured to have a forward side field of view of the vehicle; obtaining feature maps of the image data and the lidar data using machine learning via a processor mounted on the vehicle; correcting a calibration matrix for correcting the image data and the lidar data based on the difference between the feature maps of the image data and the lidar data; and correcting the image data and the lidar data using the corrected calibration matrix. Attached Figure Description

[0020] These and / or other aspects of this disclosure will become apparent and more readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 The configuration of the vehicle and driver assistance equipment according to the embodiment is illustrated;

[0022] Figure 2 An example is shown of the field of view of a camera and radar included in a driver assistance device according to an embodiment;

[0023] Figure 3 The functional modules of a controller included in a driver assistance device according to an embodiment are illustrated;

[0024] Figure 4 An example of a method for calibrating a camera and lidar for a driver assistance device according to an embodiment is illustrated;

[0025] Figure 5 An example is illustrated of a driver assistance device performing convolution of image data according to an embodiment; and

[0026] Figure 6 This is a diagram illustrating the convolution of image data by a driver assistance device according to an embodiment. Detailed Implementation

[0027] Throughout this specification, the same reference numerals denote the same elements. Furthermore, this specification does not describe all elements according to embodiments of this disclosure, and omits descriptions or overlapping portions known in the art to which this disclosure pertains. Terms such as “~part,” “~component,” “~module,” “~block,” etc., may refer to at least one process processed by at least one piece of hardware or software. According to embodiments, multiple “~parts,” “~components,” “~modules,” and “~blocks” may be embodied as a single element, or a single “~part,” “~component,” “~module,” or “~block” may include multiple elements.

[0028] It should be understood that when an element is referred to as "connected" to another element, it can be directly or indirectly connected to the other element, where indirect connections include "connections" via wireless communication networks.

[0029] It is also understood that the term "comprising," when used in this specification, specifies the presence of a defined feature, element, step, operation, element, and / or component, rather than excluding the presence or addition of one or more other features, elements, steps, operations, elements, components, and / or combinations thereof.

[0030] It should be understood that when a component is described in this specification as being "on another component", the component can not only contact the other component, but there can also be another component between the two components.

[0031] It should be understood that although the terms first, second, etc. may be used in this document to describe the various elements, these elements should not be limited to these terms.

[0032] Unless the context explicitly indicates otherwise, the singular form is also intended to include the plural form.

[0033] The labels used for method steps are for illustrative purposes only and do not restrict the order of steps. Therefore, unless the context explicitly states otherwise, the written order can be practiced in other ways.

[0034] The operating principle and implementation method will be described in detail below with reference to the accompanying drawings.

[0035] Figure 1 An example of a vehicle configuration according to an implementation method is shown. Figure 2 An example is shown of the field of view of a camera and radar included in a driver assistance device according to an embodiment;

[0036] like Figure 1As shown, vehicle 1 includes: a drive unit 20, a braking unit 30, a steering unit 40, and / or driver assistance equipment 100. These devices can communicate with each other via a vehicle communication network (NT). For example, electronic devices 20, 30, 40, and 100 included in vehicle 1 can send / receive data via Ethernet, System Transmission to Media (MOST), FlexRay, Controller Area Network (CAN), Local Interconnect Network (LIN), etc.

[0037] The drive unit 20 enables the vehicle 1 to move and includes, for example, an engine, an engine management system (EMS), a transmission, and a transmission control unit (TCU). The engine generates power to drive the vehicle 1, and the EMS can control the engine in response to the driver's acceleration intention via the accelerator pedal or a request from the driver assistance device 100. The transmission decelerates and transmits the power generated by the engine to the wheels, and the TCU can control the transmission in response to the driver's shift command via the shift lever and / or a request from the driver assistance device 100.

[0038] Braking device 30 can bring vehicle 1 to a stop and includes, for example, brake calipers and an electronic brake control module (EBCM). The brake calipers can decelerate or stop vehicle 1 by using friction with the brake discs, and the EBCM can control the brake calipers in response to the driver's braking intention via the brake pedal and / or requests from driver assistance device 100. For example, the EBCM can receive a deceleration request from driver assistance device 100, including deceleration, and control the brake calipers electrically or hydraulically to decelerate vehicle 1 based on the requested deceleration.

[0039] The steering system 40 may include an electronic power steering control module (EPS). The steering system 40 can change the direction of travel of the vehicle 1. The EPS can assist the operation of the steering system 40, allowing the driver to easily manipulate the steering wheel according to the driver's steering intention. Furthermore, the EPS can control the steering system 40 in response to requests from the driver assistance device 100. For example, the EPS can receive a steering request including steering torque from the driver assistance device 100 and control the steering system 40 so that the vehicle 1 steers based on the requested steering torque.

[0040] The driver assistance device 100 can communicate with the drive unit 20, braking unit 30, steering unit 40, display device 50, and audio device 60 via a vehicle communication network. The driver assistance device 100 can receive route information to the destination and the location information of the vehicle 1 from the navigation device, and acquire information about the driving speed, acceleration, and / or angular velocity of the vehicle 1 from the plurality of sensors.

[0041] Driver assistance device 100 can provide drivers with a variety of safety functions. For example, driver assistance device 100 can provide functions such as lane departure warning (LDW), lane keeping assist (LKA), high beam assist (HBA), automatic emergency braking (AEB), traffic sign recognition (TSR), adaptive cruise control (ACC), blind spot detection (BSD), etc.

[0042] The driver assistance device 100 may include: a camera 110, a radar 120, a light detection and ranging (LiDAR) device 130, and a controller 140. The driver assistance device 100 is not limited to... Figure 1 The driver assistance devices illustrated. For example, in Figure 1 In the illustrated driver assistance device 100, at least one of the camera 110, radar 120 or lidar 130 may be omitted, or various detection devices capable of detecting objects around the vehicle 1 may be added.

[0043] The camera 110, radar 120, lidar 130, and controller 140 can be configured to be separate from each other. For example, the controller 140 can be installed in a housing separate from the housings of each of the camera 110, radar 120, and lidar 130. The controller 140 can send data to / receive data from the camera 110, radar 120, or lidar 130 via a broadband network.

[0044] Camera 110 can capture images of the area around vehicle 1 and acquire image data of the area around vehicle 1. For example, such as... Figure 2 As shown, the camera 110 can be mounted on the windshield of the vehicle 1 and has a field of view 110a facing forward of the vehicle 1.

[0045] The camera 110 may include multiple lenses and an image sensor. The image sensor may include multiple photodiodes that convert light into electrical signals, and the multiple photodiodes may be arranged in a two-dimensional (2D) matrix.

[0046] Image data may include information about other vehicles, pedestrians, cyclists, or lane lines (markers that distinguish lanes on a road) located around vehicle 1.

[0047] Camera 110 may include a graphics processor for processing image data and for detecting objects around vehicle 1 based on the processing of the image data. For example, camera 110 may use image processing to generate tracks representing objects and classify those tracks. For example, camera 110 may determine whether the track is another vehicle, pedestrian, cyclist, etc.

[0048] Camera 110 can be electrically connected to controller 140. For example, camera 110 can be connected to controller 140 via a vehicle communication network, hardwire, or printed circuit board (PCB). Camera 110 can send image data (or the location and classification of the trajectory) about the surroundings of vehicle 1 to controller 140.

[0049] Radar 120 can emit transmission waves around vehicle 1 and detect objects around vehicle 1 based on reflected waves from objects. For example, Figure 2 As shown, the radar 120 can be mounted on the grille or bumper of the vehicle 1 and has a sensing field 120a facing the front of the vehicle 1.

[0050] Radar 120 may include a transmitting antenna (or transmitting antenna array) that transmits transmitted waves toward the vicinity of vehicle 1, and a receiving antenna (or receiving antenna array) that receives reflected waves reflected from an object.

[0051] Radar 120 can acquire radar data from transmitted waves emitted by the transmitting antenna and reflected waves received by the receiving antenna. The radar data may include position information (e.g., distance information) and / or speed information of other objects located in front of vehicle 1.

[0052] Radar 120 may include a signal processor for processing radar data and generate a trajectory representing an object by clustering reflection points from reflected waves. Radar 120 may obtain the distance to the trajectory based on the time difference between the transmission time of the transmitted wave and the reception time of the reflected wave (i.e., the time elapsed from transmission until reception). Furthermore, radar 120 may obtain the relative velocity of the trajectory based on the frequency difference between the transmitted and reflected waves.

[0053] For example, radar 120 can be connected to controller 140 via vehicle communication network, hardwire, or PCB, and radar data (or distance to the track or relative speed) can be sent to controller 140.

[0054] The lidar 130 can emit light (e.g., infrared light) around the vehicle 1 and detect objects around the vehicle 1 based on reflected light from objects. For example, as Figure 2 As shown, the lidar 130 can be mounted on the roof of the vehicle 1 and has a field of view 130a facing all directions around the vehicle 1.

[0055] The lidar 130 may include a light source (e.g., a light-emitting diode (LED), an array of LEDs, a laser diode, or an array of laser diodes) that emits light (e.g., infrared light), and a light sensor (e.g., a photodiode or an array of photodiodes) that receives light (e.g., infrared light). Furthermore, the lidar 130 may, as needed, include a driving mechanism for rotating the light source and / or the light sensor.

[0056] The lidar 130 can acquire lidar data by emitting light through the light source and receiving light reflected from the object through the light sensor while the light source and / or light sensor rotate.

[0057] The lidar data may include the relative position (distance from the nearby object and / or direction of the nearby object) and / or relative velocity of nearby objects around vehicle 1. LiDAR 130 may include a signal processor capable of processing lidar data and generating a trajectory representing the object by clustering reflection points from reflected waves. LiDAR 130 may obtain the distance to the object based on the time difference between light emission and light reception (i.e., the time elapsed from emission until light is received). Furthermore, LiDAR 130 may obtain the direction (or angle) of the object relative to the direction of travel of vehicle 1 based on the direction of light emitted by the light source when the light sensor receives the reflected light.

[0058] For example, the lidar 130 can be connected to the controller 140 via a vehicle communication network, hardwire, or PCB, and the lidar data (or the relative position and relative speed of the trajectory) can be sent to the controller 140.

[0059] The controller 140 can be electrically connected to the camera 110, radar 120, and / or lidar 130. Furthermore, the controller 140 can be connected via a vehicle communication network to a navigation device, drive unit 20, braking device 30, steering device 40, display device 50, audio device 60, and / or the aforementioned sensors.

[0060] The controller 140 can process image data from the camera 110, radar data from the radar 120, and / or lidar data from the lidar 130, and provide control signals to the drive unit 20, the braking unit 30, and / or the steering unit 40.

[0061] The controller 140 may include a processor 141 and a memory 142.

[0062] The processor 141 can process image data from the camera 110, radar data from the radar 120, and / or lidar data from the lidar 130. Furthermore, based on the processing of the image data from the camera 110, the radar data from the radar 120, and / or the lidar data from the lidar 130, the processor 141 can generate drive signals, braking signals, and / or steering signals for controlling the drive unit 20, the braking unit 30, and / or the steering unit 40, respectively.

[0063] For example, processor 141 may include: an image processor for processing image data of camera 110, and / or a signal processor for processing radar data of radar 120 and lidar data of lidar 130, and / or a microcontroller unit (MCU) for generating drive signals, braking signals and / or steering signals.

[0064] The memory 142 may store programs and / or data for the processor 141 to process image data, radar data and / or lidar data, and also stores programs and / or data for the processor 141 to generate drive signals, braking signals and / or steering signals.

[0065] The memory 142 can temporarily store image data received from the camera 110, radar data received from the radar 120, and / or lidar data received from the lidar 130. Furthermore, the memory 142 can temporarily store the processing results of the processor 141 on the image data, radar data, and / or lidar data.

[0066] The memory 142 may include volatile memory such as static random access memory (S-RAM) and dynamic random access memory (D-RAM), as well as non-volatile memory such as flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM).

[0067] As described above, the controller 140 can provide drive signals, braking signals, or steering signals based on image data from the camera 110, radar data from the radar 120, and / or lidar data from the lidar 130.

[0068] The operation of the driver assistance device 100 will be described in detail below.

[0069] Figure 3 The functional modules of a controller included in a driver assistance device according to an embodiment are illustrated.

[0070] Functionally, the controller 140 may include: a perception module 146 for collecting information about the vehicle 1 and / or objects around the vehicle 1; a decision module 147 for generating a detailed route to avoid collisions with objects around the vehicle 1; and a control module 148 for controlling the vehicle 1 to travel along the generated detailed route. Each of the modules in the perception module 146, decision module 147, and control module 148 may be a hardware module (e.g., an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)) included in the processor 141, or a software module (e.g., an application program or data) stored in the memory 142.

[0071] The perception module 146 of the controller 140 may include sensor fusion 146a, which detects objects around the vehicle 1 by fusing image data from the camera 110, radar data from the radar 120, and lidar data from the lidar 130.

[0072] For example, the controller 140 can detect objects around the vehicle 1 (e.g., other vehicles, pedestrians, cyclists, etc.) based on image data from the camera 110, radar data from the radar 120, and lidar data from the lidar 130.

[0073] The controller 140 can acquire the relative position (angle relative to the direction of travel) and / or type (e.g., whether the object is another vehicle, pedestrian, cyclist, etc.) of objects around the vehicle 1 based on image data from the camera 110. The controller 140 can acquire the relative position (distance from the vehicle) and / or relative speed of objects around the vehicle 1 based on radar data from the radar 120. The controller 140 can acquire the relative position (distance from the vehicle and / or angle relative to the direction of travel) and / or relative speed of objects around the vehicle 1 based on lidar data from the lidar 130. Furthermore, the controller 140 can match objects detected by radar data and / or lidar data with objects detected by image data, and based on the matching result, acquire the type, relative position, and / or relative speed of objects around the vehicle 1.

[0074] Furthermore, the controller 140 can obtain lane information about the road on which the vehicle 1 is traveling based on the image data from the camera 110. For example, the controller 140 can determine the lateral position of the lane.

[0075] The perception module 146 may include a positioning 146b, which determines the position of vehicle 1 based on image data from camera 110, lidar data from lidar 130, map data, and / or vehicle 1 position data.

[0076] For example, controller 140 can determine landmarks around vehicle 1 from map data based on vehicle 1's location data. Controller 140 can determine the relative position of landmarks with respect to vehicle 1 based on image data and LiDAR data. Controller 140 can correct vehicle 1's location data based on the absolute coordinates of landmarks based on map data and their relative coordinates based on image data and / or LiDAR data. In other words, controller 140 can determine the absolute coordinates of vehicle 1.

[0077] The perception module 146 may include a map projection 146c that projects vehicle 1 and / or objects around vehicle 1 onto map data.

[0078] For example, controller 140 can project vehicle 1 onto map data based on the calibrated position data of vehicle 1, and project objects around vehicle 1 onto map data based on the relative positions and / or relative speeds of objects around vehicle 1.

[0079] The determination module 147 of the controller 140 may include a collision risk determination 147a, which predicts a collision with an object around the vehicle 1 based on the relative position and relative speed of the object around the vehicle 1 and confirms the collision risk.

[0080] For example, controller 140 can calculate the time to collision (TTC) (or distance to collision (DTC)) between vehicle 1 and objects around vehicle 1 based on the relative position (distance and / or orientation) and relative speed of objects around vehicle 1. Furthermore, controller 140 can assess the risk of collision with objects around vehicle 1 based on either TTC or DTC.

[0081] The determination module 147 may include a target selection 147b, which selects a target for collision avoidance or a target for following.

[0082] For example, in automatic emergency braking, controller 140 can select a nearby object with the highest collision risk (or shortest collision time) as the target based on collision risk (or collision time). Furthermore, in adaptive cruise control, controller 140 can select a vehicle traveling in the same lane as vehicle 1 as the target.

[0083] The decision module 147 may include path generation 147c, which generates a detailed path to avoid collisions with a target or a detailed path to reach a destination.

[0084] For example, controller 140 can generate a detailed path for maintaining lane position to follow a target or for changing lanes to avoid a collision.

[0085] The control module 148 of the controller 140 may include: a drive control 148a for controlling the drive device 20, a brake control 148b for controlling the brake device 30, and a steering control 148c for controlling the steering device 40.

[0086] The controller 140 can generate drive signals, braking signals, or steering signals for following a target or avoiding a collision with the target. For example, the controller 140 can send drive signals and / or braking signals to the drive unit 20 and / or the braking unit 30 to allow the distance to the target (or the time taken to reach the position of the preceding vehicle) to be a distance set by the driver. Furthermore, the controller 140 can send braking signals and / or steering signals to the braking unit 30 and / or the steering unit 40 to avoid a collision with the target.

[0087] As described above, the driver assistance device 100 can fuse image data from the camera 110, radar data from the radar 120, and lidar data from the lidar 130 to detect and identify objects around the vehicle 1.

[0088] The driver assistance device 100 can synchronize the image data of the camera 110, the radar data of the radar 120, and the lidar data of the lidar 130, so as to fuse the image data of the camera 110, the radar data of the radar 120, and the lidar data of the lidar 130.

[0089] Specifically, the driver assistance device 100 can perform calibration of the camera 110 and the lidar 130 respectively, so as to synchronize the image data of the camera 110 and the lidar data of the lidar 130.

[0090] Typically, calibration can be performed by someone using a checkerboard (or markings).

[0091] In traditional calibration methods using grid patterns, the grid pattern size is too large, requiring significant space for calibration. Users cannot perform calibration independently, and it cannot be performed automatically at any desired time. That is, the standard calibration method for cameras and LiDAR can only be performed initially upon vehicle delivery. Furthermore, when cameras and LiDAR become misaligned due to vehicle impacts or vibrations, calibration must be performed at a repair shop or factory, thus preventing the cameras and LiDAR from functioning properly before delivery to the repair shop or factory.

[0092] However, the driver assistance device 100 can use machine learning to perform calibration of each of the cameras 110 and lidar 130 while the vehicle 1 is in motion.

[0093] Figure 4An example of a method for calibrating a camera and lidar for a driver assistance device according to an embodiment is illustrated. Figure 5 An example is illustrated of a driver assistance device according to an embodiment performing convolution of image data. Figure 6 This is a diagram illustrating the convolution of image data by a driver assistance device according to an embodiment.

[0094] Reference Figure 4 , Figure 5 as well as Figure 6 A method 1000 for calibrating camera 110 and lidar 130 for driver assistance device 100 is described.

[0095] The driver assistance device 100 can acquire image data from the camera 110 and lidar data from the lidar 130 (1010).

[0096] Camera 110 can capture images of the area around vehicle 1 and acquire image data. LiDAR 130 can also capture images of the area around vehicle 1 and acquire LiDAR data.

[0097] The controller 140 can acquire image data and lidar data from the camera 110 and the lidar 130, respectively.

[0098] The driver assistance device 100 can use calibration parameters to correct the image data of the camera 110 and the lidar data of the lidar 130 (1020).

[0099] The controller 140 may include a calibration matrix for calibrating the image data of the camera 110 and the lidar data of the lidar 130.

[0100] For example, calibration parameters may include matrices for correcting image data of camera 110 based on calibration results using a grid pattern (or markings) and matrices for correcting lidar data of lidar 130. The calibration matrix may include matrices for parallel shifting of the image data or lidar data, or matrices for rotational shifting of the image data or lidar data.

[0101] As described above, the calibration matrix may include an intrinsic parameter matrix for correcting distortions caused by the internal effects of each of the camera 110 and the lidar 130.

[0102] Additionally, the calibration matrix may include a calibration matrix for fusing image data from camera 110 and lidar data from lidar 130. For example, the calibration matrix may include an extrinsic parameter matrix for correcting differences in the mounting positions of camera 110 and lidar 130.

[0103] The controller 140 can correct individual data in the image data and lidar data by applying intrinsic and extrinsic parameter matrices to the individual data in the image data and lidar data.

[0104] The driver assistance device 100 can convert lidar data into two-dimensional (2D) data and convert the coordinates (1030) of the 2D data.

[0105] The LiDAR data may include the relative positions (distances and / or directions) of objects around vehicle 1. That is, the LiDAR data may include three-dimensional (3D) data, which includes the distances between vehicle 1 and objects around vehicle 1.

[0106] The controller 140 can convert 3D LiDAR data into 2D LiDAR data. That is, the controller 140 can reproduce 2D LiDAR data from 3D LiDAR data. For example, the controller 140 can acquire 2D LiDAR data by projecting 3D LiDAR data onto a specific plane.

[0107] Furthermore, the controller 140 can convert the 2D LiDAR data converted from 3D LiDAR data into pixel coordinates corresponding to the image data of the camera 110. Specifically, the controller 140 may include a transformation matrix for converting the 2D LiDAR data into the pixel coordinate system of the image data, and use this transformation matrix to convert the 2D LiDAR data into the pixel coordinate system. The transformation matrix may include a matrix for parallel movement of the object containing the 2D LiDAR data or a matrix for rotating the object containing the 2D LiDAR data.

[0108] The driver assistance device 100 can synchronize the size of the image data and the size of the 2D lidar data (1040).

[0109] like Figure 2 As shown, the LiDAR 130 is mounted at a different location than the camera 110. Therefore, when the camera 110 and the LiDAR 130 each capture a single object, the size of the object in the 2D LiDAR data can differ from the size of the object in the image data. The controller 140 can zoom in or out of the image data or the 2D LiDAR data to synchronize the size of the image data and the 2D LiDAR data. For example, the controller 140 can use a matrix for zooming in or out of the objects in the 2D LiDAR data to zoom in or out of the 2D LiDAR data.

[0110] Moreover, such as Figure 2As shown, the field of view 130a of the lidar 130 is larger than the field of view 110a of the camera 110. The controller 140 can segment the 2D lidar data to synchronize the image data and the 2D lidar data. For example, the controller 140 can exclude a portion of the 2D lidar data that corresponds to the field of view 110a of the camera 110.

[0111] Based on the above operation, the 2D lidar data of lidar 130 can correspond to the image data of camera 110.

[0112] The driver assistance device 100 can obtain feature maps (1050) from various data in the image data and 2D lidar data.

[0113] The controller 140 can use machine learning to obtain feature maps from individual data in image data and 2D LiDAR data.

[0114] For example, controller 140 can use a convolutional neural network (CNN) to obtain feature maps from individual data in image data and 2D LiDAR data. CNN is a widely used algorithm for identifying or classifying objects from image data.

[0115] Typically, a CNN can include feature extraction for extracting feature maps of an image and classification for classifying (or identifying) objects based on the extracted feature maps.

[0116] The controller 140 can use CNN feature extraction to obtain feature maps of image data and feature maps of 2D LiDAR data.

[0117] like Figure 5 As shown, convolution and pooling can be repeated in feature extraction.

[0118] Convolution is the process of applying filters (or filter matrices) to image data (image matrices). For example... Figure 6 As shown in (a) and (b), each of the image data and the filter can be in matrix form. When the filter matrix traverses the image matrix at predetermined intervals, the controller 140 can perform a convolution between the image matrix and the filter matrix. For example, the controller 140 can obtain values ​​obtained by multiplying the values ​​of the filter matrix by the corresponding values ​​of the image matrix, and obtain the sum of the obtained values.

[0119] Feature maps can be obtained through convolution between image data and filters. Controller 140 can obtain feature maps based on image data and feature maps based on 2D LiDAR data.

[0120] Pooling is a process that reduces the size of feature maps to decrease the associated load on processor 141. Furthermore, specific data within a feature map can be emphasized through pooling.

[0121] One of the following can be used: max pooling that obtains the maximum value in a specific region, average pooling that obtains the average value in a specific region, or min pooling that obtains the minimum value in a specific region. For example, controller 140 can perform max pooling to obtain the maximum value in a neighboring 2x2 region of the feature map.

[0122] The controller 140 can obtain feature maps of image data by repeatedly performing convolution and pooling of image data, and obtain feature maps of 2D LiDAR data by repeatedly performing convolution and pooling of 2D LiDAR data.

[0123] The driver assistance device 100 can integrate the feature map of the image data with the feature map of the 2D lidar data (1060).

[0124] For example, controller 140 can obtain a feature map based on the matrix difference between the feature map of the image data and the feature map of the 2D LiDAR data. Specifically, controller 140 can perform matrix subtraction between the feature map of the image data and the feature map of the 2D LiDAR data to obtain a feature matrix.

[0125] The driver assistance device 100 can correct the calibration matrix (1070) based on the integrated feature map.

[0126] The controller 140 can use machine learning to correct the calibration matrix based on the feature matrix. For example, the controller 140 can use a CNN to obtain parameters for correcting the calibration matrix from the feature matrix.

[0127] The controller 140 can use feature extraction from the CNN to obtain feature maps of the feature matrix. The controller 140 can obtain parameters for correcting the calibration matrix by applying the feature maps of the feature matrix to the CNN's classification.

[0128] The classification of CNNs can include neural networks.

[0129] A neural network may include: an input layer that takes image data as input, an output layer that outputs information about the identified object, and hidden layers between the input and output layers. The input layer may include multiple input nodes, to which the brightness values ​​of the image's pixels are respectively input. The hidden layer may include multiple hidden nodes, to which values ​​obtained by applying a first weight to the values ​​of the multiple input nodes are respectively input. The output layer may include multiple output nodes, to which values ​​obtained by applying a second weight to the values ​​of the multiple hidden nodes are respectively input.

[0130] The controller 140 can obtain parameters for correcting the calibration matrix based on the values ​​of the plurality of output nodes.

[0131] The controller 140 can use the obtained parameters to correct the calibration matrix. Furthermore, the controller 140 can use the corrected calibration matrix to correct individual data in the image data and the LiDAR data.

[0132] As described above, the driver assistance device 100 can use machine learning to perform calibration of each of the camera 110 and the lidar 130. Therefore, the driver assistance device 100 can use machine learning-based calibration to correct misalignment of the camera 110 and lidar 130 caused by vibrations of the vehicle 1 during driving.

[0133] As is evident from the foregoing, according to embodiments of the present disclosure, the device and method for assisting driving of a vehicle can perform calibration of a camera and a lidar for synchronizing image data from the camera and lidar data from the lidar.

[0134] Devices and methods used for assisted driving of vehicles can use machine learning to perform calibration of cameras and lidar.

[0135] Therefore, devices and methods for assisting vehicle driving can use machine learning to correct the differences between camera image data and lidar data that occur during driving.

[0136] Therefore, any of the exemplary embodiments described above can be implemented by computer-readable code / instructions in / on a medium (e.g., a computer-readable medium) to control at least one processing element. This medium can correspond to any medium that allows the storage and / or transmission of computer-readable code.

[0137] The computer-readable code can be recorded on a medium or transmitted over the Internet. This medium may include: read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, and optical recording media.

[0138] The computer-readable code may be provided in the form of a non-transitory storage medium. Here, when a storage medium is referred to as "non-transitory," it is understood that the storage medium is tangible and does not contain signals (e.g., electromagnetic waves), while the data is stored in the storage medium in a semi-permanent or temporary manner. For example, a "non-transitory storage medium" may include a cache in which data is temporarily stored.

[0139] Although the embodiments are described for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the scope and spirit of this disclosure. Therefore, the embodiments are not described for limiting purposes.

[0140] Cross-references to related applications

[0141] This application is based on and claims priority to Korean Patent Application No. 10-2021-0051193, filed on April 20, 2021, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.

Claims

1. A device for assisting driving a vehicle, the device comprising: A camera, which is mounted on the vehicle and configured to have a forward field of view of the vehicle and acquire image data; A lidar, which is mounted on the vehicle and configured to have a frontal field of view of the vehicle and acquire lidar data; as well as A controller, configured to process at least one of the image data or the lidar data, The controller includes a calibration matrix for correcting the image data and the LiDAR data, and the controller is configured to: Machine learning is used to obtain feature maps of the image data and feature maps of the lidar data; The calibration matrix is ​​corrected based on the difference between the feature map of the image data and the feature map of the lidar data; and The image data and the lidar data are corrected using a calibrated calibration matrix.

2. The device according to claim 1, wherein, The controller is configured to convert the lidar data into two-dimensional (2D) lidar data, and to convert the 2D lidar data into pixel coordinates corresponding to the image data.

3. The device according to claim 2, wherein, The controller is configured to synchronize the size of the image data with the size of the 2D LiDAR data.

4. The device according to claim 3, wherein, The controller is configured to use a convolutional neural network algorithm to obtain feature maps of synchronized image data, and to use the convolutional neural network algorithm to obtain feature maps of synchronized 2D LiDAR data.

5. The device according to claim 4, wherein, The controller is configured to obtain the feature map of the synchronized image data based on the convolution between the synchronized image data and the filter, and to obtain the feature map of the synchronized 2D LiDAR data based on the convolution between the synchronized 2D LiDAR data and the filter.

6. The device according to claim 4, wherein, The controller is configured to obtain a feature matrix in which the feature map of synchronized image data is integrated with the feature map of synchronized 2D LiDAR data.

7. The device according to claim 6, wherein, The controller is configured to obtain the feature matrix based on the difference between the feature map of the synchronized image data and the feature map of the synchronized 2D LiDAR data.

8. The device according to claim 6, wherein, The controller is configured to use the convolutional neural network algorithm to obtain correction parameters from the feature matrix for correcting the calibration matrix.

9. The device according to claim 8, wherein, The controller is configured to use the calibration matrix, which is corrected by the correction parameters, to correct the image data and the lidar data.

10. A method for assisting driving a vehicle, the method comprising the following steps: Image data is acquired through a camera, which is mounted on the vehicle and configured to have a forward field of view of the vehicle; LiDAR data is acquired by a LiDAR, which is mounted on the vehicle and configured to have a frontal field of view of the vehicle. The feature maps of the image data and the lidar data are obtained by using machine learning through a processor installed on the vehicle. The calibration matrix used to correct the image data and the lidar data is corrected based on the difference between the feature map of the image data and the feature map of the lidar data; and The image data and the lidar data are corrected using a calibrated calibration matrix.

11. The method according to claim 10, wherein, The steps for obtaining the feature map of the image data and the feature map of the lidar data include: Convert the lidar data into 2D lidar data; and The 2D LiDAR data is converted into pixel coordinates corresponding to the image data.

12. The method according to claim 11, wherein, The steps of obtaining the feature map of the image data and the feature map of the lidar data include: synchronizing the size of the image data with the size of the 2D lidar data.

13. The method according to claim 12, wherein, The steps for obtaining the feature map of the image data and the feature map of the lidar data include: Using convolutional neural network algorithms to obtain feature maps of synchronized image data; and The convolutional neural network algorithm is used to obtain feature maps of synchronized 2D LiDAR data.

14. The method according to claim 13, wherein, The steps for obtaining the feature map of the image data and the feature map of the lidar data include: The feature map of the synchronized image data is obtained by convolution between the synchronized image data and the filter; and The feature map of the synchronized 2D LiDAR data is obtained by convolution between synchronized 2D LiDAR data and filters.

15. The method according to claim 13, wherein, The step of obtaining the feature map of the image data and the feature map of the lidar data includes obtaining a feature matrix in which the feature map of the synchronized image data and the feature map of the synchronized 2D lidar data are integrated.

16. The method according to claim 15, wherein, The step of obtaining the feature map of the image data and the feature map of the lidar data includes: obtaining the feature matrix based on the difference between the feature map of the synchronized image data and the feature map of the synchronized 2D lidar data.

17. The method according to claim 15, wherein, The steps of obtaining the feature map of the image data and the feature map of the lidar data include: using the convolutional neural network algorithm to obtain correction parameters from the feature matrix for correcting the calibration matrix.

18. The method according to claim 17, wherein, The step of obtaining the feature map of the image data and the feature map of the lidar data includes: correcting the image data and the lidar data using the calibration matrix corrected by the correction parameters.

19. A computer-readable recording medium storing a program for implementing the method according to claim 10.

Images