Device and method for assisting driving

Abstract

Device and method for assisting driving. Disclosed herein is a device for assisting driving of a vehicle, comprising: a camera mounted in the vehicle, the camera having a field of view surrounding the vehicle and acquiring image data; and a controller configured to process the image data. The controller can identify at least one object located around the vehicle based on the processed image data, update the vehicle's trajectory based on interference between the trajectory of the at least one object and the vehicle's trajectory, and control at least one of a drive device, a brake device, and a steering device of the vehicle based on the updated trajectory.

Description

Technical Field

[0001] The present disclosure relates to an apparatus and method for assisting driving of a vehicle, and more particularly, to an apparatus and method for assisting driving of the vehicle that can detect an object located near the vehicle. Background Art

[0002] Generally speaking, vehicles are the most common means of transportation in modern society, and the number of people using vehicles is increasing. The development of vehicle technology has advantages such as ease of long-distance travel and convenience of life, but also has disadvantages such as worsening road traffic conditions in places with high population density (such as South Korea), resulting in severe traffic congestion.

[0003] Recently, research on vehicles equipped with advanced driver assistance systems (ADAS) that proactively provide information on vehicle conditions, driver conditions, and / or surrounding environments in order to reduce driver burden and increase convenience is actively underway.

[0004] Examples of ADAS installed on a vehicle may include lane departure warning (LDW), lane keeping assist (LKA), high beam assist (HBA), automatic emergency braking (AEB), traffic sign recognition (TSR), adaptive cruise control (ACC), or blind spot detection (BSD).

[0005] ADAS can collect information about the surrounding environment and process the collected information. Moreover, ADAS can recognize objects and design a route for the vehicle to travel based on the results of processing the collected information.

[0006] However, such conventional ADAS controls the movement of the vehicle based only on information related to a selected target (position information and movement information). Summary of the Invention

[0007] One aspect of the present disclosure is to provide an apparatus and method for assisting driving of a vehicle, capable of predicting trajectories of a plurality of objects located near the vehicle.

[0008] Additional aspects of the disclosure will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the disclosure.

[0009] According to one aspect of the present disclosure, a device for assisted driving of a vehicle is provided, the device comprising: a camera installed in the vehicle, the camera having a field of view surrounding the vehicle and obtaining image data; and a controller configured to process the image data, wherein the controller is configured to confirm at least one object located around the vehicle based on the processed image data, update the trajectory of the vehicle based on interference between the trajectory of the at least one object and the trajectory of the vehicle, and control at least one of a driving device, a braking device, and a steering device of the vehicle based on the updated trajectory of the vehicle.

[0010] The controller may project the trajectory of the vehicle and the predicted trajectory of the at least one object onto a coordinate system, and confirm whether the trajectory of the vehicle interferes with the trajectory of the at least one object on the one coordinate system.

[0011] The controller may determine at least one of a lateral velocity, a lateral acceleration, an alignment angle, a longitudinal velocity, or a longitudinal acceleration of the at least one object based on processing the image data.

[0012] The controller may confirm whether the at least one object changes lanes based on at least one of a lateral speed, a lateral acceleration, or an alignment angle of the at least one object.

[0013] The controller may predict a lateral trajectory of the at least one object based on at least one of a lateral velocity, a lateral acceleration, or an alignment angle of the at least one object, predict a longitudinal trajectory of the at least one object based on a longitudinal velocity or a longitudinal acceleration of the at least one object, and predict a trajectory of the at least one object based on the lateral trajectory and the longitudinal trajectory.

[0014] The controller may generate the trajectory of the vehicle based on a route to a destination obtained from a navigation device of the vehicle.

[0015] The controller may predict the trajectory of the vehicle based on at least one of the vehicle's speed, acceleration, or angular velocity.

[0016] The controller may update the trajectory of the vehicle to avoid the trajectory of the at least one object in response to the trajectory of the vehicle interfering with the trajectory of the at least one object.

[0017] The device may also include a radar installed in the vehicle, the radar having a field of view around the vehicle and obtaining radar data; wherein the controller is configured to confirm at least one object located around the vehicle based on processing the radar data, and integrate the at least one object based on the processed image data and the at least one object based on the processed radar data.

[0018] According to another aspect of the present disclosure, a method for assisted driving of a vehicle is provided, the method comprising the following steps: obtaining image data by a camera installed in the vehicle and having a field of view surrounding the vehicle; confirming at least one object located around the vehicle based on processing of the image data by a processor installed in the vehicle; updating the trajectory of the vehicle based on interference between the trajectory of the at least one object and the trajectory of the vehicle; and controlling at least one of a drive device, a braking device, or a steering device of the vehicle based on the updated trajectory of the vehicle.

[0019] The method may further include projecting the trajectory of the vehicle and the predicted trajectory of the at least one object onto a coordinate system, and confirming on the coordinate system whether the trajectory of the vehicle interferes with the trajectory of the at least one object.

[0020] The method may further include determining at least one of a lateral velocity, a lateral acceleration, an alignment angle, a longitudinal velocity, or a longitudinal acceleration of the at least one object based on processing the image data.

[0021] The method may further include confirming whether the at least one object changes lanes based on at least one of a lateral speed, a lateral acceleration, or an alignment angle of the at least one object.

[0022] The method may further include the steps of predicting a lateral trajectory of the at least one object based on at least one of a lateral velocity, a lateral acceleration, or an alignment angle of the at least one object, predicting a longitudinal trajectory of the at least one object based on a longitudinal velocity or a longitudinal acceleration of the at least one object, and predicting a trajectory of the at least one object based on the lateral trajectory and the longitudinal trajectory.

[0023] The method may further comprise the step of generating a trajectory of the vehicle based on a route to the destination obtained from a navigation device of the vehicle.

[0024] The method may further include predicting a trajectory of the vehicle based on at least one of a speed, an acceleration, or an angular velocity of the vehicle.

[0025] The method may further include the step of updating the trajectory of the vehicle to avoid the trajectory of the at least one object in response to the trajectory of the vehicle interfering with the trajectory of the at least one object.

[0026] The method may further include the steps of obtaining radar data by a radar installed in the vehicle and having a field of view around the vehicle; identifying, by a processor, at least one object located around the vehicle based on the processed radar data; and integrating, by the processor, the at least one object based on the processed image data and the at least one object based on the processed radar data.

[0027] According to another aspect of the present disclosure, a computer-readable storage medium storing a program for executing a method for assisting driving of a vehicle is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] These and / or other aspects of the present disclosure will become more apparent and easier to understand from the following description of embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a view illustrating the configuration of a vehicle and a driver assistance device according to an embodiment of the present disclosure;

[0030] Figure 2 is a diagram illustrating fields of view of a camera, a radar, and a lidar included in a driver assistance device according to an embodiment of the present disclosure;

[0031] Figure 3 is a diagram illustrating functional modules of a controller included in a driver assistance device according to an embodiment of the present disclosure;

[0032] Figure 4 This is an example Figure 3 A view of the prediction module shown;

[0033] Figures 5A to 5C is a view illustrating an object trajectory of an object and an ego trajectory of a vehicle; and

[0034] Figure 6 are views illustrating the operation of the driver assistance device according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Throughout this specification, like numbers refer to like elements. Not all elements of the embodiments of the present disclosure will be described, and descriptions of contents that are well known in the art or that overlap with each other in the embodiments will be omitted.

[0036] It should also be understood that the term "connect" or its derivatives refer to both direct and indirect connections, and indirect connections include connections through wireless communication networks.

[0037] It should also be understood that, unless the context clearly indicates otherwise, the term “comprise” when used in this document specifies the presence of specified features, elements, steps, operations, components and / or components, but does not preclude the presence or addition of one or more other features, elements, steps, operations, elements, components and / or combinations thereof.

[0038] In addition, when it is stated that a component is "on another component," the component may be directly on the other component, or a third component may be interposed between them.

[0039] Terms such as "unit", "group", "block", "component", and "module" used in the specification may be implemented in software or hardware. Terms such as "unit", "group", "block", "component", and "module" may refer to a unit that processes at least one function or operation. In addition, terms such as "unit", "group", "block", "component", and "module" are used for at least one piece of hardware, such as a field programmable gate array (FPGA) / application specific integrated circuit (ASIC), or at least one software or processor stored in a memory.

[0040] Although terms “first,” “second,” “A,” “B,” etc. may be used to describe various components, the terms do not limit the corresponding components but are used merely for the purpose of distinguishing one component from another.

[0041] As used herein, descriptions in the singular are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0042] The numbers used for the method steps are used only for convenience of description and do not limit the order of the steps. Therefore, unless the context clearly indicates otherwise, the written order can be specifically practiced in other ways.

[0043] Hereinafter, the operation principle and embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0044] Figure 1 is a view illustrating a configuration of a vehicle according to an embodiment of the present disclosure. Figure 2 is a diagram illustrating fields of view of a camera, a radar, and a light detection and ranging (lidar) included in an apparatus for assisting driving of a vehicle according to an embodiment of the present disclosure.

[0045] like Figure 1As shown, vehicle 1 may include: a navigation device 10, a drive device 20, a brake device 30, a steering device 40, a display device 50, an audio device 60, and / or a driver assistance device 100 (also referred to as an auxiliary driving device). In addition, vehicle 1 may also include sensors 91, 92, and 93 for detecting the dynamics of vehicle 1. For example, vehicle 1 may also include: a vehicle speed sensor 91 for detecting the longitudinal speed of vehicle 1, an acceleration sensor 92 for detecting the longitudinal and lateral accelerations of vehicle 1, and / or a gyro sensor 93 for detecting the yaw rate, roll rate, and pitch rate of vehicle 1.

[0046] The sensors can communicate with each other via the vehicle's communication network (NT). For example, the electronic devices 10, 20, 30, 40, 50, 60, 91, 92, 93, and 100 included in the vehicle 1 can exchange data via Ethernet, Media Oriented Systems Transport (MOST), Flexray, Controller Area Network (CAN), Local Interconnect Network (LIN), etc.

[0047] The navigation device 10 can generate a route to a destination input by the driver and provide the generated route to the driver. The navigation device 10 can receive a GNSS signal from a global navigation satellite system (GNSS) and confirm the absolute position (coordinates) of the vehicle 1 based on the GNSS signal. The navigation device 10 can generate a route to the destination based on the position (coordinates) of the destination input by the driver and the current position (coordinates) of the vehicle 1.

[0048] The navigation device 10 can provide the driver assistance device 100 with the location information and map data of the vehicle 1. Furthermore, the navigation device 10 can provide the driver assistance device 100 with information about the route to the destination. For example, the navigation device 10 can provide the driver assistance device 100 with information such as the distance to a road entrance where the vehicle 1 enters a new road or the distance from a road currently traveled by the vehicle 1 to a road exit.

[0049] The drive device 20 enables the vehicle 1 to move and may include, for example, an engine, an engine management system (EMS), a transmission, and a transmission control unit (TCU). The engine generates power for driving the vehicle 1, and the EMS controls the engine in response to the driver's acceleration intent via the accelerator pedal or a request from the driver assistance device 100. The transmission decelerates and transmits the power generated by the engine to the vehicle's wheels, and the TCU controls the transmission in response to the driver's shift commands via the shift lever and / or a request from the driver assistance device 100.

[0050] The braking device 30 stops the vehicle 1 and may include, for example, a brake caliper and an electronic brake control module (EBCM). The brake caliper can decelerate or stop the vehicle 1 by using friction with the brake disc, and the EBCM can control the brake caliper in response to the driver's intention to brake via the brake pedal and / or a request from the driver assistance device 100. For example, the EBCM receives a deceleration request including a deceleration degree from the driver assistance device 100 and can electrically or hydraulically control the brake caliper according to the requested deceleration degree to decelerate the vehicle 1.

[0051] The steering device 40 may include an electronic power steering (EPS) module. The steering device 40 may change the direction of travel of the vehicle 1, and the EPS module assists the operation of the steering device 40 so that the driver can easily manipulate the steering wheel in response to the driver's intention to steer through the steering wheel. In addition, the EPS module may control the steering device in response to a request from the driver assistance device 100. For example, the EPS module may receive a steering request including a steering torque from the driver assistance device 100 and control the steering device to steer the vehicle 1 according to the requested steering torque.

[0052] The display device 50 may include a cluster, a head-up display, a fascia monitor, etc., and may provide the driver with various information and entertainment through images and sounds. For example, the display device 50 may provide the driver with driving information and warning messages of the vehicle 1.

[0053] The audio device 60 may include a plurality of speakers and may provide the driver with various information and entertainment through sound. For example, the audio device 60 may provide the driver with driving information of the vehicle 1, warning messages, and the like.

[0054] The driver assistance device 100 can communicate with the navigation device 10, the plurality of sensors 91, 92, and 93, the drive device 20, the brake device 30, the steering device 40, the display device 50, and the audio device 60 via the NT. The driver assistance device 100 can receive information about the route to the destination and position information of the vehicle 1 from the navigation device 10, and obtain information about the vehicle speed, acceleration, and / or angular velocity of the vehicle 1 from the plurality of sensors 91, 92, and 93.

[0055] The driver assistance device 100 can provide various safety functions for the driver. For example, the driver assistance device 100 can include lane departure warning (LDW), lane keeping assist (LKAS), high beam assist (HBA), automatic emergency braking (AEB), traffic sign recognition (TSR), adaptive cruise control (ACC), and blind spot detection (BSD).

[0056] The driver assistance device 100 may include: a camera 110, a radar 120, a laser radar 130, and a controller 140. The driver assistance device 100 is not limited to Figure 1 For example, in the example of the driver assistance device Figure 1 In the illustrated driver assistance device 100 , at least one detection device among the camera 110 , the radar 120 , and the laser radar 130 is omitted, or various detection devices capable of detecting surrounding objects of the vehicle 1 may be added.

[0057] The camera 110, the radar 120, the laser radar 130, and the controller 140 may be provided separately from one another. For example, the controller 140 may be installed in a housing separate from the housings of the camera 110, the radar 120, and the laser radar 130. The controller 140 may exchange data with the camera 110, the radar 120, or the laser radar 130 via a broadband network.

[0058] The camera 110 can capture the surrounding environment of the vehicle 1 and obtain image data around the vehicle 1. For example, the camera 110 can be used as Figure 2 It is shown mounted on the front windshield of the vehicle 1 and may have a field of view 110 a facing forward of the vehicle 1 .

[0059] The camera 110 may include a plurality of lenses and an image sensor. The image sensor may include a plurality of photodiodes that convert light into electrical signals, and the plurality of photodiodes may be arranged in a two-dimensional matrix.

[0060] The image data may include information about other vehicles, pedestrians, cyclists, or lane markings (markings for distinguishing lanes) located around the vehicle 1 .

[0061] The camera 110 may include a graphics processor that processes image data and may detect objects around the vehicle 1 based on the processed image data. The camera 110 may, for example, use image processing to generate a track representing an object and classify the track. For example, the camera 110 may determine whether the track represents another vehicle, a pedestrian, a cyclist, etc.

[0062] The camera 110 may be electrically connected to the controller 140. For example, the camera 110 may be connected to the controller 140 via an NT, connected to the controller 140 via a hard wire, or connected to the controller 140 via a printed circuit board (PCB). The camera 110 may transmit image data (or the position and classification of the track) around the vehicle 1 to the controller 140.

[0063] The radar 120 may transmit a transmitted wave to the surroundings of the vehicle 1 and detect surrounding objects of the vehicle 1 based on the reflected wave reflected from the surrounding objects. Figure 2 It is shown mounted on the grille or bumper of the vehicle 1 and has a sensing field 120 a facing the front of the vehicle 1 .

[0064] The radar 120 may include a transmitting antenna (or a transmitting antenna array) that radiates transmission waves toward the surroundings of the vehicle 1 , and a receiving antenna (or a receiving antenna array) that receives reflected waves reflected by an object.

[0065] The radar 120 may obtain radar data from the transmission wave transmitted by the transmission antenna and the reflected wave received by the reception antenna. The radar data may include position information (eg, distance information) and / or speed information of an object located in front of the vehicle 1.

[0066] Radar 120 may include a signal processor to process radar data and may generate a trajectory representing an object by clustering reflection points from reflected waves. Radar 120 may determine the distance to the trajectory based on the time difference between the transmission time of the transmitted wave and the reception time of the reflected wave (in other words, the time it takes for the radio wave to travel from transmission to reception). Furthermore, radar 120 may determine the relative velocity of the trajectory based on the frequency difference between the transmitted wave and the reflected wave.

[0067] The radar 120 may be connected to the controller 140 , for example, through an NT or hard wire or a PCB, and transmit radar data (or a distance from a track and a relative speed) to the controller 140 .

[0068] The laser radar 130 may emit light (eg, infrared rays) to the surrounding environment of the vehicle 1 and detect surrounding objects of the vehicle 1 based on the reflected light reflected from the surrounding objects. Figure 2 It is shown mounted on the roof of the vehicle 1 and may have a field of view 130a facing in all directions around the vehicle 1 .

[0069] The laser radar 130 may include a light source (e.g., a light-emitting diode, a light-emitting diode array, a laser diode, or a laser diode array) that emits light (e.g., infrared light), and an optical sensor (e.g., a photodiode or a photodiode array) that receives light (e.g., infrared light). Furthermore, if necessary, the laser radar 130 may further include a drive device for rotating the light source and / or the light sensor.

[0070] When the light source and / or the light sensor rotates, the lidar 130 may emit light through the light source and receive light reflected from the object through the light sensor, thereby obtaining lidar data.

[0071] The lidar data may include the relative position of the vehicle 1 (the distance from the surrounding objects and / or the direction of the surrounding objects) and / or the relative speed of the surrounding objects. The lidar 130 may include a signal processor capable of processing lidar data, and may generate a trajectory representing the object by clustering reflection points by reflected light. The lidar 130 may obtain the distance to the object based on the time difference between the light emission time and the light reception time (in other words, the time it takes for the light to travel from emission to reception). Moreover, the lidar 130 may obtain the direction (or angle) of the object relative to the driving direction of the vehicle 1 based on the direction in which the light source emits light when the light sensor receives the reflected light.

[0072] The laser radar 130 may be connected to the controller 140 , for example, via an NT or hard wire or PCB, and transmit laser radar data (or relative position and relative speed of a track) to the controller 140 .

[0073] The controller 140 may be electrically connected to the camera 110, the radar 120, and / or the lidar 130. Furthermore, the controller 140 may be connected to the navigation device 10, the driving device 20, the braking device 30, the steering device 40, the display device 50, and the audio device 60, and / or the plurality of sensors 91, 92, and 93 via the NT.

[0074] The controller 140 may process image data from the camera 110 , radar data from the radar 120 , and / or lidar data from the lidar 130 , and provide control signals to the drive device 20 , the braking device 30 , and / or the steering device 40 .

[0075] The controller 140 may include a processor 141 and a memory 142 .

[0076] The processor 141 may process the image data from the camera 110, the radar data from the radar 120, and / or the lidar data from the lidar 130. Based on the processing of the image data from the camera 110, the radar data from the radar 120, and / or the lidar data from the lidar 130, the processor 141 may generate a drive signal, a brake signal, and / or a steering signal for controlling the drive device 10, the brake device 20, and / or the steering device 30, respectively.

[0077] For example, the processor 141 may include: an image processor that processes image data from the camera 110, a signal processor that processes radar data from the radar 120 and / or lidar data from the lidar 130, or a microcontroller unit (MCU) that generates driving / braking / steering signals.

[0078] The memory 142 may store programs and / or data for the processor 141 to process image data, radar data, and / or lidar data. The memory 142 may also store programs and / or data for the processor 141 to generate driving / braking / steering signals.

[0079] The memory 142 can temporarily store image data received from the camera 110, radar data received from the radar 120, and / or lidar data received from the lidar 130, and temporarily store the processing results of the processor 141 on the image data, radar data and / or lidar data.

[0080] The memory 142 may be not only a volatile memory such as a static random access memory (S-RAM) and a dynamic random access memory (D-RAM), but also a nonvolatile memory such as a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and the like.

[0081] As described above, the controller 140 may provide a driving signal, a braking signal, or a steering signal based on the image data of the camera 110 , the radar data of the radar 120 , or the lidar data of the lidar 130 .

[0082] The specific operation of the driver assistance device 100 will be described in more detail below.

[0083] Figure 3 is a view illustrating functional modules of a controller included in a driver assistance device according to an embodiment of the present disclosure. Figure 4 This is an example Figure 3 A view of the prediction module shown. Figure 5A It is a diagram illustrating an example in which any other vehicle changes lanes from another lane to the lane of the own vehicle. Figure 5B It is a diagram illustrating an example in which any other vehicle changes lanes from the lane of the own vehicle to another lane. Figure 5C is a view illustrating an example of a lane change of a vehicle to any other vehicle.

[0084] like Figure 3 、 Figure 4 as well as Figures 5A to 5C As shown, the controller 140 may functionally include multiple modules. Each of the modules may be a hardware module (e.g., ASIC or FPGA) included in the processor 141 or a software module (e.g., application or data) stored in the memory 142.

[0085] The controller 140 may include a sensor fusion module 210 , a prediction module 220 , a positioning module 230 , a trajectory planning module 240 , and a control module 250 .

[0086] The sensor fusion module 210 of the controller 140 may detect surrounding objects of the vehicle 1 by fusing image data of the camera 110 , radar data of the radar 120 , and lidar data of the lidar 130 .

[0087] The sensor fusion module 210 can obtain the relative position (angle relative to the direction of travel) of the camera track and / or the classification of the camera track (e.g., whether the object is any other vehicle, pedestrian, or cyclist, etc.) from the camera 110. The sensor fusion module 210 can obtain the relative position (distance from the vehicle) and / or relative speed of the radar track from the radar 120. Furthermore, the sensor fusion module 210 can obtain the relative position (distance from the vehicle and / or angle relative to the direction of travel) and / or relative speed of the lidar track from the lidar 130.

[0088] The sensor fusion module 210 may match the camera trajectory, the radar trajectory, and the lidar trajectory, and obtain a common trajectory based on the matching results. For example, the sensor fusion module 210 may identify overlapping common trajectories among the camera trajectory, the radar trajectory, and the lidar trajectory based on the position information of the camera trajectory, the radar trajectory, and the lidar trajectory.

[0089] Furthermore, the sensor fusion module 210 may integrate information about camera tracks, information about radar tracks, and / or information about lidar tracks. The sensor fusion module 210 may integrate common track information (e.g., position information and velocity information) obtained from the camera 110, common track information (e.g., position information and velocity information) obtained from the radar 120, and common track information (e.g., position information and velocity information) obtained from the lidar 130.

[0090] The sensor fusion module 210 may provide the common trajectory and information about the common trajectory (eg, information about classification, position, and velocity) to the prediction module 220 .

[0091] The prediction module 220 of the controller 140 may receive information about the common track from the sensor fusion module 210 and predict the trajectory of the track.

[0092] like Figure 4 As shown, the prediction module 220 may perform object selection 221 , feature estimation 222 , lane change prediction 223 , and / or object trajectory prediction 224 for each of the plurality of trajectories.

[0093] The object selection 221 may include a process of selecting a trajectory related to the travel of the vehicle 1 from among the plurality of common trajectories. For example, in the object selection 221, the controller 140 may select a trajectory located in the same lane as the lane in which the vehicle 1 is traveling or a trajectory located in a lane adjacent to the lane in which the vehicle 1 is traveling from among the plurality of common trajectories.

[0094] Feature estimation 222 may include a process for obtaining information that can predict lane changes of the selected trajectory from the selected trajectory information. For example, in feature estimation 222, controller 140 may obtain the lateral position, lateral velocity, alignment angle, etc. of the selected trajectory from the selected trajectory information. Here, the alignment angle may refer to the angle between the direction of the lane and the direction of the trajectory movement.

[0095] Lane change prediction 223 may include a process of predicting whether the lane of a track will change based on information about the track selected by feature estimation 222 (eg, the track's lateral position, lateral velocity, alignment angle, etc.).

[0096] In lane change prediction 223 , the controller 140 may predict whether to change lanes of the trajectory based on the lateral position, lateral velocity, and / or alignment angle of the trajectory.

[0097] For example, Figure 5A As shown, the prediction module 220 of the controller 140 can obtain the lateral position, lateral speed, and / or alignment angle of the object 2 traveling in the lane adjacent to the lane in which the vehicle 1 is traveling. The prediction module 220 can predict a first object trajectory OT1 based on the selected information about the object 2, in which the object 2 changes lanes from the adjacent lane to the lane in which the vehicle 1 is traveling.

[0098] Moreover, if Figure 5BAs shown, the prediction module 220 of the controller 140 can obtain the lateral position, lateral speed, and / or alignment angle of the object 3 traveling in the same lane as the vehicle 1. The controller 140 can predict a second object trajectory OT2 based on the selected information about the object 3, wherein the object 3 changes lanes from the lane in which the vehicle 1 is traveling to an adjacent lane.

[0099] Based on information about each of the trajectories, object trajectory prediction 224 may include a lateral trajectory prediction for predicting the lateral trajectory of the object corresponding to the trajectory and a longitudinal trajectory prediction for predicting the longitudinal trajectory of the object. For example, in object trajectory prediction 224, prediction module 220 may predict the lateral trajectory of the object based on the lateral position, lateral velocity, and / or alignment angle of the trajectory. Furthermore, prediction module 220 may predict the longitudinal trajectory of the object based on the longitudinal position and / or longitudinal velocity of the trajectory.

[0100] In the object trajectory prediction 224 , the controller 140 may predict the trajectory of each of the objects based on the transverse trajectory and the longitudinal trajectory of each of the objects.

[0101] For example, Figure 5C As shown, the prediction module 220 can obtain the longitudinal position and longitudinal speed of the object 4 traveling in the lane adjacent to the lane in which the vehicle 1 is traveling. Based on the information about the object 4, the prediction module 220 can predict a third object trajectory OT3 that the object 4 would travel if it did not change lanes into the lane in which the vehicle 1 is traveling.

[0102] In this way, the prediction module 220 may provide the predicted trajectory of each of the objects to the trajectory planning module 240 .

[0103] The positioning module 230 of the controller 140 can obtain the location information and map data of the vehicle 1 from the navigation device 10. The positioning module 230 can confirm the location of the vehicle 1 based on the location information and map data of the vehicle 1. In other words, the controller 140 can confirm the absolute coordinates of the vehicle 1. The positioning module 230 can provide the map data and information about the location of the vehicle 1 to the trajectory planning module 240.

[0104] The trajectory planning module 240 of the controller 140 may obtain the position and predicted trajectory of the object from the prediction module 220 , and obtain the position and map data of the vehicle 1 from the positioning module 230 .

[0105] The trajectory planning module 240 may project the vehicle 1 and / or the object onto the map data based on the position of the object and the position of the vehicle 1. For example, the trajectory planning module 240 may project the vehicle 1 onto the map data based on the position information of the vehicle 1, and project the object onto the map data based on the relative positions and / or predicted trajectories of objects surrounding the vehicle 1.

[0106] The trajectory planning module 240 may determine whether the vehicle 1 and the object will collide based on the predicted trajectory of the object, and generate a trajectory of the vehicle 1 for avoiding the collision with the object.

[0107] The trajectory planning module 240 can obtain information about the dynamics of the vehicle 1, such as vehicle speed, acceleration, and / or angular velocity, from the plurality of sensors 91, 92, and 93. The trajectory planning module 240 can predict the trajectory of the vehicle 1 based on the vehicle speed, acceleration, and / or angular velocity of the vehicle 1. For example, the trajectory planning module 240 of the controller 140 can predict a first ego trajectory ET1 for the vehicle 1 to continue traveling along the current lane.

[0108] Furthermore, the trajectory planning module 240 may obtain information about a route from the navigation device 10 to the destination. The trajectory planning module 240 may generate a trajectory of the vehicle 1 based on the route to the destination. For example, Figure 5B and Figure 5C As shown, the trajectory planning module 240 may generate a second ego trajectory ET2 and a third ego trajectory ET3 , in which the vehicle 1 changes lanes to an adjacent lane based on a route to a destination.

[0109] The trajectory planning module 240 may determine whether the predicted object trajectories OT1 , OT2 , and OT3 of the objects 2 , 3 , and 4 overlap or interfere with the ego trajectories ET1 , ET2 , and ET3 of the vehicle 1 .

[0110] As described above, the trajectory planning module 240 may project the vehicle 1 onto the map and project the objects 2, 3, and 4. The trajectory planning module 240 may project the ego trajectories ET1, ET2, and ET3 of the vehicle 1, the predicted object trajectories OT1, OT2, and OT3 of the objects 2, 3, and 4, as well as the vehicle 1 and the objects 2, 3, and 4 onto the map. The trajectory planning module 240 may confirm whether the ego trajectories ET1, ET2, and ET3 of the vehicle 1 overlap or interfere with the predicted object trajectories OT1, OT2, and OT3 of the objects 2, 3, and 4 on the coordinate system of the map.

[0111] The trajectory planning module 240 can generate a new ego trajectory of the vehicle 1 that does not interfere with the predicted object trajectories OT1, OT2, and OT3 of objects 2, 3, and 4 based on the overlap or interference between the ego trajectory ET1, ET2, and ET3 of the vehicle 1 and the predicted object trajectories OT1, OT2, and OT3 of objects 2, 3, and 4.

[0112] For example, Figure 5A As shown, the trajectory planning module 240 can generate a fourth ego trajectory ET4 that does not interfere with the first object trajectory OT1. In the fourth ego trajectory ET4, the travel distance is shorter than that of the first ego trajectory ET1, and the vehicle 1 needs to decelerate.

[0113] Moreover, if Figure 5B As shown, the trajectory planning module 240 can generate a fifth ego trajectory ET5 that does not interfere with the second object trajectory OT2. In the fifth ego trajectory ET5, compared with the second ego trajectory ET2, it is required to travel along the current lane without changing lanes.

[0114] Moreover, if Figure 5C As shown, the trajectory planning module 240 can generate a sixth ego trajectory ET6 that does not interfere with the third object trajectory OT3. In the sixth ego trajectory ET6, compared with the third ego trajectory ET3, it is necessary to travel along the current lane and decelerate without changing lanes.

[0115] The control module 250 of the controller 140 may include a driving control for controlling the driving device 20 , a braking control for controlling the braking device 30 , and a steering control for controlling the steering device 40 .

[0116] The control module 250 may generate a driving signal, a braking signal, or a steering signal to follow the selected ego trajectories ET4 , ET5 , and ET6 .

[0117] For example, the control module 250 may provide a braking signal to the braking device 30 to decelerate the vehicle 1 to follow the Figure 5A The control module 250 can provide a driving signal to the driving device 20 so that the vehicle 1 maintains the current speed in order to follow the fourth self-trajectory ET4. Figure 5B Furthermore, the control module 250 can provide a braking signal to the braking device 30 so that the vehicle 1 decelerates to follow the fifth self-trajectory ET5. Figure 5C The sixth ego trajectory ET6 is shown.

[0118] As described above, the driver assistance device 100 can obtain information about surrounding objects and predict the trajectories of the surrounding objects. The driver assistance device 100 compares the predicted (or generated) trajectory of the vehicle 1 with the predicted trajectories of the surrounding objects on a coordinate system, and thereby determines whether the predicted (or generated) trajectory of the vehicle 1 interferes with the predicted trajectories of the surrounding objects. The driver assistance device 100 can change the trajectory of the vehicle 1 based on whether the predicted (or generated) trajectory of the vehicle 1 interferes with the predicted trajectories of the surrounding objects.

[0119] Figure 6 are views illustrating the operation of the driver assistance device according to the embodiment of the present disclosure.

[0120] Reference Figure 6 , describes the operation 1000 of the driver assistance device 100 .

[0121] The driver assistance device 100 performs sensor fusion and trajectory selection ( 1010 ).

[0122] The controller 140 can detect surrounding objects of the vehicle 1 by fusing the image data of the camera 110, the radar data of the radar 120, and the lidar data of the lidar 130. The controller 140 can match the camera trajectory, the radar trajectory, and the lidar trajectory with each other and obtain a common trajectory based on the matching results.

[0123] The controller 140 may select a trajectory related to the travel of the vehicle 1 from among the plurality of common trajectories. For example, the controller 140 may select a trajectory located in the same lane as the lane in which the vehicle 1 is traveling or a trajectory located in a lane adjacent to the lane in which the vehicle 1 is traveling from among the plurality of common trajectories.

[0124] The driver assistance device 100 evaluates characteristics of the trajectory ( 1020 ).

[0125] The controller 140 may obtain information for predicting lane changes of the trajectory from the information about the selected trajectory. For example, the driver assistance device 100 may obtain the lateral position, lateral speed, alignment angle, etc. of the trajectory from the information about the selected trajectory.

[0126] The driver assistance device 100 predicts a lane change of the trajectory ( 1030 ).

[0127] The controller 140 may predict whether to change the lane of the trajectory based on the selected trajectory information (eg, the lateral position, lateral speed, alignment angle, etc. of the trajectory).

[0128] The driver assistance device 100 predicts a trajectory of the track ( 1040 ).

[0129] The controller 140 may predict the lateral trajectory of the object corresponding to each of the trajectories based on information about the trajectories, and may also predict the longitudinal trajectory of the object. For example, the controller 140 may predict the lateral trajectory of the object based on the lateral position, lateral velocity, and / or alignment angle of the trajectory. Furthermore, the controller 140 may predict the longitudinal trajectory of the object based on the longitudinal position and / or longitudinal velocity of the trajectory.

[0130] The controller 140 may predict the trajectory of each of the objects based on the transverse trajectory and the longitudinal trajectory of each of the objects.

[0131] The driver assistance device 100 confirms whether the trajectory of the vehicle 1 interferes with the trajectory of the track ( 1050 ).

[0132] The controller 140 may project the vehicle 1 and / or the object onto the map data based on the position of the track and the position of the vehicle 1. Furthermore, the controller 140 may project the ego trajectory of the vehicle 1 and the predicted trajectory of the track together with the vehicle 1 and the track onto the map.

[0133] The controller 140 may confirm whether the trajectory of the vehicle 1 overlaps or interferes with the predicted trajectory on the coordinate system of the map.

[0134] When the trajectory of the vehicle 1 interferes with the trajectory of the track (Yes in 1050 ), the driver assistance device 100 generates a new trajectory of the vehicle 1 based on the predicted trajectory of the track ( 1055 ).

[0135] The controller 140 may generate a trajectory of the vehicle 1 that does not interfere with the trajectory of the trajectory based on the trajectory of the vehicle 1 interfering with the trajectory of the trajectory.

[0136] Furthermore, the controller 140 may confirm whether the new trajectory of the vehicle 1 interferes with the trajectory of the track.

[0137] When the trajectory of the vehicle 1 does not interfere with the trajectory of the track (No in 1050 ), the driver assistance device 100 controls the vehicle 1 to travel along the generated trajectory ( 1060 ).

[0138] The controller 140 may generate a driving signal, a braking signal, or a steering signal based on the trajectory of the vehicle 1 , and provide the driving signal, the braking signal, or the steering signal to the driving device 20 , the braking device 30 , or the steering device 40 .

[0139] In this way, the driver assistance device 100 compares the predicted trajectory of the surrounding objects with the predicted (or generated) trajectory of the vehicle 1 on one coordinate system, and can thereby generate a trajectory of the vehicle 1 that does not interfere with the predicted trajectory.

[0140] As is apparent from the above, embodiments of the present disclosure may provide an apparatus and method for assisting driving of a vehicle that is capable of predicting trajectories of a plurality of objects located near the vehicle.

[0141] Therefore, the device that assists vehicle driving can predict and avoid various dangerous situations caused by objects other than the target.

[0142] On the other hand, the embodiments of the present disclosure can be implemented in the form of a recording medium storing instructions that can be executed by a computer. The instructions can be stored in the form of program code, and when executed by a processor, a program module can be created to perform the operations of the embodiments of the present disclosure. The recording medium can be implemented as a computer-readable recording medium.

[0143] The computer-readable recording medium includes any type of recording medium in which computer-readable instructions are stored, such as a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like.

[0144] The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, non-transitory means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), and the term does not distinguish between storing data semi-permanently in the storage medium and temporarily storing data in the storage medium. For example, a non-transitory storage medium may include a cache in which data is temporarily stored.

[0145] The embodiments of the present disclosure have been described above with reference to the accompanying drawings. It will be apparent to those skilled in the art that the present disclosure may be practiced in other forms than those described above without changing the technical concept or basic features of the present disclosure. The above embodiments are provided as examples only and should not be construed in a limiting sense.

[0146] CROSS-REFERENCE TO RELATED APPLICATIONS

[0147] This application is based upon and claims the benefit of priority from Korean Patent Application No. 10-2021-0051164 filed on April 20, 2021, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. A device for assisting vehicle driving, comprising: a camera mounted in the vehicle, the camera having a field of view around the vehicle and acquiring image data; as well as a controller configured to process the image data; Wherein, the controller is configured to: generating a vehicle trajectory of the vehicle based on a route to a destination obtained from a navigation device of the vehicle, determining whether the vehicle is to change its lane based on the own vehicle trajectory, identifying at least one object located around the vehicle based on processing the image data, determining whether the at least one object is to change the lane of the at least one object based on the lateral position, lateral velocity, and alignment angle of the at least one object, the alignment angle being an angle between a direction of the lane of the at least one object and a direction in which the at least one object is moving, predicting a lateral trajectory of the at least one object based on at least one of the lateral velocity, the lateral acceleration, and the alignment angle of the at least one object, predicting a longitudinal trajectory of the at least one object based on a longitudinal velocity and a longitudinal acceleration of the at least one object, predicting an object trajectory of the at least one object based on the transverse trajectory and the longitudinal trajectory, Project the vehicle trajectory and the predicted object trajectory onto a coordinate system. when it is determined that the vehicle is to change the lane of the vehicle to a target lane and the at least one object is to change the lane of the at least one object to the same target lane, confirming whether the host vehicle trajectory and the object trajectory interfere with each other on the one coordinate system, updating the host trajectory of the vehicle based on interference between the predicted object trajectory of the at least one object and the host trajectory of the vehicle, and At least one of a driving device, a braking device, and a steering device of the vehicle is controlled based on the updated trajectory of the vehicle.

2. The device according to claim 1, wherein The controller is further configured to: At least one of the lateral velocity, the lateral acceleration, the alignment angle, the longitudinal velocity, or the longitudinal acceleration of the at least one object is determined based on processing the image data.

3. The device according to claim 1, wherein The controller is further configured to predict the own vehicle trajectory of the vehicle based on at least one of the speed, acceleration, or angular velocity of the vehicle.

4. The apparatus according to claim 1, wherein The controller is further configured to, in response to the host vehicle trajectory of the vehicle interfering with the object trajectory of the at least one object, update the host vehicle trajectory of the vehicle to avoid the object trajectory of the at least one object.

5. The apparatus of claim 1 , further comprising a radar installed in the vehicle, the radar having a field of view around the vehicle and acquiring radar data; in, The controller is further configured to: identifying at least one object located around the vehicle based on processing the radar data, and The at least one object based on processing the image data and the at least one object based on processing the radar data are integrated.

6. A method for assisting driving of a vehicle, the method comprising the following steps: obtaining image data by a camera mounted in the vehicle and having a field of view surrounding the vehicle; generating a vehicle trajectory of the vehicle based on a route to a destination obtained from a navigation device of the vehicle, determining whether the vehicle is to change its lane based on the own vehicle trajectory, identifying at least one object located around the vehicle based on processing of the image data by a processor installed in the vehicle; determining whether the at least one object is to change the lane of the at least one object based on the lateral position, lateral velocity, and alignment angle of the at least one object, the alignment angle being an angle between a direction of the lane of the at least one object and a direction in which the at least one object is moving, predicting a lateral trajectory of the at least one object based on at least one of the lateral velocity, the lateral acceleration, and the alignment angle of the at least one object, predicting a longitudinal trajectory of the at least one object based on a longitudinal velocity and a longitudinal acceleration of the at least one object, predicting an object trajectory of the at least one object based on the transverse trajectory and the longitudinal trajectory, Project the vehicle trajectory and the predicted object trajectory onto a coordinate system. when it is determined that the vehicle is to change the lane of the vehicle to a target lane and the at least one object is to change the lane of the at least one object to the same target lane, confirming whether the host vehicle trajectory and the object trajectory interfere with each other on the one coordinate system, updating the host trajectory of the vehicle based on interference between the predicted object trajectory of the at least one object and the host trajectory of the vehicle; as well as At least one of a driving device, a braking device, or a steering device of the vehicle is controlled based on the updated trajectory of the vehicle.

7. The method according to claim 6, further comprising the steps of: At least one of the lateral velocity, the lateral acceleration, the alignment angle, the longitudinal velocity, or the longitudinal acceleration of the at least one object is determined based on processing the image data.

8. The method according to claim 6, further comprising the steps of: The own vehicle trajectory of the vehicle is predicted based on at least one of the speed, acceleration, or angular velocity of the vehicle.

9. The method according to claim 6, further comprising the steps of: In response to the host vehicle trajectory of the vehicle interfering with the object trajectory of the at least one object, the host vehicle trajectory of the vehicle is updated to avoid the object trajectory of the at least one object.

10. The method according to claim 6, further comprising the steps of: obtaining radar data via a radar mounted in the vehicle and having a field of view around the vehicle; identifying, by the processor, at least one object located around the vehicle based on processing the radar data, and The at least one object based on processing the image data and the at least one object based on processing the radar data are integrated. 11 . A computer-readable storage medium storing a program for executing the method according to claim 6 .

Images