Apparatus and method for object tracking and apparatus and method for controlling vehicle using the same

Abstract

The present disclosure relates to an apparatus for tracking an object and a method thereof, and an apparatus for controlling a vehicle using the same and a method thereof. The apparatus may include a camera configured to obtain image data, a sensor configured to obtain a set of points, and a processor. The processor is configured to: detect an object external to the vehicle; determining a trajectory as a partial trajectory based on a determination that the trajectory is within a threshold distance from a reference trajectory generated based on the image data, the trajectory being generated based on information obtained by the sensor; determining an object trajectory including the partial trajectory; and generating a final trajectory indicating the position of the object by fusing the object trajectory and the reference trajectory; and outputting a signal indicative of the final trajectory.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority benefit of Korean Patent Application No. 10-2023-0133515 filed in the Korean Intellectual Property Office on October 6, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an apparatus for tracking an object and a method thereof, and an apparatus for controlling a vehicle using the same, and more particularly, to a technology for tracking an object using a sensor fusion method. Background Art

[0004] An autonomous vehicle is a vehicle that can drive itself without driver or passenger intervention. An autonomous driving system is a system that monitors and controls an autonomous vehicle so that it can drive itself. In a broad sense, an autonomous vehicle can refer to a vehicle that monitors the outside of the vehicle to assist the driver in driving and is equipped with various driving assistance devices based on the monitored external environment of the vehicle.

[0005] An autonomous vehicle or a vehicle equipped with a driving assistance device may detect an object by monitoring the exterior of the vehicle and drive based on a scene determined based on the detected object. In other words, autonomous driving or driving based on a driving assistance device may use a process for determining the type of an object outside the vehicle.

[0006] Sensors for monitoring the exterior of the vehicle may include cameras, radio detection and ranging (RADAR), light imaging detection and ranging (LIDAR), etc. Methods of detecting or tracking objects may be used by fusing the sensing results of two or more sensors to increase or maximize the advantages of each.

[0007] If the density of the point cloud obtained by LIDAR is not satisfactory in the process of fusing the information obtained by LIDAR through sensor fusion, the point cloud corresponding to the object may not be correctly reflected. In particular, because the density of the point cloud obtained by LIDAR is reduced if the object is far away or the reflectivity of the object is not satisfactory, the point cloud for the object may be missing.

[0008] As such, due to the lack of LIDAR information, the position of the object may not be accurately determined. Because the exact position of the object is not determined or is inaccurately determined, the reliability of driving according to autonomous driving or driving according to a driving assistance device may be reduced. Summary of the invention

[0009] According to the present disclosure, an apparatus includes a camera configured to obtain image data, a sensor configured to obtain a point set, and a processor, wherein the processor is configured to: detect an object outside a vehicle; determine the trajectory as a partial trajectory based on determining that the trajectory is within a threshold distance from a reference trajectory generated based on the image data, wherein the trajectory is generated based on information obtained by the sensor; determine an object trajectory including the partial trajectory; and generate a final trajectory indicating the position of the object by fusing the object trajectory and the reference trajectory; and output a signal indicating the final trajectory.

[0010] The apparatus, wherein the reference trajectory is obtained by representing the object in a top view in a coordinate system, wherein the object is detected from the image data.

[0011] The apparatus, wherein the trajectory is represented in a top view in a coordinate system based on a clustering state of a point set.

[0012] The apparatus, wherein the processor is configured to: determine a first reference line indicating a side surface of an object, wherein the object is in a reference trajectory; and determine whether a shortest distance between the first reference line and a center point of the trajectory is within a threshold distance.

[0013] The apparatus, wherein the processor is configured to determine, as the first reference line, a line segment indicating a side surface selected from a plurality of side surfaces of the object that is closer to the vehicle.

[0014] The device, wherein the processor is configured to: determine a straight line connecting one end of a front bumper of the object and one end of a rear bumper of the object as a first reference line; and compare the shortest distance between a center point of the trajectory and the first reference line with a threshold distance.

[0015] The apparatus, wherein the processor is configured to include the trajectory in the object trajectory based on determining that a center point of the trajectory is located between a second reference line indicating a front side of the object and a third reference line indicating a back side of the object.

[0016] The apparatus, wherein the processor is configured to increase the threshold distance as the reflectivity of the object decreases.

[0017] The apparatus wherein the processor is configured to increase the threshold distance as weather conditions in an area including the vehicle deteriorate.

[0018] The apparatus wherein the processor is configured to control the vehicle based on the signal indicative of the final trajectory.

[0019] According to the present disclosure, a method may include: generating a reference trajectory indicating a position of an object detected using a sensor other than a light imaging detection and ranging (LIDAR) sensor; determining the trajectory as a partial trajectory based on determining that the trajectory is within a threshold distance from the reference trajectory, wherein the trajectory is generated based on information obtained by the LIDAR sensor; and determining an object trajectory including the partial trajectory; generating a final trajectory indicating the position of the object by fusing the object trajectory and the reference trajectory, and outputting a signal indicating the final trajectory.

[0020] Generating the reference trajectory may include representing the object in a top view in a coordinate system, wherein the object is detected from the image data.

[0021] Determining the trajectory as a partial trajectory may include representing the trajectory in a top view in a coordinate system, the trajectory being determined based on a clustering state of a point set obtained by the LIDAR sensor.

[0022] Determining the trajectory as a partial trajectory may include determining a first reference line indicating a side surface of the object, wherein the object is in the reference trajectory; and determining whether a shortest distance between the first reference line and a center point of the trajectory is within a threshold distance.

[0023] Determining the first reference line may include determining, as the first reference line, a line segment indicating a side surface selected from a plurality of side surfaces of the object that is closer to the vehicle.

[0024] Determining the trajectory as a partial trajectory may include comparing a shortest distance between a center point of the trajectory and the first reference line to a threshold distance.

[0025] Determining the trajectory as a partial trajectory may include: obtaining a second reference line indicating the front side of the object based on the shortest distance between the center point of the trajectory and the first reference line being within a threshold distance; obtaining a third reference line indicating the back side of the object; and including the trajectory in the object trajectory based on the center point of the trajectory being between the second reference line and the third reference line.

[0026] The method may further include increasing the threshold distance as the reflectivity of the object decreases.

[0027] The method may also include increasing the threshold distance as weather conditions in an area including the vehicle deteriorate.

[0028] The method may also include controlling the vehicle based on the signal indicative of the final trajectory, wherein the location of the object is external to the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, features, and advantages of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings:

[0030] Figure 1 An example of a configuration of an object tracking device and a vehicle control device including the same according to an example of the present disclosure is shown;

[0031] Figure 2 An example of a vehicle equipped with a sensor device of an object tracking apparatus according to an example of the present disclosure is shown;

[0032] Figure 3 An example of the operation of a processor is shown;

[0033] Figure 4 An example of an object tracking method according to an example of the present disclosure is shown;

[0034] Figure 5 An example of an image represented by image data obtained by a camera is shown;

[0035] Figure 6 shows an example of a reference trajectory generated by a processor;

[0036] Figure 7 and Figure 8 An example of a method of determining whether a LIDAR track is included in a first area is shown;

[0037] Fig. 9 An example of a method of determining whether a LIDAR track belongs to a second area is shown;

[0038] Fig.10 An example of a LIDAR track that is within a reference track and a threshold distance is shown;

[0039] Fig.11 An example of a final trajectory according to a comparative example is shown;

[0040] Fig.12 shows an example of a final trajectory according to an example of the present disclosure; and

[0041] Fig.13 An example of a computing system according to an example of the present disclosure is shown. DETAILED DESCRIPTION

[0042] Hereinafter, some examples of the present disclosure will be described in detail with reference to the accompanying drawings. When adding reference numerals to the components of each of the accompanying drawings, it should be noted that the same components have the same reference numerals even if the same components are shown on another accompanying drawing. In addition, when describing examples of the present disclosure, detailed descriptions associated with well-known functions or configurations will be omitted if the detailed descriptions associated with the well-known functions or configurations may make the subject matter of the present disclosure unnecessarily obscure.

[0043] When describing the elements of the examples of the present disclosure, the terms first, second, A, B, (a), (b), etc. may be used in this article. These terms are only used to distinguish one element from another element, but do not limit the corresponding elements, regardless of the nature, order, or priority of the corresponding elements. In addition, unless otherwise defined, all terms including technical and scientific terms used herein should be interpreted as being customary in the field to which the present disclosure belongs. It should be understood that the terms used herein should be interpreted as having a meaning consistent with their meaning in the context of the present disclosure and the related art, and should not be interpreted in an idealized or overly formal sense, unless explicitly defined in this article.

[0044] In the following, reference will be made to Figures 1 to 13 Various examples of the present disclosure are described in detail.

[0045] Figure 1 An example of a configuration of an object tracking device and a vehicle control device including the same according to an example of the present disclosure is shown. Figure 2 An example of a vehicle equipped with a sensor device of an object tracking apparatus according to an example of the present disclosure is shown.

[0046] refer to Figure 1 and Figure 2 , a vehicle control apparatus according to an example of the present disclosure may include a sensor device 100 , a processor 200 , a memory 300 , a drive controller 400 , a communication device 500 , and a notification device 600 .

[0047] The sensor device 100 may include at least one of a camera 110 , a LIDAR 120 , or a RADAR 130 for detecting external objects of the vehicle VEH.

[0048] The camera 110 can be used to obtain an external image of the vehicle VEH, and can obtain a front image or a front outer image of the vehicle VEH. For example, the camera 110 can be placed around the front windshield to obtain a front image of the vehicle VEH.

[0049] The LIDAR 120 can be used to determine an object using a reflected wave of laser light reflected from an object after the laser light is emitted to the object, and can be implemented by a time-of-flight (TOF) method or a phase shift method. The LIDAR 120 can be installed to be exposed to the outside of the vehicle and can be placed around the front bumper or front grille.

[0050] The RADAR 130 may include an electromagnetic wave transmitting module and an electromagnetic wave receiving module. In view of the radio transmission principle, the RADAR 130 may be implemented by a pulse radar method or a continuous wave radar method. According to the signal waveform, the RADAR 130 may be implemented by a frequency modulated continuous wave (FMCW) method or a frequency shift keying (FSK) method in the continuous wave radar method. The RADAR 130 may include a front RADAR 131 located at the front center of the vehicle VEH, a front outer side RADAR 132 located at both ends of the front bumper, and a rear RADAR 133 located at the rear of the vehicle VEH.

[0051] The positions of the camera 110, LIDAR 120, and RADAR 130 may not be limited to Figure 2 The example shown in .

[0052] In addition to or in lieu of those shown in the accompanying drawings, the sensor device may include an ultrasonic sensor and an infrared sensor. The ultrasonic sensor may include an ultrasonic transmitting module and an ultrasonic receiving module. The ultrasonic sensor may detect an object based on ultrasonic waves, and may detect the position of the detected object, the distance to the detected object, and the relative speed. The ultrasonic sensor may be positioned at an appropriate position outside the vehicle to detect an object located at the front of the vehicle, the rear of the vehicle, or the side of the vehicle. The infrared sensor may include an infrared transmitting module and an infrared receiving module. The infrared sensor may detect an object based on infrared light, and may detect the position of the detected object, the distance to the detected object, and the relative speed. The infrared sensor may be positioned outside the vehicle to detect an object located at the front of the vehicle, the rear of the vehicle, or the side of the vehicle.

[0053] Additionally or alternatively, the sensor device 100 may further include a brake-pedal position sensor (BPS) and an accelerator position sensor (APS) that generate a speed control command for shifting the vehicle.

[0054] The BPS may output a BPS signal according to the degree of depression of a brake pedal provided in the vehicle. For example, the BPS signal may output data ranging from 0 to 100 according to the depression of the brake pedal. A value of 0 may indicate that the brake pedal is not depressed, and a value of 100 may indicate that the brake pedal is depressed to the maximum extent.

[0055] The APS may output an APS signal according to the degree of depression of an accelerator pedal provided in the vehicle. For example, the APS signal may output data ranging from 0 to 100 according to the depression of the accelerator pedal. A value of 0 may indicate that the accelerator pedal is not depressed, and a value of 100 may indicate that the accelerator pedal is depressed to the maximum extent.

[0056] Processor 200 may track objects outside the vehicle and may perform evasive driving based on the location of the objects. Figure 3, the operation of processor 200 is as follows.

[0057] Figure 3 An example of the operation of a processor is shown.

[0058] refer to Figure 3 , the processor 200 may generate a trajectory (eg, a LIDAR trajectory) based on a point set (eg, a point cloud) obtained by a sensor (eg, the LIDAR 120) (S1).

[0059] The LIDAR trajectory can be generated based on the clustering state of the point cloud. The LIDAR trajectory can be obtained by representing the shape and position of the object using a box on a plane parallel to the road surface in a coordinate system (e.g., a world or global coordinate system that defines the position and orientation of an object or point in a three-dimensional space (e.g., x, y, and z axes) or a higher dimensional space). In other words, the LIDAR trajectory can be represented in a top view in the world coordinate system. Figure 3 The example of FIG. 2 shows an example in which the processor 200 generates the LIDAR trajectory. However, the LIDAR trajectory may be generated by the LIDAR 120 or another processor.

[0060] In addition, the processor 200 may generate a reference trajectory RT based on the image data obtained by the camera 110 ( S2 ).

[0061] The reference trajectory may be obtained by representing the objects detected from the image data as a top view in a world coordinate system.

[0062] In addition, the processor 200 may determine the object trajectory based on the positional relationship between the reference trajectory and the LIDAR trajectory (S3).

[0063] The object track may include object candidate tracks and partial tracks.

[0064] The object candidate track may be determined based on an overlap level between the object candidate track and the reference track and a matching level of directions of the object candidate track and the reference track. In other words, the object candidate track is determined based on an overlap level of a certain level or more and a point cloud having a clustering degree that the point cloud is determined to be one object.

[0065] The partial trajectory may refer to a trajectory that can be estimated as an object trajectory in the fragmented LIDAR trajectory. The partial trajectory may be determined based on a positional relationship with a reference trajectory. According to an example, the processor 200 may determine a LIDAR trajectory within a predetermined threshold distance from the reference trajectory as a partial trajectory (e.g., a fragmented trajectory).

[0066] According to an example of the present disclosure, the position of an object may be accurately determined because the object trajectory is determined to include a partial trajectory in addition to or instead of the object candidate trajectory.

[0067] Furthermore, the processor 200 may generate a final trajectory by fusing or merging the object trajectory and the reference trajectory ( S4 ).

[0068] In order to perform at least one of the processes for generating the final trajectory, the processor 200 may include an artificial intelligence (AI) processor. The AI ​​processor may use a pre-stored program to learn a neural network. The neural network for detecting target vehicles and dangerous vehicles may be designed to simulate the structure of the human brain on a computer, and may include a plurality of network nodes, each of which has a weight, and a plurality of network nodes simulate neurons of a human neural network. A plurality of network nodes may exchange data according to each connection relationship, so that the neurons simulate the synaptic activity of neurons that exchange signals through synapses. The neural network may include a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes may exchange data according to a convolutional connection relationship while being located on different layers. Examples of neural network models may include various deep learning techniques, such as a deep neural network (DNN), a convolutional deep neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), and a deep Q network.

[0069] The memory 300 may store algorithms for the operations of the processor 200 and the AI ​​processor. The memory 300 may use a hard disk drive, a flash memory, an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a ferroelectric RAM (FRAM), a phase change RAM (PRAM), or a magnetic RAM (MRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate-SDRAM (DDR-SDRAM), etc.

[0070] The driving controller 400 may be used to control the steering and acceleration / deceleration of the vehicle in response to a control signal from the processor 200 , and may include a steering control module, an engine control module, a brake control module, and a transmission control module.

[0071] The steering control module may include a hydraulic power steering (HPS) system that controls steering using hydraulic pressure generated by a hydraulic pump, and a motor-driven power steering system (hereinafter referred to as "MDPS") that controls steering using the output torque of an electric motor.

[0072] The engine control module controls the acceleration of the vehicle as an actuator for controlling the engine of the vehicle. The engine control module can be implemented by an engine management system (EMS). The engine control module controls the driving torque of the engine according to the accelerator pedal position information output from the accelerator pedal position sensor. During automatic driving or driving according to a driving assistance device, the engine control module controls the output of the engine to follow the driving speed of the vehicle required by the processor 200.

[0073] The brake control module, as an actuator for controlling the deceleration of the vehicle, may be implemented by an electronic stability control (ESC). The brake control module controls the brake pressure in order to follow the target speed required by the processor 200. In other words, the brake control module controls the deceleration of the vehicle.

[0074] The transmission control module is an actuator for controlling the transmission of the vehicle, which can be implemented by shift-by-wire (SBW). The shift controller controls the transmission of the vehicle according to the gear position and the gear state range.

[0075] The communication device 500 may communicate with a user terminal, another vehicle, or an external server, and may receive weather information or vehicle information of surrounding vehicles.

[0076] The communication device 500 may support short-range communications using at least one of Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), ZigBee, near field communication (NFC), wireless fidelity (Wi-Fi), Wi-Fi Direct, or wireless universal serial bus (wireless USB) technologies.

[0077] The communication device 500 may include a global positioning system (GPS) module or a differential global positioning system (DGPS) module to obtain location information.

[0078] In addition, the communication device 500 may include a V2X communication module. The V2X communication module may include an RF circuit for a wireless communication protocol with a server (vehicle to infrastructure (V2I)), another vehicle (vehicle to vehicle (V2V)), or a pedestrian (vehicle to pedestrian (V2P)). The communication device 500 may receive sensing data obtained by a sensor device of another vehicle through the V2X communication module, and may provide the sensing data to the processor 200.

[0079] The notification device 600 may notify the occupant of the vehicle tracking situation and the vehicle avoidance situation through the processor 200. The notification device 600 may include a display, a speaker, and the like.

[0080] Figure 4 An example of an object tracking method according to an example of the present disclosure is shown. Figure 4 The process shown in Figure 1 and Figure 3 The object tracking method according to the example of the present disclosure will refer to Figure 1 and Figure 4 In the following, the examples described later will focus on Figure 5 The target vehicle detected in the image obtained by the camera shown in FIG.

[0081] In S410, the processor 200 may generate a reference trajectory based on the image data obtained by the camera 110. Figure 5 and Figure 6 , the reference trajectory is described as follows.

[0082] Figure 5 An example of an image represented by image data obtained by a camera is shown. Figure 6 An example of a reference trajectory generated by a processor is shown.

[0083] refer to Figure 5 , the processor 200 may detect any feature points fp1, fp3, and fp4 from the image data. The first feature point fp1 may indicate the left end of the front side of the target vehicle Vtg; the third feature point fp3 may indicate the left end of the rear side of the target vehicle Vtg; and the fourth feature point fp4 may be the right end of the rear side of the target vehicle Vtg.

[0084] The feature points fp1, fp3, and fp4 in the image data may include image coordinate information. The processor 200 may convert the image coordinates of the feature points fp1, fp3, and fp4 into world coordinates (e.g., x, y, z axes) in a world coordinate system. The world coordinate system can be used to represent the position of any point in the actual physical space. In the example of the present disclosure, the origin in the world coordinate system may be the point where a straight line perpendicular to the horizontal plane and passing through the center of the front bumper of the vehicle intersects the horizontal plane. In the world coordinate system, the x-axis may be the lateral direction of the vehicle's driving direction, the y-axis may be the vehicle's driving direction, and the z-axis is perpendicular to the x-axis and the y-axis. The processor 200 may convert the image coordinates into world coordinates using a homography matrix.

[0085] In addition, if Figure 6 As shown, the processor 200 may display points P1, P3, and P4 corresponding to feature points fp1, fp3, and fp4 on the xy plane of the world coordinate system. Figure 6 , the first point P1 can be obtained by representing the first feature point fp1 in world coordinates; the third point P3 can be obtained by representing the third feature point fp3 in world coordinates; and the fourth point P4 can be obtained by representing the fourth feature point fp4 in world coordinates.

[0086] Furthermore, the processor 200 may determine a second point P2 corresponding to the right end of the front side of the vehicle based on the feature points fp1 , fp3 , and fp4 .

[0087] Alternatively or additionally or alternatively, the processor 200 may calculate the first to fourth points P1, P2, P3, and P4 based on the center point CP1 of the target vehicle Vtg, the center point CP2 of the rear bumper of the target vehicle Vtg, the width W of the target vehicle Vtg, the length L of the target vehicle Vtg, and the heading angle θ of the target vehicle Vtg. In this specification, the heading angle θ of the target vehicle Vtg may refer to the inner angle between the driving direction HD of the target vehicle Vtg and the (+) direction of the x-axis.

[0088] The x-coordinate and the y-coordinate of each of the first to fourth points P1 , P2 , P3 , and P4 may be calculated as a trigonometric function based on the following Equation 1, for example.

[0089] [Equation 1]

[0090] x 1 =x C +0.5*L*cos(θ)-0.5*W*sin(θ)

[0091] y 1 =y C +0.5*L*sin(θ)+0.5*W*cos(θ)

[0092] x 2 =x C +0.5*L*cos(θ)+0.5*W*sin(θ)

[0093] y 2 =y C +0.5*L*sin(θ)-0.5*W*cos(θ)

[0094] x 3 =x C -0.5*L*cos(θ)-0.5*W*sin(θ)

[0095] y 3 =y C -0.5*L*sin(θ)+0.5*W*cos(θ)

[0096] x 4 =x C -0.5*L*cos(θ)+0.5*W*sin(θ)

[0097] y 4 =y C -0.5*L*sin(θ)-0.5*W*cos(θ)

[0098] To this end, the processor 200 may receive information about the center point CP1 of the target vehicle Vtg, the center point CP2 of the rear bumper of the target vehicle Vtg, the width W of the target vehicle Vtg, the length L of the target vehicle Vtg, and the heading angle θ of the target vehicle Vtg from the camera 110 or another processor.

[0099] Alternatively or additionally or alternatively, as in Equation 2 below, the processor 200 may calculate the center point CP1 of the target vehicle Vtg based on the rear bumper center point CP2 of the target vehicle Vtg, the length L of the target vehicle Vtg, and the heading angle θ of the target vehicle Vtg.

[0100] [Equation 2]

[0101] x C =x R +0.5*L*cos(θ)

[0102] y C =y R +0.5*L*sin(θ)

[0103] In S420, the processor 200 may determine the LIDAR trajectory as a partial trajectory based on a positional relationship between the LIDAR trajectory and the reference trajectory RT.

[0104] The LIDAR trajectory may be generated based on information obtained by the LIDAR 120 and may be determined based on the clustering state of the point cloud.

[0105] The processor 200 may determine the LIDAR trajectory within a threshold distance from the reference trajectory RT as a partial trajectory. The method of determining the partial trajectory will be described in detail as follows.

[0106] Figures 7 to 10 An example of a method of determining a partial trajectory is shown.

[0107] Figure 7 and Figure 8 An example of a method of determining whether a LIDAR track is included in a first area is shown.

[0108] refer to Figure 7 and Figure 8 The first area AR1 may be an area located within a threshold distance Dth from any line segment in the reference trajectory RT.

[0109] The processor 200 may determine whether the shortest distance is within the threshold distance Dth by comparing the shortest distance between the first reference line RL1 and the center point of each of the LIDAR trajectories LT1 , LT2 , and LT3 .

[0110] The first reference line RL1 may include a line segment indicating a side surface of the target vehicle Vtg in the reference trajectory RT. For example, the first reference line RL1 may include a line segment indicating a side surface selected from two side surfaces of the target vehicle Vtg that is closer to the host vehicle VEH. The host vehicle VEH may refer to a vehicle that performs object estimation and vehicle control using the vehicle control device CM according to an example of the present disclosure.

[0111] like Figure 7 As shown, the first reference line RL1 may be a straight line for connecting the first point P1 and the third point P3. The first reference line RL1 may be a straight line denoted as “A 1 x+B 1 y+C 1 =0” straight line.

[0112] like Figure 8 As shown, when the coordinates of the center point of the first LIDAR trajectory LT1 are (x0, y0), the distance d1 between the first reference line RL1 and the center point of the first LIDAR trajectory LT1 may be calculated based on the following Equation 3.

[0113] [Equation 3]

[0114]

[0115] If the distance d1 is less than or equal to the threshold distance Dth, the processor 200 may determine that the first LIDAR track LT1 is included in the first area AR1. Similarly, if the shortest distance between the center point of the second LIDAR track LT2 and the first reference line RL1 is less than or equal to the threshold distance Dth, the processor 200 may determine that the second LIDAR track LT2 is included in the first area AR1. In addition, if the shortest distance between the third LIDAR track LT3 and the first reference line RL1 is less than or equal to the threshold distance Dth, the processor 200 may determine that the third LIDAR track LT3 is included in the first area AR1.

[0116] According to an example, the threshold distance Dth may vary according to driving conditions.

[0117] In the case where it is predicted that the reception rate of the LIDAR 120 decreases, the processor 200 may set the threshold distance Dth to be larger.

[0118] For example, the lower the reflectivity Vth of the target vehicle, the larger the threshold distance Dth may be set by the processor 200.

[0119] In addition, when the weather condition in the host vehicle VEH driving area deteriorates or becomes worse, the processor 200 may set the threshold distance Dth to be larger. The situation where the weather condition is bad may mean that it is raining or snowing (e.g., exceeding a threshold amount, such as 2-3 inches of rain or snow), or the visibility is of a certain level or lower (e.g., nautical miles) due to cloudy weather. In order to determine the weather condition, the processor 200 may receive weather information through the communication device 500.

[0120] The processor 200 may determine whether the LIDAR tracks LT1 , LT2 , and LT3 included in the first area AR1 are included in the second area.

[0121] Fig. 9 An example of a method of determining whether a LIDAR track belongs to the second area is shown.

[0122] refer to Fig. 9 , the second area AR2 may be an area located between a straight line connecting the front side of the vehicle and a straight line connecting the rear side of the vehicle.

[0123] The second reference line RL2 connecting the front side of the vehicle can be represented as “A 2 x+B 2 y+C 2 =0". Additionally or alternatively, the third reference line RL3 connecting the rear side of the vehicle may be represented as "A 3 x+B 3 y+C 3 =0".

[0124] The processor 200 may determine whether the center point of the first LIDAR track LT1 is located between the second reference line RL2 and the third reference line RL3. The processor 200 may determine the center point position of the first LIDAR track LT1 by substituting the coordinates (x0, y0) of the center point of the first LIDAR track LT1 into the x-coordinate and y-coordinate of each of the second reference line RL2 and the third reference line RL3.

[0125] If the result of substituting (x0, y0) into the x-coordinate and the y-coordinate of the second reference line RL2 is greater than 0, the processor 200 may determine that the center point of the first LIDAR trajectory LT1 is located below the second reference line RL2.

[0126] Furthermore, if the result of substituting (x0, y0) into the x-coordinate and the y-coordinate of the third reference line RL3 is less than 0, the processor 200 may determine that the center point of the first LIDAR trajectory LT1 is located above the third reference line RL3.

[0127] Therefore, if conditions 1 and 2 expressed in the following equation 4 are satisfied, the processor 200 may determine that the center point (x0, y0) of the first LIDAR trajectory LT1 is located in the second area AR2, and the second area AR2 is located between the second reference line RL2 and the third reference line RL3.

[0128] [Equation 4]

[0129] Condition 1:A 2 x0+B 2 y0+C 2 >0

[0130] Condition 2:A 3 x0+B 3 y0+C 3 <0

[0131] Fig.10 Examples of LIDAR tracks that are within a reference track and a threshold distance are shown.

[0132] refer to Fig.10 , the processor 200 may determine the LIDAR trajectories LT1, LT2, and LT3 whose center points are located in the area AR_O overlapping the first area AR1 and the second area AR2 as partial trajectories.

[0133] In S430 , the processor 200 may determine an object trajectory including partial trajectories, and may generate a final trajectory indicating a position of the object by fusing the object trajectory and the reference trajectory RT.

[0134] The object trajectory may include object candidate trajectories in addition to or instead of the partial trajectory.

[0135] The object candidate trajectory may be determined based on the overlap level between the object candidate trajectory and the reference trajectory RT, and the matching level of the directions of the object candidate trajectory and the reference trajectory RT. Fig.10 As shown, the fourth LIDAR trajectory LT4 may overlap the reference trajectory RT by a certain level or more. The position of the fourth LIDAR trajectory LT4 may be determined to be within the rear bumper area of ​​the target vehicle, and the shape of the fourth LIDAR trajectory LT4 may be determined to be the rear bumper of the target vehicle. Therefore, the processor 200 may determine that the fourth LIDAR trajectory LT4 is located within the rear bumper area of ​​the target vehicle. In other words, even if the center point of the fourth LIDAR trajectory LT4 is not located within the overlapping area AR_O between the first area AR1 and the second area AR2, the fourth LIDAR trajectory LT4 may be determined to be a part of the target vehicle Vtg.

[0136] According to the example of the present disclosure, the position of the object can be accurately determined because the object trajectory is determined to include a partial trajectory in addition to or instead of the object candidate trajectory. The final trajectory generated based on the example of the present disclosure and the final trajectory generated based on the comparative example will be referred to as Fig.11 and Fig.12 Description is as follows.

[0137] Fig.11 An example of a final trajectory according to a comparative example is shown. Fig.12 An example of a final trajectory according to an example of the present disclosure is shown. Fig.11 and Fig.12 Can be used for Figure 5 The final trajectory of the target vehicle is shown in .

[0138] like Figure 5 As shown, if the target vehicle Vtg is assumed to be black, the target vehicle Vtg has a low reflectance (e.g., below a threshold reflectance), then the point set (e.g., point cloud) obtained by the LIDAR 120 may be lost. In particular, in an area where the incident angle of the laser emitted by the LIDAR is large (such as the side surface of the front vehicle), the point cloud may be largely lost.

[0139] For example, Fig.11 As shown, according to the comparative example, the point cloud corresponding to the left surface of the target vehicle Vtg is lost, so the LIDAR trajectories LT1, LT2, and LT3 can be obtained in a fragmented form, such as Figure 7 shown. Figure 7 The fragmented LIDAR tracks LI1, LT2, and LT3 shown in FIG. 1 are inconsistent in shape, direction, and size, and thus the LIDAR tracks LI1, LT2, and LT3 may not be identified as object candidate tracks constituting the target vehicle Vtg. Therefore, according to the comparative example, in the process of identifying the target vehicle Vtg, the fragmented point cloud of the side area of ​​the target vehicle Vtg (e.g., Fig.11 and Fig.12 PC1).

[0140] Therefore, the position of the target vehicle Vtg may be somewhat inaccurate because Fig.11 The first final trajectory FT1 shown in FIG. 1 is formed based on a point cloud that does not include the fragmented point cloud PC1. For example, Figure 5 The target vehicle Vtg shown in FIG. is in a lane violation state. However, it is determined according to Fig.11 The first final trajectory F1 of the illustrated comparative example is in a state of not violating the lane L11. Therefore, since the position and driving state of the target vehicle Vtg are inaccurately determined, the vehicle may not be smoothly controlled.

[0141] On the other hand, according to an example of the present disclosure, the fragmented LIDAR trajectories LT1, LT2, and LT3 can be used Figures 7 to 10 The method shown in is included in the object trajectory. In other words, according to the example of the present disclosure, Fig.12 The fragmented point cloud PC1 shown in FIG. 1 can be reflected in the process of determining the second final trajectory ft2. Therefore, by accurately determining the state of the target vehicle Vtg, it can be determined Fig.12 The second final trajectory ft2 shown in FIG. 1 intrudes into the lane L11. In addition, the processor 200 may allow the driving controller 400 to safely drive the vehicle based on the second final trajectory ft2.

[0142] Fig.13 A computing system according to an example of the present disclosure is shown.

[0143] refer to Fig.13 , the computing system 1000 may include at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , a storage device 1600 , and a network interface 1700 connected to each other via a bus 1200 .

[0144] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and / or the storage device 1600. Each of the memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a read-only memory (ROM) and a random access memory (RAM).

[0145] Therefore, the operations of the methods or algorithms described in conjunction with the examples disclosed in the specification may be directly implemented using a hardware module, a software module, or a combination of a hardware module and a software module executed by the processor 1100. The software module may reside on a storage medium (i.e., the memory 1300 and / or the storage device 1600), such as a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk drive, a removable disk, or a compact disk-ROM (CD-ROM).

[0146] The storage medium may be coupled to the processor 1100. The processor 1100 may read information from the storage medium and may write information to the storage medium. Alternatively or additionally or alternatively, the storage medium may be integrated with the processor 1100. The processor and the storage medium may be implemented with an application specific integrated circuit (ASIC). The ASIC may be provided in a user terminal. Alternatively or additionally or alternatively, the processor and the storage medium may be implemented with separate components in a user terminal.

[0147] The above description is merely an example of the technical idea of ​​the present disclosure, and those skilled in the art may make various modifications and changes without departing from the essential features of the present disclosure.

[0148] The present disclosure is made to solve the above-mentioned problems occurring in the prior art while keeping the advantages achieved in the prior art unchanged.

[0149] An example of the present disclosure provides an object tracking device, and a vehicle control device using the object tracking device, which is capable of making full use of information obtained by LIDAR.

[0150] Furthermore, an example of the present disclosure provides an object tracking device, and a vehicle control device using the object tracking device, which is capable of accurately tracking the position of an object during a sensor fusion process.

[0151] Furthermore, examples of the present disclosure provide an object tracking device, and a vehicle control device using the object tracking device, which is capable of smoothly controlling a vehicle based on object tracking using accurate sensor fusion.

[0152] Furthermore, examples of the present disclosure provide an object tracking device, and a vehicle control device using the object tracking device, which can accurately determine the position of an object even if a point cloud obtained by LIDAR is missing due to the distance to the object and the external reflectivity of the object.

[0153] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art to which the present disclosure belongs from the following description.

[0154] According to an example of the present disclosure, an object tracking device may include a camera that obtains image data, a light imaging detection and ranging (LIDAR) that obtains a point cloud, and a processor that detects an object outside a host vehicle. If a LIDAR track generated based on information obtained by the LIDAR is within a predetermined threshold distance from a reference track generated based on the image data, the processor may determine the LIDAR track as a partial track, may determine an object track including the partial track, and may generate a final track indicating a position of the object by fusing the object track and the reference track.

[0155] According to an example, the reference trajectory may be obtained by representing the object detected from the image data in a top view in a world coordinate system.

[0156] According to an example, the LIDAR trajectory may be represented in a top view in a world coordinate system based on the clustering state of the point cloud.

[0157] According to an example, the processor may determine a first reference line in the reference track indicating a side surface of the target vehicle, and may determine whether a shortest distance between the first reference line and a center point of the LIDAR track is within a threshold distance.

[0158] According to an example, the processor may determine, as the first reference line, a line segment indicating a side surface selected from two side surfaces of the target vehicle that is closer to the host vehicle.

[0159] According to an example, the processor may determine a straight line connecting one end of the front bumper of the target vehicle and one end of the rear bumper of the target vehicle as a first reference line, and may compare the shortest distance between the center point of the LIDAR trajectory and the first reference line with a threshold distance.

[0160] According to an example, if the center point of the LIDAR trajectory is located between a second reference line indicating a front side of the target vehicle and a third reference line indicating a rear side of the target vehicle, the processor may include the LIDAR trajectory in the object trajectory.

[0161] According to an example, when the reflectivity of the target vehicle is low, the processor may set the threshold distance to be large.

[0162] According to an example, when weather conditions in the area where the host vehicle is traveling are severe, the processor may set the threshold distance to be large.

[0163] According to an example of the present disclosure, a vehicle control device includes a camera that obtains image data, a LIDAR that obtains a point cloud, a processor that detects an object outside the vehicle, and a drive controller that drives the vehicle under the control of the processor. If a LIDAR trajectory generated based on information obtained by the LIDAR is within a predetermined threshold distance from a reference trajectory generated based on the image data, the processor may determine the LIDAR trajectory as a partial trajectory, may determine an object trajectory including the partial trajectory, and may generate a final trajectory indicating the position of the object by fusing the object trajectory and the reference trajectory.

[0164] According to an example of the present disclosure, an object tracking method may include: generating a reference trajectory indicating a position of an object detected using a sensor other than LIDAR; determining the LIDAR trajectory as a partial trajectory if the LIDAR trajectory generated based on information obtained by the LIDAR is within a threshold distance from the reference trajectory; and determining an object trajectory including the partial trajectory; and generating a final trajectory indicating the position of the object by fusing the object trajectory and the reference trajectory.

[0165] According to an example, generating the reference trajectory may include representing the object detected from the image data in a top view in a world coordinate system.

[0166] According to an example, determining the LIDAR trajectory as a partial trajectory may include representing the LIDAR trajectory in a top view in a world coordinate system, the LIDAR trajectory being determined based on a clustering state of a point cloud obtained by the LIDAR.

[0167] According to an example, determining the LIDAR track as a partial track may include determining a first reference line in the reference track indicating a side surface of the target vehicle, and determining whether a shortest distance between the first reference line and a center point of the LIDAR track is within a threshold distance.

[0168] According to an example, determining the first reference line may include determining, as the first reference line, a line segment indicating a side surface selected from two side surfaces of the target vehicle that is closer to the host vehicle.

[0169] According to an example, determining the LIDAR track as a partial track may include comparing a shortest distance between a center point of the LIDAR track and the first reference line to a threshold distance.

[0170] According to an example, determining the LIDAR track as a partial track may include obtaining a second reference line indicating a front side of the target vehicle based on the fact that a shortest distance between a center point of the LIDAR track and the first reference line is within a threshold distance, obtaining a third reference line indicating a rear side of the target vehicle, and including the LIDAR track in the object track based on the fact that the center point of the LIDAR track is located between the second reference line and the third reference line.

[0171] According to an example, the object tracking method may further include setting the threshold distance to be large when the reflectivity of the target vehicle is low.

[0172] According to an example, the object tracking method may further include setting the threshold distance to be large when weather conditions in the host vehicle driving area are bad.

[0173] According to an example of the present disclosure, a method of controlling a vehicle may include: generating a reference trajectory indicating a position of an object outside the vehicle detected using a sensor; determining the LIDAR trajectory as a partial trajectory if a LIDAR trajectory generated based on information obtained by the LIDAR is within a threshold distance from the reference trajectory; determining an object trajectory including the partial trajectory; and generating a final trajectory indicating a position of the object by fusing the object trajectory and the reference trajectory; and controlling driving of the vehicle based on the final trajectory.

[0174] Therefore, the examples of the present disclosure are intended not to limit but to explain the technical ideas of the present disclosure, and the scope and spirit of the present disclosure are not limited by the above examples. The protection scope of the present disclosure should be interpreted by the appended claims, and all equivalents thereof should be interpreted as included in the scope of the present disclosure.

[0175] According to an example of the present disclosure, even if a LIDAR point cloud is partially missing, LIDAR information corresponding to an object may be fully utilized.

[0176] Furthermore, according to examples of the present disclosure, object position tracking performance may be improved because the object can be determined by reflecting LIDAR information that may be missing due to a lost point cloud.

[0177] Furthermore, according to examples of the present disclosure, vehicle control may be smoothly performed based on object tracking using accurate sensor fusion.

[0178] Furthermore, according to an example of the present disclosure, even if a point cloud obtained by LIDAR is missing due to the distance to the object and the external reflectivity of the object, the position of the object can be accurately determined based on the fragmented point cloud.

[0179] Furthermore, various effects understood directly or indirectly through the specification can be provided.

[0180] Hereinabove, although the present disclosure has been described with reference to the examples and the accompanying drawings, the present disclosure is not limited thereto but may be variously modified and altered by those skilled in the art without departing from the spirit and scope of the present disclosure as claimed in the following claims.

Claims

1. A device comprising: a camera configured to obtain image data; The sensor,is configured to obtain a point set; as well as processor, wherein the processor is configured to: Detecting objects outside the vehicle; determining the trajectory as a partial trajectory based on determining that the trajectory is within a threshold distance from a reference trajectory generated based on the image data, wherein the trajectory is generated based on information obtained by the sensor; determining an object trajectory including the partial trajectory; as well as generating a final trajectory indicating a position of the object by fusing the object trajectory and the reference trajectory; as well as A signal indicative of the final trajectory is output.

2. The apparatus according to claim 1, wherein the reference trajectory is obtained by representing the object in a top view in a coordinate system, wherein the object is detected from the image data. The apparatus according to claim 1 , wherein the trajectory is represented in a top view in a coordinate system based on a clustering state of the point set.

4. The apparatus of claim 1 , wherein the processor is configured to: determining a first reference line indicating a side surface of the object, wherein the object is in the reference trajectory; and It is determined whether the shortest distance between the first reference line and the center point of the trajectory is within the threshold distance.

5. The apparatus of claim 4, wherein the processor is configured to: A line segment indicating a side surface selected from a plurality of side surfaces of the object and closer to the vehicle is determined as the first reference line.

6. The apparatus of claim 4, wherein the processor is configured to: determining a straight line connecting one end of the front bumper of the object and one end of the rear bumper of the object as the first reference line; and The shortest distance between the center point of the trajectory and the first reference line is compared to the threshold distance.

7. The apparatus of claim 4, wherein the processor is configured to: The trajectory is included in the object trajectory based on determining that a center point of the trajectory is located between a second reference line indicating a front side of the object and a third reference line indicating a back side of the object.

8. The apparatus of claim 1, wherein the processor is configured to: As the reflectivity of the object decreases, the threshold distance is increased.

9. The apparatus of claim 1, wherein the processor is configured to: The threshold distance is increased as weather conditions in an area including the vehicle deteriorate.

10. The device according to claim 1, Wherein the processor is configured to control the vehicle based on the signal indicative of the final trajectory.

11. A method comprising the following steps: generating a reference trajectory indicating a position of an object detected using a sensor other than a light imaging detection and ranging (LIDAR) sensor; determining the trajectory as a partial trajectory based on determining that the trajectory is within a threshold distance from the reference trajectory, wherein the trajectory is generated based on information obtained by the LIDAR sensor; as well as determining an object trajectory including the partial trajectory; generating a final trajectory indicating the position of the object by fusing the object trajectory and the reference trajectory, and A signal indicative of the final trajectory is output.

12. The method of claim 11, wherein generating the reference trajectory comprises: The object is represented in a top view in a coordinate system, wherein the object is detected from the image data.

13. The method of claim 11, wherein determining the trajectory as the partial trajectory comprises: The trajectory is represented as a top view in a coordinate system, and the trajectory is determined based on a clustering state of a point set obtained by the LIDAR sensor.

14. The method of claim 11, wherein determining the trajectory as the partial trajectory comprises: determining a first reference line indicating a side surface of the object, wherein the object is in the reference trajectory; as well as It is determined whether the shortest distance between the first reference line and the center point of the trajectory is within the threshold distance.

15. The method of claim 14, wherein determining the first reference line comprises: A line segment indicating a side surface selected from a plurality of side surfaces of the object and closer to the vehicle is determined as the first reference line.

16. The method of claim 15, wherein determining the trajectory as the partial trajectory comprises: The shortest distance between the center point of the trajectory and the first reference line is compared to the threshold distance.

17. The method of claim 14, wherein determining the trajectory as the partial trajectory comprises: obtaining a second reference line indicating a front side of the object based on the shortest distance between the center point of the trajectory and the first reference line being within the threshold distance; obtaining a third reference line indicating a rear side of the object; as well as Based on the center point of the trajectory being between the second reference line and the third reference line, the trajectory is included in the object trajectory.

18. The method according to claim 11, further comprising the steps of: As the reflectivity of the object decreases, the threshold distance is increased.

19. The method according to claim 11, further comprising the steps of: As weather conditions in an area including the vehicle deteriorate, the threshold distance is increased.

20. The method according to claim 11, further comprising the steps of: The vehicle is controlled based on a signal indicative of the final trajectory, wherein the location of the object is external to the vehicle.

Images