Vehicle control apparatus and method thereof

Abstract

An apparatus for controlling a vehicle is presented. The apparatus may include at least three different types of sensors. The first sensor may obtain visual trajectory information about an external object within the image. The second sensor may provide trajectory information in the form of a plot corresponding to an external object. A third sensor may collect trajectory information in a point cluster or virtual frame corresponding to an external object. The apparatus may set a reference angular range and distance based on the first and second trajectory information. The apparatus may then associate these with third trajectory information and output a signal for vehicle control.

Description

Technical Field

[0001] The present disclosure relates to a vehicle control device and method, and more particularly, to sensor fusion. Background Art

[0002] Various studies are underway to identify external objects by using various sensors to assist the driving of a vehicle.

[0003] In particular, during the driving of a vehicle in a driving assistance device activation mode or an autonomous driving mode, external objects can be identified by using various sensors (e.g., light detection and ranging (LiDAR), a camera, or radio detection and ranging (RADAR)).

[0004] If external objects are identified by using various sensors, the external objects can be identified by fusing sensor data obtained via the sensors. Summary of the Invention

[0005] According to the present disclosure, a device for controlling a vehicle may include: a first sensor configured to obtain first trajectory information associated with an external object, where the first trajectory information may include a visual object corresponding to the external object within an image obtained by the first sensor; a second sensor configured to obtain second trajectory information associated with the external object, where the second trajectory information may include a drawing in a plurality of drawings obtained by the second sensor, and where the drawing corresponds to the external object; a third sensor configured to obtain third trajectory information associated with the external object, where the third trajectory information may include at least one of a point cluster in a plurality of point clusters obtained by the third sensor or a virtual box in a plurality of virtual boxes obtained by the third sensor, and where the point cluster or the virtual box corresponds to the external object; and a processor, where the processor is configured to: set a reference angle range based on selecting the first sensor as a reference sensor; set a reference distance based on the second trajectory information; associate at least one of the first trajectory information, the second trajectory information, and the third trajectory information based on the reference angle range and the reference distance; and output a signal for controlling the vehicle based on the association, and where the first sensor includes a camera, the second sensor includes a radio detection and ranging sensor, and the third sensor includes a light detection and ranging sensor.

[0006] The device, where the processor is configured to: determine a drawing in the plurality of drawings as a part of the second trajectory information, where the drawing is included within the reference angle range and is the closest to the vehicle in distance; and associate the second trajectory information and the first trajectory information based on the determination.

[0007] The device, wherein the processor is configured to: based on determining third trajectory information, associate at least one of the third trajectory information, the first trajectory information, and the second trajectory information, wherein the third trajectory information indicates that the third trajectory is included within a reference angular range and a reference distance.

[0008] The device, wherein the processor is configured to: determine a reference angle for setting the reference angular range based on at least one of a first length between the center of the image and one end of the image, a second length between the center of the image and one end of the first trajectory indicated by the first trajectory information, and the field of view of the first sensor.

[0009] The device, wherein the processor is configured to: determine the first trajectory information based on a first axis parallel to the traveling direction of the vehicle and a second axis perpendicular to the first axis; and determine a reference angle for setting the reference angular range based on the center coordinates from the first trajectory information.

[0010] The device, wherein the processor is configured to: set the reference angular range based on the angular resolution from the second sensor and the center coordinates from the first trajectory information; and associate the first trajectory information and the second trajectory information based on determining that the second trajectory from the second trajectory information is within the reference angular range from the first trajectory information.

[0011] The device, wherein the processor is configured to: set the reference distance based on the position resolution associated with the third sensor and the longitudinal distance from the vehicle indicated by the second trajectory information.

[0012] The device, wherein the processor is configured to: set the reference distance to a first reference distance based on the longitudinal distance being less than or equal to a first longitudinal distance; and based on the longitudinal distance exceeding the first longitudinal distance and being less than or equal to a second longitudinal distance, set the reference distance to a second reference distance exceeding the first reference distance at the second longitudinal distance by gradually increasing the reference distance from the first reference distance.

[0013] The device, wherein the processor is configured to: assign a first identifier to the first trajectory information, wherein the first identifier indicates that the first sensor is a reference sensor; assign a second identifier to the second trajectory information, wherein the second identifier indicates that the second sensor is a target sensor different from the reference sensor; and assign the second identifier or a third identifier to the third trajectory information, wherein the second identifier or the third identifier indicates that the third sensor is a target sensor.

[0014] The apparatus, wherein the processor is configured to: based on associating first trajectory information, second trajectory information, and third trajectory information, assign at least one of the following to the first trajectory information, the second trajectory information, and the third trajectory information: a sensor flag, a data structure, and an identifier indicating that the first trajectory information, the second trajectory information, and the third trajectory information are associated.

[0015] According to the present disclosure, a method for controlling a vehicle may include: setting a reference angle range by a processor based on selecting a first sensor as a reference sensor, wherein the first trajectory information obtained from the first sensor may include a visual object corresponding to an external object within an image obtained by the first sensor; setting a reference distance based on second trajectory information obtained from a second sensor, wherein the second trajectory information may include a drawing in a plurality of drawings obtained from the second sensor, and wherein the drawing corresponds to an external object; associating at least one of the first trajectory information obtained from the first sensor, the second trajectory information, and third trajectory information obtained from a third sensor based on the reference angle range and the reference distance, wherein the third trajectory information may include at least one of: a point cluster in a plurality of point clusters obtained from the third sensor or a virtual box in a plurality of virtual boxes obtained from the third sensor, and wherein the point cluster or the virtual box corresponds to an external object; and outputting a signal for controlling the vehicle based on the association.

[0016] The method may further include: determining a drawing in the plurality of drawings as part of the second trajectory information, wherein the drawing is included within the reference angle range and is closest to the vehicle in distance; and associating the second trajectory information and the first trajectory information based on the determination.

[0017] The method may further include: associating at least one of the third trajectory information, the first trajectory information, and the second trajectory information based on determining the third trajectory information, wherein the third trajectory information indicates that the third trajectory is included within the reference angle range and the reference distance.

[0018] The method may further include: determining a reference angle for setting the reference angle range based on at least one of a first length between the center of the image and one end of the image, a second length between the center of the image and one end of the first trajectory indicated by the first trajectory information, and a field of view of the first sensor.

[0019] The method may further include: determining the first trajectory information based on a first axis parallel to the traveling direction of the vehicle and a second axis perpendicular to the first axis; and determining a reference angle for setting the reference angle range based on center coordinates from the first trajectory information.

[0020] The method may further include: setting a reference angle range based on the angular resolution from the second sensor and the central coordinates from the first trajectory information; and associating the first trajectory information and the second trajectory information based on determining that the second trajectory from the second trajectory information is within the reference angle range from the first trajectory information.

[0021] The method may further include: setting a reference distance based on the position resolution associated with the third sensor and the longitudinal distance from the vehicle indicated by the second trajectory information.

[0022] The method may further include: setting the reference distance to a first reference distance based on the longitudinal distance being less than or equal to a first longitudinal distance, or setting the reference distance to a second reference distance exceeding the first reference distance at the second longitudinal distance by gradually increasing the reference distance from the first reference distance based on the longitudinal distance exceeding the first longitudinal distance and being less than or equal to a second longitudinal distance.

[0023] The method may further include: assigning a first identifier to the first trajectory information, where the first identifier indicates that the first sensor is a reference sensor; and assigning a second identifier to the second trajectory information, where the second identifier indicates that the second sensor is a target sensor different from the reference sensor, or assigning a second identifier or a third identifier to the third trajectory information, where the second identifier or the third identifier indicates that the third sensor is a target sensor.

[0024] The method may further include: based on associating the first trajectory information, the second trajectory information, and the third trajectory information, assigning at least one of the following to the first trajectory information, the second trajectory information, and the third trajectory information: a sensor flag, a data structure, and an identifier indicating that the first trajectory information, the second trajectory information, and the third trajectory information are associated. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features, and advantages of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings:

[0026] Figure 1 An example of a block diagram associated with a vehicle control device according to an example of the present disclosure is shown;

[0027] Figure 2 An example of a block diagram associated with the processing of a trajectory in an example of the present disclosure is shown;

[0028] Figure 3 An example of a block diagram associated with a trajectory generated by each sensor in an example of the present disclosure is shown;

[0029] Figure 4A and 4BIllustrates an example for setting a reference angle or a reference angle range in an example of the present disclosure;

[0030] Figure 5 Illustrates an example for associating a camera trajectory and a RADAR trajectory in an example of the present disclosure;

[0031] Figure 6A and 6B Illustrates an example for setting a reference distance from a RADAR trajectory in an example of the present disclosure;

[0032] Figure 7 Illustrates an example for performing association on a camera trajectory, a RADAR trajectory, and a LiDAR trajectory corresponding to an external object in an example of the present disclosure;

[0033] Figure 8 Illustrates an example of a flowchart associated with a vehicle control method according to an example of the present disclosure; and

[0034] Figure 9 Illustrates an example of a computing system associated with a vehicle control device or a vehicle control method according to an example of the present disclosure. Detailed Description of the Embodiments

[0035] Hereinafter, some examples of the present disclosure will be described in detail with reference to the reference drawings. When adding reference numerals to the components of each drawing, it should be noted that the same components include the same reference numerals, although they are indicated on another drawing. In addition, when describing the examples of the present disclosure, if the detailed description related to well-known functions or configurations may unnecessarily obscure the subject matter of the present disclosure, such descriptions will be omitted.

[0036] When describing the elements of an example of the present disclosure, terms such as first, second, A, B, (a), (b), etc. may be used herein. These terms are only used to distinguish one element from another element, but do not limit the corresponding elements, and are not related to the nature, order, or priority of the corresponding elements. In addition, unless otherwise defined, all terms, including the technical and scientific terms used herein, should be interpreted as the customary terms in the field to which the present disclosure belongs. It should be understood that the terms used herein should be interpreted as including meanings consistent with their meanings in the context of the present disclosure and the related fields, and will not be interpreted in an idealized or overly formal sense unless clearly defined in this document.

[0037] Hereinafter, reference will be made to Figures 1 to 9 to describe various examples of the present disclosure in detail.

[0038] Figure 1 Illustrates an example of a block diagram associated with a vehicle control device according to an example of the present disclosure.

[0039] Refer toFigure 1 , according to an example of the present disclosure, the vehicle control device 100 can be implemented inside or outside the vehicle, and some components included in the vehicle control device 100 can be implemented inside or outside the vehicle. At this time, the vehicle control device 100 can be integrated with the internal control unit of the vehicle and can be implemented by a separate device so as to be coupled to the control unit of the vehicle by separate connection means. For example, the vehicle control device 100 can further include Figure 1 components not shown in

[0040] According to an example of the present disclosure, the vehicle control device 100 can include a processor 110 and a plurality of sensors. The plurality of sensors at least include a first sensor, a second sensor, and a third sensor (for example, a camera 120, a RADAR 130, and a LiDAR 140). The processor 110, the camera 120, the RADAR 130, or the LiDAR 140 can be electrically coupled and / or operably coupled to each other through electronic components including a communication bus and / or wirelessly.

[0041] Hereinafter, the fact that hardware is operably coupled can include the fact that a direct and / or indirect connection is established between the hardware through wired and / or wireless means such that a second piece of hardware among the hardware is controlled by a first piece of hardware.

[0042] Although different boxes are shown, the examples are not limited thereto. Figure 1 Some of the hardware in Figure 1 can be included in a single integrated circuit (including a system-on-chip (SoC)). The type and / or quantity of the hardware included in the vehicle control device 100 are not limited to Figure 1 shown. For example, the vehicle control device 100 can only include

[0043] some of the hardware shown in

[0043] According to an example, the vehicle control device 100 can include hardware for processing data based on one or more instructions. The hardware for processing data can include a processor 110.

[0044] For example, the hardware for processing data can include an arithmetic and logic unit (ALU), a floating-point unit (FPU), a field-programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP).

[0045] The processor 110 can include a structure of a single-core processor or can include a structure of a multi-core processor, including dual-core, quad-core, six-core, or octa-core.

[0046] The camera 120 included in the vehicle control device 100 according to the example may include one or more optical sensors (e.g., a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor), which generate an electrical signal indicating the color and / or brightness of light. The plurality of optical sensors (e.g., photodiodes, CCD sensors, CMOS sensors, phototransistors, fiber optic sensors, infrared sensors, color sensors, etc.) included in the camera 120 may be arranged in a two-dimensional array.

[0047] The camera 120 may obtain electrical signals from the plurality of optical sensors substantially simultaneously and may generate an image or a frame, each image or frame corresponding to the light arriving at the two-dimensional array of optical sensors, each image or frame including a plurality of pixels arranged in a two-dimensional manner.

[0048] For example, the photo data captured by using the camera 120 may refer to the plurality of images obtained from the camera 120.

[0049] For example, the video data captured by using the camera 120 may represent a sequence of a plurality of images obtained from the camera 120 at a specified frame rate.

[0050] For example, the processor 110 may identify a visual object corresponding to an external object within the image obtained by the camera 120. For example, the processor 110 may identify a camera trajectory corresponding to an external object. For example, the camera trajectory may include one of the plurality of visual objects, each visual object being assigned an identifier for tracking the external object, each visual object corresponding to the external object, and each visual object being identified within the image.

[0051] For example, the RADAR 130 of the vehicle control device 100 according to the example may detect an external object by using electromagnetic (EM) scattering. For example, the RADAR 130 may identify at least one of the distance, speed, shape, or any combination thereof from the RADAR 130 to the external object.

[0052] For example, the RADAR 130 may generate a RADAR plot corresponding to each external object based on the electromagnetic wave reflected from the external object. For example, the processor 110 may generate a RADAR trajectory corresponding to a specific external object among the RADAR plots corresponding to the external objects respectively. For example, the RADAR trajectory may include one of the RADAR plots assigned an identifier for tracking the external object.

[0053] The LiDAR 140 of the vehicle control device 100 according to the example can obtain a data set from recognizing objects around the vehicle control device 100. For example, the LiDAR 140 can identify at least one of the position of the surrounding object, the moving direction of the surrounding object, the speed of the surrounding object, and any combination thereof, based on the pulsed laser signal emitted from the LiDAR 140 being reflected by the surrounding object and returning.

[0054] For example, the LiDAR 140 can obtain a data set for expressing the surrounding external object in the space defined by a first axis (e.g., the X-axis parallel to the vehicle traveling direction), a second axis (e.g., the Y-axis perpendicular to the X-axis), and a third axis (e.g., the Z-axis perpendicular to the X-axis and the Y-axis), based on the pulsed laser signal reflected from the surrounding object.

[0055] For example, the LiDAR 140 can obtain a data set including a plurality of points in the space formed by the first axis, the second axis, and the third axis, based on receiving the pulsed laser signal during a specified period.

[0056] For example, the plurality of points can be clustered into point clusters (e.g., point clouds) corresponding to the external object (or surrounding object).

[0057] For example, the processor 110 can obtain a data set including a plurality of points based on the time required or taken for emitting light from the vehicle and the time required or taken for receiving the light emitted from the vehicle. The processor 110 can obtain a data set for expressing the plurality of points in a three-dimensional virtual coordinate system including a first axis, a second axis, and a third axis.

[0058] For example, the first axis can include the x-axis. For example, the second axis can include the y-axis. For example, the third axis can include the z-axis. For example, the first axis, the second axis, and the third axis can be perpendicular to each other and can intersect with each other with the origin as a reference. The first axis, the second axis, and the third axis are not limited to the above examples. Hereinafter, for the sake of convenience of description, the first axis is described as the x-axis; the second axis is described as the y-axis; and the third axis is described as the z-axis.

[0059] The camera 120 of the vehicle control device 100 according to the example can obtain a camera trajectory corresponding to the external object. For example, the camera trajectory can include visual objects corresponding to the external object within the image obtained by the camera 120. For example, the camera trajectory can include visual objects within the image obtained by the camera 120 that are assigned identifiers for tracking the external object.

[0060] The RADAR 130 of the vehicle control device 100 according to the example can obtain a RADAR trajectory corresponding to an external object. For example, the RADAR trajectory can include the RADAR drawing corresponding to the external object among the RADAR drawings obtained by the RADAR 130. For example, the RADAR trajectory can include the RADAR drawing assigned with an identifier for tracking the external object.

[0061] The LiDAR 140 of the vehicle control device 100 according to the example can obtain a LiDAR trajectory corresponding to an external object. For example, the LiDAR trajectory can include the point cloud or virtual box corresponding to the external object among the point cloud or virtual box obtained by the LiDAR 140. For example, the LiDAR trajectory can include the visual object assigned with an identifier for tracking the external object in the point cloud or virtual box corresponding to the external object.

[0062] In the example, the processor 110 can assign a first identifier indicating that the camera 120 is a reference sensor to the camera trajectory. The processor 110 can assign a second identifier indicating that the RADAR 130 is a target sensor different from the reference sensor to the RADAR trajectory. The processor 110 can assign a second identifier or a third identifier indicating that the LiDAR 140 is a target sensor to the LiDAR trajectory.

[0063] The processor 110 of the vehicle control device 100 according to the example can perform an association for forming at least one group of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, and any combination thereof.

[0064] For example, performing the association can include: performing an association function (or association process) for creating at least one group of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, and any combination thereof.

[0065] For example, the processor 110 can set a reference angle range for performing the association of forming at least one group of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, and any combination thereof based on selecting the camera 120 as the reference sensor.

[0066] The processor 110 according to the example can identify the camera trajectory in a vehicle coordinate system represented with respect to the vehicle, on a first axis representing an increasing value toward the front of the vehicle and a second axis representing an increasing value toward the left side. For example, the first axis can include the x-axis (e.g., the x-axis parallel to the vehicle traveling direction). For example, the second axis can include the y-axis (e.g., the y-axis perpendicular to the x-axis).

[0067] For example, the processor 110 may identify the center coordinates of the camera trajectory based on the identified camera trajectory. The processor 110 may calculate a reference angle for setting a reference angle range based on the center coordinates of the camera trajectory.

[0068] For example, the processor 110 may calculate or determine a reference angle for setting a reference angle range based on at least one of a first length between the center of the image obtained by the camera 120 and one end of the image, a second length between the center of the image and one end of the camera trajectory, the field of view (FOV) of the camera, and any combination thereof.

[0069] For example, the center of the image obtained by the camera 120 may include the center line of the image.

[0070] In an example, the processor 110 may set a reference angle range based on the angular resolution of the RADAR 130 and the center coordinates of the camera trajectory.

[0071] For example, the processor 110 may set a reference angle range that rotates clockwise and / or counterclockwise by a specified angle from the reference angle based on the specified angle corresponding to the angular resolution of the RADAR 130. For example, the processor 110 may apply a specified margin to the specified angle corresponding to the angular resolution of the RADAR 130, and then may set a reference angle range that rotates clockwise and / or counterclockwise by the specified angle with the margin applied from the reference angle.

[0072] In an example, the processor 110 may use, as the RADAR trajectory, the RADAR plot in the RADAR plot obtained by the RADAR 130 that is included in the reference angle range and is identified at the closest distance to the vehicle. The processor may perform an association on the identified RADAR trajectory and the camera trajectory.

[0073] In an example, the processor 110 may perform an association on the camera trajectory and the RADAR trajectory based on identifying the RADAR trajectory within the reference angle range of the camera trajectory. For example, the processor 110 may assign an identifier that indicates that the camera trajectory and the RADAR trajectory correspond to the same external object based on identifying the RADAR trajectory within the reference angle range of the camera trajectory.

[0074] In an example, the processor 110 may set a reference distance associated with the RADAR trajectory for performing an association on at least one of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, or any combination thereof based on the RADAR trajectory.

[0075] In an example, the processor 110 may set a reference distance based on the position resolution of the LiDAR 140 and the longitudinal distance between the RADAR trajectory and the vehicle. For example, the processor 110 may identify the longitudinal distance between the RADAR trajectory and the vehicle based on the minimum x coordinate of the RADAR trajectory.

[0076] In an example, the processor 110 may set the reference distance to a first reference distance based on the longitudinal distance between the RADAR trajectory and the vehicle being less than or equal to a first longitudinal distance (e.g., about 25 - 35 m).

[0077] In an example, the processor 110 may, based on the longitudinal distance between the RADAR trajectory and the vehicle exceeding the first longitudinal distance and being less than or equal to a second longitudinal distance (e.g., about 55 - 65 m), set the reference distance to a second reference distance that exceeds the first reference distance at the second longitudinal position by gradually increasing the reference distance from the first reference distance.

[0078] In an example, the processor 110 may identify LiDAR trajectories that are included within a reference angle range and within a reference distance from the RADAR trajectory. The processor 110 may perform an association on at least one of the LiDAR trajectory, the camera trajectory, the RADAR trajectory, or any combination thereof based on identifying the LiDAR trajectories that are included within the reference angle range and within the reference distance from the RADAR trajectory.

[0079] In an example, the processor 110 may perform an association on at least one of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, or any combination thereof based on the reference angle range and the reference distance.

[0080] In an example, the processor 110 may, based on performing an association on the camera trajectory, the RADAR trajectory, and the LiDAR trajectory, assign at least one of a sensor flag, a data structure, an identifier, or any combination thereof indicating that the camera trajectory, the RADAR trajectory, and the LiDAR trajectory are associated to the camera trajectory, the RADAR trajectory, and the LiDAR trajectory.

[0081] As described above, the processor 110 of the vehicle control device 100 according to the example can set a reference angle by using the camera 120, can perform association on the camera trajectory and the RADAR trajectory based on the set reference angle, can set a reference distance by using the associated RADAR trajectory, and can perform association on the LiDAR trajectory, the RADAR trajectory, and the camera trajectory based on the set reference distance. By performing the above process, the processor 110 can more accurately identify the distance and position of an external object (especially a pedestrian). The processor 110 of the vehicle control device 100 can more accurately identify the distance and position of an external object, so that when the vehicle is traveling in a state where the driving assistance function or the autonomous driving function of the vehicle including the vehicle control device 100 is activated, an accident can be prevented.

[0082] Figure 2 An example of a block diagram related to the processing of trajectories in an example of the present disclosure is shown.

[0083] Refer to Figure 2 , the processor (for example, Figure 1 the processor 110 in Figure 1 ) of the vehicle control device (for example, Figure 1 the vehicle control device 100 in Figure 1 ) according to the example can obtain a trajectory or trajectory information corresponding to an external object through the camera 220 (for example, Figure 1 the camera 120 in

[0084] ), the RADAR 230 (for example,

[0085] the RADAR 130 in

[0086] Figure 1 ), and / or the LiDAR 240 (for example, Figure 1 the LiDAR 140 in

[0084] ). For example, the processor can obtain a camera trajectory through the camera 220. For example, the processor can obtain a RADAR trajectory or RADAR trajectory information through the RADAR 230. For example, the processor can obtain a LiDAR trajectory or LiDAR trajectory information through the LiDAR 240.

[0085] In the example, the processor can perform trajectory preprocessing 251 on the trajectories obtained through the camera 220, the RADAR 230, and / or the LiDAR 240. For example, the trajectory preprocessing 251 can include a process of removing at least one trajectory with a reliability less than a reference reliability based on the FOV of the camera 220, the RADAR 230, and / or the LiDAR 240.

[0086] In an example, the processor may perform trajectory association 253 based on execution of trajectory preprocessing 251. For example, trajectory association 253 may include the following processes: selecting one of the camera 220, RADAR 230, and LiDAR 240 as a reference sensor; performing association of the trajectories of target sensors different from the reference sensor based on the trajectory of the selected reference sensor; and forming a sensor trajectory cluster. For example, the sensor trajectory cluster may include a group indicating a specific external object.

[0087] In an example, the processor may perform trajectory fusion 255 based on execution of trajectory association 253. For example, trajectory fusion 255 may include the following process: generating an output trajectory used in the driving assistance mode or autonomous driving mode of the vehicle by fusing information of the sensor trajectory cluster generated based on trajectory association 253.

[0088] In an example, the processor may perform trajectory management 257 based on execution of trajectory fusion 255. For example, trajectory management 257 may include the following process: managing the output trajectory generated by trajectory fusion 255.

[0089] As described above, the processor of the vehicle control device according to the example may perform various processes on the trajectories obtained through the camera 220, RADAR 230, and / or LiDAR 240, so as to help prevent accidents by accurately identifying external objects when the vehicle is operating in the driving assistance mode or autonomous driving mode.

[0090] Figure 3 An example of a block diagram associated with the trajectories generated by each sensor in an example of the present disclosure is shown.

[0091] The processor (e.g., Figure 1 the processor 110 in Figure 1 ) of the vehicle control device according to the example (e.g., Figure 1 the vehicle control device 100 in Figure 1 ) may obtain trajectories or trajectory information corresponding to external objects based on the camera 320 (e.g., Figure 1 the camera 120 in

[0092] In an example, the camera 320 may perform single trajectory generation 321 based on a visual object corresponding to an external object recognized in the image. For example, the camera 320 may generate a camera trajectory or camera trajectory information based on single trajectory generation 321.

[0093] In an example, the RADAR 330 may perform multi-trajectory generation 331 based on a RADAR drawing corresponding to an external object. For example, the RADAR 330 may generate a RADAR trajectory or RADAR trajectory information based on the multi-trajectory generation 331. The processor may set one of the generated RADAR trajectories or one of the RADAR drawings as a reference RADAR trajectory.

[0094] In an example, the LiDAR 340 may perform single-trajectory generation 341 based on a point cloud or a virtual box corresponding to an external object. For example, the LiDAR 340 may generate a LiDAR trajectory or LiDAR trajectory information based on the single-trajectory generation 341.

[0095] In an example, the processor may perform a process associated with a reference angle range setting 351 based on a camera trajectory obtained from the camera 320. For example, the processor may perform a process associated with the reference angle range setting 351 centered on the camera trajectory.

[0096] In an example, the processor may perform an association between the camera trajectory and the RADAR trajectory based on performing a process associated with the reference angle range setting 351. For example, the processor may perform a RADAR trajectory association 353 for performing an association between the camera trajectory and the RADAR trajectory.

[0097] In an example, the processor may perform a reference distance setting 355 based on the RADAR trajectory.

[0098] In an example, the processor may perform a LiDAR trajectory association 357 for performing an association between the RADAR trajectory and the LiDAR trajectory based on performing the reference distance setting 355.

[0099] Details regarding the above reference angle range setting 351 and / or reference distance setting 355 will be described later.

[0100] Figure 4A and 4B illustrates an example for setting a reference angle or a reference angle range in an example of the present disclosure.

[0101] Referring Figure 4A and 4B , according to an example, a processor (e.g., Figure 1 the processor 110 in) of a vehicle control device (e.g., Figure 1 the vehicle control device 100 of) may obtain an image through a camera (e.g., Figure 1 the camera 120 in).

[0102] Referring Figure 4A, in the example, the processor can identify a reference angle t1 based on an image. For example, the processor can calculate the reference angle t1 based on at least one of the focus 411 of the camera, the focal length d1, the FOV t2 of the camera, the first length 'd' between the center of the image and one end of the image, the second length 'l' between the center of the image and one end of the camera trajectory 413, and any combination thereof.

[0103] For example, the processor can identify the reference angle t1 based on Equation 1 below.

[0104]

Equation 1

[0105]

[0106] As mentioned above, t1 in Equation 1 can include the reference angle t1. 'l' in Equation 1 can include the second length 'l' between the center of the image and one end of the camera trajectory 413. 'd' in Equation 1 can include the first length 'd' between the center of the image and one end of the image. 't2' in Equation 1 can include the FOV t2 of the camera.

[0107] As described above, the processor can calculate the reference angle t1 by using Equation 1 based on the camera trajectory output in the form of a bounding box included in the image.

[0108] Referring to Figure 4B , the processor of the vehicle control device according to the example can calculate a reference angle t3 in a vehicle coordinate system formed with the vehicle 420 as a reference. For example, the vehicle coordinate system formed with the vehicle 420 as a reference can be represented as a bird's-eye view (BEV). For example, the processor can identify a camera trajectory 421 corresponding to an external object. The bird's-eye view can take a view at a certain distance above the ground and / or the object and can capture an area larger than a threshold. The bird's-eye image can indicate (and / or can be associated with) the perspective from the vehicle (e.g., the roll, yaw, pitch information of the vehicle and / or one or more cameras of the vehicle). The bird's-eye image can indicate (and / or can be associated with) the time information of the frame of the bird's-eye image and / or other indicators. The bird's-eye image can indicate (and / or can be associated with) one or more landmark images included in the bird's-eye image.

[0109] The processor can identify the center coordinates 423 of the camera trajectory 421 based on the identified camera trajectory 421. For example, the center coordinates 423 of the camera trajectory 421 can be represented as (x, y). For example, the x coordinate of the center coordinates 423 can include an x-axis coordinate representing a value that increases towards the front of the vehicle 420. For example, the y coordinate of the center coordinates 423 can include a y-axis coordinate representing a value that increases towards the left side of the vehicle 420.

[0110] In the example, the processor may calculate the reference angle t3 based on Equation 2 below.

[0111]

Equation 2

[0112]

[0113] As described above, t3 in Equation 2 may include the reference angle t3. "x" in Equation 2 may refer to the x coordinate included in the center coordinate 423 of the camera trajectory 421. "y" in Equation 2 may refer to the y coordinate included in the center coordinate 423 of the camera trajectory 421.

[0114] As described above, the processor of the vehicle control device according to the example may calculate the reference angle t3 based on the camera trajectory 421. The processor may perform the following process based on calculating the reference angle t3 to associate the camera trajectory 421 and the RADAR trajectory. Hereinafter, reference will be made to Figure 5 Describe an example of associating the camera trajectory 421 and the RADAR trajectory. Since the angle information of the camera is relatively more accurate compared to RADAR and / or LiDAR, the camera can help identify the exact position of the external object by setting the reference angle t3 based on the camera trajectory 421.

[0115] Figure 5 An example of associating the camera trajectory and the RADAR trajectory in the example of the present disclosure is shown.

[0116] Refer to Figure 5 , the processor (e.g., Figure 1 the processor 110 in) of the vehicle control device (e.g., Figure 1 the vehicle control device 100) according to the example may associate the camera trajectory 511 and the RADAR trajectory 523.

[0117] In the example, as Figure 4A and 4B described, the processor may calculate or determine the reference angle t4 based on the center coordinate 513 of the camera trajectory 511.

[0118] In the example, the processor may identify or determine the angular resolution of the RADAR (e.g., Figure 1 the RADAR 130 in). The processor may, based on the angular resolution of the RADAR, identify a reference angle range including a radius rotated clockwise and / or counterclockwise by a specified angle (e.g., k1) from the reference angle t4. For example, the reference angle range may be referred to as an "angle gate". For example, the reference angle range may include the range between t4 + k1 and t4 - k1.

[0119] In an example, the processor may identify the RADAR drawing 521 included within a reference angle range from the RADAR drawing 521 (or RADAR trajectory) obtained via RADAR. Hereinafter, for ease of description, the RADAR drawing and the RADAR trajectory are described separately, but the RADAR drawing 521 may include the RADAR trajectory.

[0120] For example, the k1 value may be set differently according to the angle resolution of the RADAR. For example, as the angle resolution of the RADAR is higher (i.e., as the angle resolution of the RADAR is smaller), the k1 value may be set smaller. For example, as the angle resolution of the RADAR decreases (i.e., as the RADAR angle resolution increases), the k1 value may be set larger. In addition, the k1 value may be set by applying a specified margin to the angle resolution of the RADAR.

[0121] In an example, the processor may identify the RADAR drawing among the RADAR drawings 521 included within the reference angle range that is closest to the vehicle 500. For example, based on identifying the RADAR drawing with the shortest straight-line distance from the vehicle 500 among the RADAR drawings 521, the processor may select the identified RADAR drawing as the camera trajectory 511 for performing association with the RADAR trajectory 523.

[0122] As described above, the processor of the vehicle control device according to the example may perform association between the camera trajectory 511 and the RADAR trajectory 523 based on the selected RADAR trajectory 523 for performing association with the camera trajectory 511. The processor may perform association between the camera trajectory 511 and the RADAR trajectory 523, may set a reference distance from the RADAR trajectory 523 based on the RADAR trajectory 523, and may identify a LiDAR trajectory for performing association with the camera trajectory 511 and / or the RADAR trajectory 523 based on the set reference distance. Hereinafter, reference will be made to Figure 6A and 6B to describe an example of identifying a LiDAR trajectory for performing association with the camera trajectory 511 and / or the RADAR trajectory 523.

[0123] Figure 6A and 6B show examples of setting a reference distance from the RADAR trajectory in an example of the present disclosure.

[0124] Referring to Figure 6A and 6B , according to the example, the processor of the vehicle control device (e.g., Figure 1 the vehicle control device 100) (e.g., Figure 1The processor 110) in can identify the reference distance 'r' based on the RADAR trajectory 611.

[0125] Referring Figure 6A , the processor of the exemplary vehicle control device can set the reference distance 'r' based on the position of the RADAR trajectory 611. For example, the processor can set the reference distance 'r' based on the longitudinal distance between the RADAR trajectory 611 and the vehicle 600.

[0126] Referring Figure 6B , the processor of the exemplary vehicle control device can be based on the RADAR trajectory (e.g., Figure 6A the RADAR trajectory 611 in, hereinafter referred to as "RADAR trajectory 611") is within the first longitudinal distance 621 (e.g., about 30 m) from the vehicle 600, and set the reference distance 'r' to the first reference distance 623 (e.g., about 1 m).

[0127] In the example, the processor can variably set the reference distance 'r' based on the RADAR trajectory 611 being between the first longitudinal distance 621 and the second longitudinal distance 631 (e.g., about 60 m).

[0128] In the example, the processor can gradually increase the reference distance 'r' between the first longitudinal distance 621 and the second longitudinal distance 631. For example, the processor can, based on the longitudinal distance of the RADAR trajectory 611 exceeding the first longitudinal distance 621 and being less than or equal to the second longitudinal distance 631, gradually increase the reference distance 'r' from the first reference distance 623 and set the reference distance 'r' to the second reference distance 633 exceeding the first reference distance 623 at the second longitudinal distance 631.

[0129] Returning to Figure 6A , the processor of the exemplary vehicle control device can identify the LiDAR trajectory 613 within the reference distance 'r' based on setting the reference distance 'r' to the RADAR trajectory 611. In the example, the processor can perform an association on the RADAR trajectory 611 and the LiDAR trajectory 613 based on identifying the LiDAR trajectory 613 within the reference distance 'r' from the RADAR trajectory 611.

[0130] For example, the processor can perform an association on the LiDAR trajectory 613 and the RADAR trajectory 611 that is identified within the reference distance 'r' from the RADAR trajectory 611 and is identified within the reference angle range. For example, the reference angle range can include the range between t5 + k2 and t5 - k2.

[0131] As described above, the processor of the vehicle control device according to the example can set the reference distance 'r' based on the RADAR trajectory 611, and can perform association on the LiDAR trajectory 613 and the RADAR trajectory 611 within the set reference distance 'r', thereby obtaining a cluster of sensor trajectories with relatively accurate position information. Since the distance information of RADAR is relatively accurate compared to that of the camera and / or LiDAR, RADAR can help identify the exact distance of an external object by setting the reference distance 'r' based on the RADAR trajectory 611.

[0132] Figure 7 An example of performing association on a camera trajectory, a RADAR trajectory, and a LiDAR trajectory corresponding to an external object in an example of the present disclosure is shown.

[0133] Referring to Figure 7 , the processor (e.g., Figure 1 the processor 110 in Figure 1 ) of the vehicle control device (e.g., Figure 1 the vehicle control device 100 in Figure 1 ) can obtain the camera trajectory 711 corresponding to the external object 710 through a camera (e.g., Figure 1 the camera 120 in

[0134] In the example, the processor can calculate or determine the reference angle t6 based on the center coordinates of the camera trajectory 711. The processor can set a reference angle range based on the angular resolution of RADAR. For example, the reference angle range can include the range between t6 + k3 and t6 - k3.

[0135] In the example, the processor can select the RADAR plot closest to the vehicle 700 among the RADAR plots included in the reference angle range as the RADAR trajectory 713, and can perform association on the selected RADAR trajectory 713 and the camera trajectory 711.

[0136] In the example, the processor can set a reference distance based on the longitudinal distance between the RADAR trajectory 713 and the vehicle 700. The processor can identify the LiDAR trajectory 715 within the set reference distance. The processor can perform association on the RADAR trajectory 713 and the LiDAR trajectory 715 based on identifying the LiDAR trajectory 715 within the reference distance of the RADAR trajectory 713.

[0137] As described above, the processor of the vehicle control device according to the example can set a reference angle range based on the camera trajectory 711 and can set a reference distance based on the RADAR trajectory 713. Therefore, the camera trajectory 711, the RADAR trajectory 713, and / or the LiDAR trajectory 715 can be associated. The processor can identify the exact position of the external object 710 by performing the above process. By identifying the exact position or a more precise position of the external object 710, accidents can be prevented when the vehicle 700 is traveling in the driving assistance mode or the autonomous driving mode.

[0138] Figure 8 An example of a flowchart associated with a vehicle control method according to an example of the present disclosure is shown.

[0139] Hereinafter, it is assumed that Figure 1 the vehicle control device 100 performs Figure 8 the process. In addition, in the Figure 8 description, it can be understood that the operations described as being performed by one device are controlled by the processor 110 of the vehicle control device 100.

[0140] Figure 8 At least one of the Figure 1 operations can be performed by Figure 8 the vehicle control device 100. Figure 1 At least one of the Figure 8 operations can be performed by

[0141] Each operation in Figure 8 can be executed in order, but does not have to be executed in order. For example, the order of the operations can be changed, and at least two operations can be executed in parallel.

[0142] For example, the camera trajectory can include visual objects corresponding to external objects within the image obtained by the camera.

[0143] For example, the RADAR trajectory can include RADAR drawings corresponding to external objects among the RADAR drawings obtained by the RADAR.

[0144] For example, the LiDAR trajectory can include point clouds and virtual frames corresponding to external objects among the point clouds or virtual frames obtained by the LiDAR.

[0145] The vehicle control method according to the example may include the following operations: identifying, among the RADAR plots obtained through RADAR, the RADAR plot that is included within the reference angle range and is identified as the closest to the vehicle as the RADAR trajectory. The vehicle control method may include the following operations: performing association on the identified RADAR trajectory and the camera trajectory.

[0146] The vehicle control method according to the example may include the following operations: based on identifying the LiDAR trajectory included within the reference angle range and within the reference distance from the RADAR trajectory, performing association on at least one of the LiDAR trajectory, the camera trajectory, the RADAR trajectory, and any combination thereof.

[0147] The vehicle control method according to the example may include the following operations: in a vehicle coordinate system represented with respect to the vehicle, based on a first axis representing values increasing towards the front of the vehicle and a second axis representing values increasing towards the left side of the vehicle, identifying the camera trajectory. The vehicle control method may include the following operations: based on the central coordinates of the camera trajectory, calculating a reference angle for setting the reference angle range.

[0148] The vehicle control method according to the example may include the following operations: based on the angular resolution of the RADAR and the central coordinates of the camera trajectory, setting the reference angle range. The vehicle control method may include the following operations: based on identifying the RADAR trajectory within the reference angle range of the camera trajectory, performing association on the camera trajectory and the RADAR trajectory.

[0149] In S803, the vehicle control method according to the example may include the following operations: based on the RADAR trajectory, setting a reference distance from the RADAR trajectory for performing association on at least one of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, and any combination thereof.

[0150] The vehicle control method according to the example may include the following operations: based on the position resolution of the LiDAR and the longitudinal distance between the RADAR trajectory and the vehicle, setting the reference distance.

[0151] For example, the vehicle control method may include the following operations: based on the longitudinal distance between the RADAR trajectory and the vehicle being less than or equal to a first longitudinal distance, setting the reference distance as a first reference distance. For example, the vehicle control method may include the following operations: based on the longitudinal distance exceeding the first longitudinal distance and being less than or equal to a second longitudinal distance, by gradually increasing the reference distance from the first reference distance, setting the reference distance as a second reference distance exceeding the first reference distance at the second longitudinal distance.

[0152] In S805, the vehicle control method according to the example may include the following operations: performing association on at least one of a camera trajectory, a RADAR trajectory, a LiDAR trajectory, and any combination thereof based on a reference angle range and a reference distance.

[0153] The vehicle control method according to the example may include the following operations: dispatching a first identifier indicating that the camera is a reference sensor to the camera trajectory. The vehicle control method may include the following operations; dispatching a second identifier indicating that the RADAR is a target sensor different from the reference sensor to the RADAR trajectory. The vehicle control method may include the following operations: dispatching a second identifier or a third identifier indicating that the LiDAR is a target sensor to the LiDAR trajectory.

[0154] The vehicle control method according to the example may include the following operations: based on performing association on the camera trajectory, the RADAR trajectory, and the LiDAR trajectory, dispatching at least one of a sensor flag, a data structure, an identifier, and any combination thereof indicating that the camera trajectory, the RADAR trajectory, and the LiDAR trajectory are associated to the camera trajectory, the RADAR trajectory, and the LiDAR trajectory.

[0155] As described above, the vehicle control method according to the example may include the following operations: setting a reference angle range by using the camera as a reference sensor, and performing association of the identified RADAR trajectory with the camera trajectory within the set reference angle range. In addition, the vehicle control method may include the following operations: setting a reference distance based on the RADAR trajectory, and performing association of the LiDAR trajectory with the RADAR trajectory and the camera trajectory based on the set reference distance. The vehicle control method may accurately identify the position of an external object (especially a pedestrian) by performing the above operations (or processes). In addition, if the vehicle is operating in a driving assistance mode or an autonomous driving mode, the vehicle control method may prevent accidents by accurately identifying the position of an external object.

[0156] Figure 9 An example of a computing system associated with a vehicle control device or a vehicle control method according to an example of the present disclosure is shown.

[0157] Referring to Figure 9 , the computing system 1000 may include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage 1600, and a network interface 1700, which are connected to each other via a bus 1200.

[0158] The processor 1100 can be a central processing unit (CPU) or a semiconductor device for processing instructions stored in the memory 1300 and / or the storage 1600. Each of the memory 1300 and the storage 1600 can include various types of volatile or non-volatile storage media. For example, the memory 1300 can include a read-only memory (ROM) and a random access memory (RAM).

[0159] Accordingly, the operations of the methods or algorithms described in connection with the examples disclosed in this specification can be implemented directly by hardware modules, software modules, or a combination of hardware modules and software modules executed by the processor 1100. The software modules can reside on a storage medium (i.e., the memory 1300 and / or the storage 1600), such as residing on a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electric EPROM (EEPROM), a register, a hard disk drive, a removable disk, or a compact disk ROM (CD-ROM).

[0160] The storage medium can be coupled to the processor 1100. The processor 1100 can read information from the storage medium and can write information in the storage medium. Alternatively, the storage medium can be integrated with the processor 1100. The processor and the storage medium can be implemented by an application specific integrated circuit (ASIC). The ASIC can be set in the user terminal. Alternatively, the processor and the storage medium can be implemented by separate components in the user terminal.

[0161] This disclosure is to solve the above problems that occur in the prior art while maintaining the advantages achieved by the prior art unchanged.

[0162] Examples of this disclosure provide a vehicle control device and a method thereof, which can accurately identify the position and the distance from the vehicle of an external object (especially a pedestrian) by using a camera as a reference sensor.

[0163] Examples of this disclosure provide a vehicle control device and a method thereof, which can accurately detect the position of an external object based on each trajectory corresponding to the external object by performing association on the trajectories obtained through various sensors.

[0164] Examples of this disclosure provide a vehicle control device and a method thereof, which can accurately output a dynamic object fusion (DOF) trajectory of an external object by identifying the position of the external object and the distance between the vehicle and the external object based on performing association on the trajectories obtained through various sensors.

[0165] The technical problems to be solved by this disclosure are not limited to the above problems, and those skilled in the art to which this disclosure pertains will clearly understand any other technical problems not mentioned herein through the following description.

[0166] According to an example of the present disclosure, a vehicle control device may include: a camera that obtains a camera trajectory corresponding to an external object (e.g., other vehicles, pedestrians, animals, traffic signals / signs, road debris, construction areas, roadblocks / obstacles, benches, bus shelters, trash cans, parked cars, bridges, overpasses, tunnels, etc.); a radio detection and ranging (RADAR) that obtains a RADAR trajectory corresponding to an external object; a light detection and ranging system (LiDAR) that obtains a LiDAR trajectory corresponding to an external object; and a processor. The processor may set a reference angle range for performing an association of at least one of a group formed by the camera trajectory, the RADAR trajectory, the LiDAR trajectory, and any combination thereof based on selecting the camera as a reference sensor; may set a reference distance from the RADAR trajectory for performing an association of at least one of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, and any combination thereof; and may perform an association of at least one of the camera trajectory, the RADAR trajectory, the LiDAR trajectory, and any combination thereof based on the reference angle range and the reference distance. The camera trajectory may include a visual object corresponding to an external object within an image obtained by the camera. The RADAR trajectory may include a RADAR plot corresponding to an external object among RADAR plots obtained by the RADAR. The LiDAR trajectory may include a point cloud and a virtual box corresponding to an external object among a point cloud or a virtual box obtained by the LiDAR.

[0167] In an example, the processor may identify, from among RADAR plots obtained by the RADAR, a RADAR plot included within the reference angle range and identified as being at the closest distance to the vehicle as the RADAR trajectory, and may perform an association of the identified RADAR trajectory and the camera trajectory.

[0168] In an example, the processor may perform an association of at least one of the LiDAR trajectory, the camera trajectory, the RADAR trajectory, and any combination thereof based on identifying a LiDAR trajectory included within the reference angle range and within the reference distance from the RADAR trajectory.

[0169] In an example, the processor may calculate a reference angle for setting the reference angle range based on at least one of a first length between the center of an image obtained by the camera and one end of the image, a second length between the center of the image and one end of the camera trajectory, the field of view (FOV) of the camera, and any combination thereof.

[0170] In an example, the processor may identify a camera trajectory in a vehicle coordinate system represented based on the vehicle, based on a first axis representing values increasing toward the front of the vehicle and a second axis representing values increasing toward the left side of the vehicle, and may calculate a reference angle for setting a reference angle range based on the center coordinates of the camera trajectory.

[0171] In an example, the processor may set a reference angle range based on the angular resolution of the RADAR and the center coordinates of the camera trajectory, and may perform association of the camera trajectory and the RADAR trajectory based on identifying the RADAR trajectory within the reference angle range of the camera trajectory.

[0172] In an example, the processor may set a reference distance based on the position resolution of the LiDAR and the longitudinal distance between the RADAR trajectory and the vehicle.

[0173] In an example, the processor may set the reference distance to a first reference distance based on the longitudinal distance between the RADAR trajectory and the vehicle being less than or equal to a first longitudinal distance, and may set the reference distance to a second reference distance exceeding the first reference distance at a second longitudinal distance by gradually increasing the reference distance from the first reference distance based on the longitudinal distance exceeding the first longitudinal distance and being less than or equal to a second longitudinal distance.

[0174] In an example, the processor may assign a first identifier indicating that the camera is a reference sensor to the camera trajectory, may assign a second identifier indicating that the RADAR is a target sensor different from the reference sensor to the RADAR trajectory, and may assign a second identifier or a third identifier indicating that the LiDAR is a target sensor to the LiDAR trajectory.

[0175] In an example, the processor may assign at least one of a sensor flag, a data structure, an identifier, and any combination thereof indicating that the camera trajectory, the RADAR trajectory, and the LiDAR trajectory are associated to the camera trajectory, the RADAR trajectory, and the LiDAR trajectory based on performing association of the camera trajectory, the RADAR trajectory, and the LiDAR trajectory.

[0176] According to an example of the present disclosure, a vehicle control method may include: setting a reference angle range based on selecting a camera as a reference sensor for performing an association of at least one of a group including forming a camera trajectory, a RADAR trajectory, a LiDAR trajectory, and any combination thereof; setting a reference distance with respect to the RADAR trajectory for performing an association of at least one of a camera trajectory, a RADAR trajectory, a LiDAR trajectory, and any combination thereof based on the RADAR trajectory; and performing an association of at least one of a camera trajectory, a RADAR trajectory, a LiDAR trajectory, and any combination thereof based on the reference angle range and the reference distance. The camera trajectory may include a visual object corresponding to an external object within an image obtained by the camera. The RADAR trajectory includes a RADAR drawing corresponding to an external object among RADAR drawings obtained by the RADAR. The LiDAR trajectory includes a point cloud and a virtual box corresponding to an external object among a point cloud or a virtual box obtained by the LiDAR.

[0177] The vehicle control method according to the example may include: identifying, from among RADAR drawings obtained by the RADAR, a RADAR drawing that is included within the reference angle range and is identified as being at the closest distance to the vehicle as the RADAR trajectory, and performing an association of the identified RADAR trajectory and the camera trajectory.

[0178] The vehicle control method according to the example may include: performing an association of at least one of a LiDAR trajectory, a camera trajectory, a RADAR trajectory, and any combination thereof based on identifying a LiDAR trajectory that is included within the reference angle range and is included within the reference distance with respect to the RADAR trajectory.

[0179] The vehicle control method according to the example may include: calculating a reference angle for setting the reference angle range based on at least one of a first length between the center of an image obtained by the camera and one end of the image, a second length between the center of the image and one end of the camera trajectory, the FOV of the camera, and any combination thereof.

[0180] The vehicle control method according to the example may include: identifying the camera trajectory in a vehicle coordinate system represented with respect to the vehicle based on a first axis representing an increasing value toward the front of the vehicle and a second axis representing an increasing value toward the left side of the vehicle, and calculating a reference angle for setting the reference angle range based on the center coordinates of the camera trajectory.

[0181] The vehicle control method according to the example may include: setting the reference angle range based on the angular resolution of the RADAR and the center coordinates of the camera trajectory, and performing an association of the camera trajectory and the RADAR trajectory based on identifying the RADAR trajectory within the reference angle range with respect to the camera trajectory.

[0182] The vehicle control method according to the example may include: setting a reference distance based on the position resolution of the LiDAR and the longitudinal distance between the RADAR trajectory and the vehicle.

[0183] The vehicle control method according to the example may include: when the longitudinal distance between the RADAR trajectory and the vehicle is less than or equal to a first longitudinal distance, setting the reference distance as a first reference distance; and when the longitudinal distance exceeds the first longitudinal distance and is less than or equal to a second longitudinal distance, gradually increasing the reference distance from the first reference distance and setting the reference distance as a second reference distance exceeding the first reference distance at the second longitudinal distance.

[0184] The vehicle control method according to the example may include: assigning a first identifier indicating that the camera is a reference sensor to the camera trajectory; assigning a second identifier indicating that the RADAR is a target sensor different from the reference sensor to the RADAR trajectory; and assigning a second identifier or a third identifier indicating that the LiDAR is a target sensor to the LiDAR trajectory.

[0185] The vehicle control method according to the example may include: based on performing an association on the camera trajectory, the RADAR trajectory, and the LiDAR trajectory, assigning at least one of a sensor flag, a data structure, an identifier, and any combination thereof indicating that the camera trajectory, the RADAR trajectory, and the LiDAR trajectory are associated to the camera trajectory, the RADAR trajectory, and the LiDAR trajectory.

[0186] The above description is only an example of the technical idea of the present disclosure. Without departing from the basic features of the present disclosure, those skilled in the art can make various modifications and variations.

[0187] Therefore, the examples of the present disclosure are not intended to limit but to explain the technical idea of the present disclosure, and the scope and spirit of the present disclosure are not limited by the above examples. The protection scope of the present disclosure should be interpreted by the appended claims, and all equivalents thereof should be construed as being included within the scope of the present disclosure.

[0188] By using the camera as a reference sensor, the present technology can more accurately identify the position and the distance from the vehicle of an external object (especially a pedestrian).

[0189] In addition, by performing an association on the trajectories obtained through various sensors, the present technology can more accurately detect the position of an external object based on each trajectory corresponding to the external object.

[0190] In addition, the present technology can more accurately output the DOF trajectory of an external object by identifying the position of the external object and the distance between the vehicle and the external object based on performing an association on the trajectories obtained through various sensors.

[0191] In addition, various effects directly or indirectly understood through the present disclosure can be provided.

[0192] In the foregoing, although the present disclosure has been described with reference to examples and drawings, the present disclosure is not limited thereto, but those skilled in the art to which the present disclosure pertains can make various modifications and changes without departing from the spirit and scope of the present disclosure claimed in the appended claims.

Claims

1. A device for controlling a vehicle, the device comprising: A first sensor configured to: obtain first trajectory information associated with an external object, wherein the first trajectory information includes a visual object corresponding to the external object within an image obtained by the first sensor; A second sensor configured to: obtain second trajectory information associated with the external object, wherein the second trajectory information includes a drawing in a plurality of drawings obtained by the second sensor, and wherein the drawing corresponds to the external object; A third sensor configured to: obtain third trajectory information associated with the external object, wherein the third trajectory information includes at least one of the following: a point cluster in a plurality of point clusters obtained by the third sensor and a virtual box in a plurality of virtual boxes obtained by the third sensor, and wherein the point cluster or the virtual box corresponds to the external object; and A processor, wherein the processor is configured to: Set a reference angle range based on selecting the first sensor as a reference sensor; Set a reference distance based on the second trajectory information; Associate at least one of the first trajectory information, the second trajectory information, and the third trajectory information based on the reference angle range and the reference distance; and Output a signal for controlling the vehicle based on the association.

2. The device according to claim 1, wherein The processor is configured to: Determine the drawing in the plurality of drawings as part of the second trajectory information, wherein the drawing is included within the reference angle range and is closest to the vehicle in distance; and Associate the second trajectory information and the first trajectory information based on the determination.

3. The apparatus according to claim 1, wherein, The processor is configured to: Associate at least one of the third trajectory information, the first trajectory information, and the second trajectory information based on determining the third trajectory information, wherein the third trajectory information indicates that the third trajectory is included within the reference angle range and the reference distance.

4. The device according to claim 1, wherein, The processor is configured to: Determine a reference angle for setting the reference angle range based on at least one of the following: A first length between the center of the image and one end of the image; A second length between the center of the image and one end of the first trajectory indicated by the first trajectory information; And The field of view of the first sensor.

5. The device according to claim 1, wherein The processor is configured to: Determine the first trajectory information based on: A first axis parallel to the driving direction of the vehicle; and A second axis perpendicular to the first axis; and Determine a reference angle for setting the reference angle range based on the center coordinates from the first trajectory information.

6. The device according to claim 1, wherein, The processor is configured to: Set the reference angle range based on the angular resolution from the second sensor and the center coordinates from the first trajectory information; And Associate the first trajectory information and the second trajectory information based on determining that the second trajectory from the second trajectory information is within the reference angle range from the first trajectory information.

7. The apparatus according to claim 1, wherein The processor is configured to: Set the reference distance based on the position resolution associated with the third sensor and the longitudinal distance from the vehicle indicated by the second trajectory information.

8. The apparatus according to claim 7, wherein The processor is configured to: Based on the longitudinal distance being less than or equal to a first longitudinal distance, set the reference distance to a first reference distance; and Based on the longitudinal distance exceeding the first longitudinal distance and being less than or equal to a second longitudinal distance, set the reference distance to a second reference distance that exceeds the first reference distance at the second longitudinal distance by gradually increasing the reference distance from the first reference distance.

9. The device according to claim 1, wherein The processor is configured to: Assign a first identifier to the first trajectory information, where the first identifier indicates that the first sensor is the reference sensor; Assign a second identifier to the second trajectory information, where the second identifier indicates that the second sensor is a target sensor different from the reference sensor; and Assign the second identifier or a third identifier to the third trajectory information, where the second identifier or the third identifier indicates that the third sensor is a target sensor.

10. The device according to claim 1, wherein, The processor is configured to: Based on associating the first trajectory information, the second trajectory information, and the third trajectory information, assign at least one of the following to the first trajectory information, the second trajectory information, and the third trajectory information: a sensor flag, a data structure, and an identifier indicating that the first trajectory information, the second trajectory information, and the third trajectory information are associated.

11. A method for controlling a vehicle, the method comprising the steps of: Set a reference angle range based on selecting a first sensor as a reference sensor, where the first trajectory information obtained from the first sensor includes a visual object corresponding to an external object within an image obtained by the first sensor; Set a reference distance based on second trajectory information obtained from a second sensor, where the second trajectory information includes a drawing among a plurality of drawings obtained by the second sensor, and where the drawing corresponds to the external object; Based on the reference angle range and the reference distance, associate at least one of the first trajectory information obtained from the first sensor, the second trajectory information, or third trajectory information obtained from a third sensor, where the third trajectory information includes at least one of the following: a point cluster among a plurality of point clusters obtained by the third sensor or a virtual box among a plurality of virtual boxes obtained by the third sensor, and where the point cluster or the virtual box corresponds to the external object; and Based on this association, output a signal for controlling the vehicle.

12. The method according to claim 11, further comprising the steps of: Determine the drawing among the plurality of drawings as part of the second trajectory information, where the drawing is included within the reference angle range and is closest to the vehicle in distance; and Based on this determination, associate the second trajectory information and the first trajectory information.

13. The method according to claim 11, further comprising the steps of: Based on determining the third trajectory information, associating at least one of the third trajectory information, the first trajectory information, or the second trajectory information, wherein the third trajectory information indicates that the third trajectory is included within the reference angular range and the reference distance.

14. The method according to claim 11, further comprising the steps of: Determining a reference angle for setting the reference angular range based on at least one of the following: A first length between the center of the image and one end of the image; A second length between the center of the image and one end of the first trajectory indicated by the first trajectory information; And The field of view of the first sensor.

15. The method according to claim 11, further comprising the steps of: Determining the first trajectory information based on: A first axis parallel to the driving direction of the vehicle; And A second axis perpendicular to the first axis; And Determining a reference angle for setting the reference angular range based on the central coordinates from the first trajectory information.

16. The method according to claim 11, further comprising the steps of: Setting the reference angular range based on the angular resolution from the second sensor and the central coordinates from the first trajectory information; And Associating the first trajectory information and the second trajectory information based on determining that the second trajectory from the second trajectory information is within the reference angular range from the first trajectory information.

17. The method according to claim 11, further comprising the steps of: Setting the reference distance based on the position resolution associated with the third sensor and the longitudinal distance from the vehicle indicated by the second trajectory information.

18. The method according to claim 17, further comprising the steps of: Setting the reference distance to a first reference distance based on the longitudinal distance being less than or equal to a first longitudinal distance; Or Based on the longitudinal distance exceeding the first longitudinal distance and being less than or equal to a second longitudinal distance, gradually increasing the reference distance from the first reference distance and setting the reference distance to a second reference distance exceeding the first reference distance at the second longitudinal distance.

19. The method according to claim 11, further comprising the steps of: Assigning a first identifier to the first trajectory information, wherein the first identifier indicates that the first sensor is the reference sensor; and Assigning a second identifier to the second trajectory information, wherein the second identifier indicates that the second sensor is a target sensor different from the reference sensor, or assigning the second identifier or a third identifier to the third trajectory information, wherein the second identifier or the third identifier indicates that the third sensor is a target sensor.

20. The method according to claim 11, further comprising the steps of: Based on correlating the first trajectory information, the second trajectory information, and the third trajectory information, at least one of the following is assigned to the first trajectory information, the second trajectory information, and the third trajectory information: a sensor flag indicating that the first trajectory information, the second trajectory information, and the third trajectory information are correlated, a data structure, and an identifier.

Images