Deep learning-based traffic participant detection and location estimation method and apparatus

Abstract

The present invention discloses a deep learning-based method and apparatus for detecting traffic participants and estimating their location. According to the present invention, a deep learning-based apparatus for detecting traffic participants and estimating their location is provided, comprising: a processor; and a memory connected to the processor, wherein the memory stores program instructions executed by the processor to input an image into a backbone including a plurality of convolutional layers to extract a feature map for the image, input the feature map into a neck to adjust the feature map, input the adjusted feature map into a first head to detect the size, type, and location of a traffic participant included in the image, input the adjusted feature map into a second head to detect first and second points of each traffic participant, and estimate the direction of travel of each traffic participant using the first and second points.

Description

Technology Field

[0001] The present invention relates to a deep learning-based method and apparatus for detecting traffic participants and estimating their locations, and more specifically, to a method and apparatus that can detect various traffic participants using a Convolutional Neural Network (CNN)-based object detection network and improve the location accuracy of the participants. Background Technology

[0002] Various applications, such as autonomous vehicles and intelligent transportation systems, require not only the detection of traffic participants but also their accurate location information.

[0003] The simplest method among conventional object detection methods is to use the center point of the bounding box. The bounding box of an object can be easily detected using widely used object detectors such as YOLO, SSD, and R-CNN.

[0004] When the center point of the bounding box is considered as the location of the object, it shows an error from the actual location because that location is not the point that touches the ground.

[0005] Figure 1 is a diagram showing the error from the actual location that occurs when the center point of a bounding box is considered as the object location.

[0006] Referring to Figure 1, the object location on the road is obtained by converting the object location coordinates detected in the image into real-world coordinates using homography that represents the relationship between real-world coordinates and image coordinates.

[0007] However, since the real-world coordinates obtained from the bounding box center point do not exist on the ground, using them results in an error compared to the actual position. Therefore, to detect the accurate position of an object, it is necessary to know the point where the object touches the ground. Prior art literature

[0008] KR Registered Patent 10-2415620 The problem to be solved

[0009] To solve the problems of the aforementioned prior art, the present invention proposes a deep learning-based traffic participant detection and location estimation method and apparatus capable of accurately detecting the location of an object while minimizing errors. means of solving the problem

[0010] To achieve the above-mentioned purpose, according to one embodiment of the present invention, a deep learning-based traffic participant detection and location estimation device is provided, comprising: a processor; and a memory connected to the processor, wherein the memory stores program instructions executed by the processor to input an image into a backbone including a plurality of convolutional layers to extract a feature map for the image, input the feature map into a neck to adjust the feature map, input the adjusted feature map into a first head to detect the size, type, and location of a traffic participant included in the image, input the adjusted feature map into a second head to detect first and second points of each traffic participant, and estimate the direction of travel of each traffic participant using the first and second points.

[0011] The above traffic participants may include at least one of a vehicle, a two-wheeled vehicle, a three-wheeled vehicle, and a pedestrian.

[0012] The first point and the second point may be defined differently depending on the type of traffic participant mentioned above.

[0013] If the traffic participant is a vehicle, the first point of the vehicle is defined as the center point of the front extension line where the first wheel of the vehicle contacts the ground within the bounding box of the vehicle, and the second point of the vehicle is defined as the center point of the rear extension line where the second wheel of the vehicle contacts the ground within the bounding box of the vehicle.

[0014] The above program commands can, when the traffic participant is a vehicle, extract a first contact point and a second contact point where one wheel of the vehicle contacts the ground within the bounding box of the vehicle, determine the front extension line and the rear extension line using the size of the vehicle, the first contact point and the second contact point, and determine the real-world coordinates of the center points of each of the front extension line and the rear extension line as the first point and the second point.

[0015] If the above traffic participant is a pedestrian, the first and second points of the pedestrian may be defined as the heels of both feet.

[0016] The above program commands can determine the heel of one foot in contact with the ground as the first point and the heel of the other foot as the second point when the traffic participant is a pedestrian.

[0017] If the traffic participant is a two-wheeled vehicle, the first and second points of the two-wheeled vehicle may be defined as the points of contact between the two wheels of the two-wheeled vehicle and the ground within the bounding box of the two-wheeled vehicle.

[0018] A deep learning-based traffic participant detection and location estimation device in which, when the traffic participant is a three-wheeled vehicle, the first point of the three-wheeled vehicle is defined as the point of contact between the first wheel of the three-wheeled vehicle and the ground within the bounding box of the three-wheeled vehicle, and the second point of the three-wheeled vehicle is defined as the center point of the rear extension line where the three-wheeled vehicle and the ground meet within the bounding box of the three-wheeled vehicle.

[0019] According to another aspect of the present invention, a method for detecting traffic participants and estimating their location based on deep learning is provided in a device comprising a processor and memory, comprising the steps of: inputting an image into a backbone comprising a plurality of convolutional layers to extract a feature map for the image; inputting the feature map into a neck to adjust the feature map; inputting the adjusted feature map into a first head to detect the size, type, and location of each traffic participant included in the image; inputting the adjusted feature map into a second head to detect first and second points of each traffic participant; and estimating the direction of travel of each traffic participant using the first and second points.

[0020] According to another aspect of the present invention, a computer program stored in a computer-readable recording medium for performing the above-described method is provided. Effects of the invention

[0021] According to the present invention, there is an advantage in that the location accuracy of a traffic participant can be improved by additionally extracting two points and their order from the bounding box of the traffic participant. Brief explanation of the drawing

[0022] Figure 1 is a diagram showing the error from the actual location that occurs when the center point of a bounding box is considered as the object location. FIG. 2 is a diagram illustrating the configuration of an object detection and location estimation network according to the present embodiment. Figure 3 is a diagram illustrating the traffic participant detection and location estimation process according to the present embodiment. FIG. 4 is a diagram illustrating a deep learning-based traffic participant detection and location estimation device according to the present embodiment. Figures 5 to 8 are diagrams showing bounding boxes and two points detected from different traffic participants. Specific details for implementing the invention

[0023] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0024] The terms used herein are merely for describing specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0025] Furthermore, the components of the embodiments described with reference to each drawing are not limited to the respective embodiments and may be implemented to be included in other embodiments within the scope of maintaining the technical spirit of the present invention. It is also obvious that multiple embodiments may be re-implemented as a single embodiment that integrates multiple embodiments, even if a separate description is omitted.

[0026] Furthermore, in the description referring to the attached drawings, identical components are assigned the same or related reference numerals regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the present invention, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the present invention, such detailed description is omitted.

[0028] In this embodiment, a method is proposed to detect traffic participants (hereinafter referred to as objects), such as vehicles, pedestrians, two-wheeled vehicles (bicycles, kickboards, etc.), and three-wheeled vehicles, in an input image and to further estimate the position of contact with the ground to improve the positional accuracy of these objects.

[0029] FIG. 2 is a diagram illustrating the configuration of an object detection and location estimation network according to the present embodiment.

[0030] As illustrated in FIG. 2, the network according to the present embodiment may include a backbone (200), a neck (202), a first head (204), and a second head (206).

[0031] The backbone (200) includes a plurality of convolutional layers, extracts feature maps from input images, and may include pre-trained VGG and ResNet50.

[0032] The neck (202) finely adjusts the extracted feature map, upsamples the previous feature map to increase its size, reflects the feature map extracted from the backbone (200) in a concat manner, and may include FPN, PANet, BiFPN, etc.

[0033] The first head (204) classifies the size, location, and type of an object from the feature map adjusted in the neck (202).

[0034] The first head (204) detects a bounding box corresponding to an object in the input image and detects the size and type of the object included in the bounding box and its position within the image.

[0035] In the network according to the present embodiment, a second head (206) is additionally provided to estimate the accurate location and direction of movement of an object.

[0036] The second head (206) detects two points of an object included in the bounding box and estimates the exact position and direction of movement of the object through the two points.

[0037] Here, the two points may include an order such as the first and second points, and are defined in different ways depending on the traffic participants.

[0038] The first head (204) and the second head (206) are trained using a dataset containing correct answer data regarding the size, type, and first and second points of various traffic participants, and after training is completed, the type, size, and location of an object are detected from the input image, and the accuracy of the location is improved and the direction of travel is estimated through the detection of the first and second points.

[0039] Figure 3 is a diagram illustrating the traffic participant detection and location estimation process according to the present embodiment.

[0040] Referring to Fig. 3, not only are bounding boxes for various traffic participants estimated, but the direction of travel is also estimated through the detection of first and second points.

[0041] Deep learning-based traffic participant detection and location estimation according to the present embodiment can be performed on a device including a processor and memory.

[0042] As illustrated in FIG. 4, the device according to the present embodiment may include a processor (400) and a memory (402).

[0043] Here, the processor (400) may include a CPU (central processing unit) capable of executing a computer program or other virtual machines.

[0044] The memory (402) may include a non-volatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, etc. The memory (402) may also include a volatile memory such as various random access memory and may be defined as a computer-readable recording medium.

[0045] In the memory (402) according to the present embodiment, program instructions for detecting traffic participants and estimating their location based on deep learning are stored.

[0046] Figures 5 to 8 are diagrams showing bounding boxes and two points detected from different traffic participants.

[0047] Referring to FIG. 5, when the traffic participant is a vehicle, the first point of the vehicle is defined as the center point of the front extension line where the first wheel of the vehicle contacts the ground within the vehicle's bounding box, and the second point of the vehicle is defined as the center point of the rear extension line where the second wheel of the vehicle contacts the ground within the vehicle's bounding box.

[0048] In Fig. 5, the bounding box and the bottom rectangle of the vehicle are indicated by a blue solid line and a black dotted line, respectively.

[0049] The center points of the sides corresponding to the front and rear of the vehicle among the four sides of the square base are marked with red and green circles, respectively, and these center points become the first and second points for estimating the position and direction of travel of the vehicle.

[0050] The device according to the present embodiment can, when a traffic participant is a vehicle, extract a first contact point and a second contact point where one wheel of the vehicle and the ground come into contact within the boundary box of the vehicle, determine a front extension line and a rear extension line using the size of the vehicle, the first contact point and the second contact point, and determine the real-world coordinates of the center points of each of the front extension line and the rear extension line as the first point and the second point.

[0051] As described above, the device according to the present embodiment determines the center point of a line extending from each contact point by considering the contact points between two wheels identified in the image and the ground and the size of the vehicle, and thereby determines the first and second points of the vehicle.

[0052] Referring to Fig. 6, when the traffic participant is a pedestrian, the pedestrian's first and second points can be defined as the heels of both feet.

[0053] The position of both feet can vary depending on the pedestrian's posture.

[0054] In the first and second photos of Fig. 6, both of the pedestrian's feet are in contact with the ground. However, in the third photo, one of the pedestrian's feet is in contact with the ground while the other foot is in the air.

[0055] Therefore, in the case of a pedestrian, instead of using the point where the foot contacts the ground, the heel of each foot is estimated as the first and second points.

[0056] In Figure 6, the foot located in front of the two points of the two feet is marked with a red circle, and the foot located behind is marked with a green circle.

[0057] The device according to the present embodiment can determine the heel of one foot in contact with the ground as the first point and the heel of the other foot as the second point for estimating the direction of movement.

[0058] Referring to FIG. 7, two-wheeled vehicles may include bicycles and kickboards.

[0059] Since the two wheels of a two-wheeled vehicle are always in contact with the ground, the points where the two wheels come into contact with the ground are determined as the first and second points. In Fig. 7, the red circle and the green circle represent the points where the front wheel and the rear wheel come into contact with the ground, respectively.

[0060] Referring to FIG. 8, when the traffic participant is a three-wheeled vehicle, the first point of the three-wheeled vehicle is defined as the point of contact between the first wheel of the three-wheeled vehicle and the ground within the bounding box of the three-wheeled vehicle, and the second point of the three-wheeled vehicle is defined as the center point of the rear extension line where the three-wheeled vehicle and the ground are in contact within the bounding box of the three-wheeled vehicle.

[0061] Figure 8 shows the case where there is one front wheel and two rear wheels. In the case of the front, the point where the wheel contacts the ground is detected, and in the case of the rear, the center point of the rear extension line is detected.

[0062] As can be seen in Fig. 8, the center point is based on real-world coordinates rather than image coordinates. The red and green circles in Fig. 8 represent the contact surface at the front of the two-wheeled vehicle and the center point at the rear, respectively, and the black triangle indicates the point where the rear wheel contacts the ground. In Fig. 7, it can be seen that the center point at the rear is off-center, which is because it is a center point based on real-world coordinates rather than image coordinates.

[0063] The embodiments of the present invention described above are disclosed for illustrative purposes only, and those skilled in the art with ordinary knowledge of the present invention may make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions should be considered to fall within the scope of the following claims.

Claims

Claim 1 As a deep learning-based traffic participant detection and location estimation device, a processor; A deep learning-based traffic participant detection and location estimation device comprising a memory connected to the processor, wherein the memory stores program instructions executed by the processor to input an image into a backbone including a plurality of convolutional layers to extract a feature map for the image, input the feature map into a neck to adjust the feature map, input the adjusted feature map into a first head to detect the size, type, and location of a traffic participant included in the image, wherein the traffic participant includes at least one of a vehicle, a two-wheeled vehicle, a three-wheeled vehicle, and a pedestrian, and input the adjusted feature map into a second head configured to detect first and second points according to the type of traffic participant detected by the first head, which are defined differently from each other according to the type of traffic participant and are set based on the location where the traffic participant contacts the ground, including a sequence for determining the direction of travel of the traffic participant. Claim 2 delete Claim 3 delete Claim 4 A deep learning-based traffic participant detection and location estimation device according to claim 1, wherein, when the traffic participant is a vehicle, the first point of the vehicle is defined as the center point of the front extension line where the first wheel of the vehicle contacts the ground within the bounding box of the vehicle, and the second point of the vehicle is defined as the center point of the rear extension line where the second wheel of the vehicle contacts the ground within the bounding box of the vehicle. Claim 5 In claim 4, the program instructions extract a first contact point and a second contact point where one wheel of the vehicle and the ground contact each other within the bounding box of the vehicle when the traffic participant is a vehicle, determine the front extension line and the rear extension line using the size of the vehicle, the first contact point and the second contact point, and determine the real-world coordinates of the center points of each of the front extension line and the rear extension line as the first point and the second point, respectively, a deep learning-based traffic participant detection and location estimation device. Claim 6 A deep learning-based traffic participant detection and location estimation device according to claim 1, wherein when the traffic participant is a pedestrian, the first point and the second point of the pedestrian are defined as the heels of both feet. Claim 7 In claim 6, the program instructions determine the heel of one foot in contact with the ground as the first point and the heel of the other foot as the second point when the traffic participant is a pedestrian, a deep learning-based traffic participant detection and location estimation device. Claim 8 A deep learning-based traffic participant detection and location estimation device according to claim 1, wherein, when the traffic participant is a two-wheeled vehicle, the first point and the second point of the two-wheeled vehicle are defined as the points of contact between the two wheels of the two-wheeled vehicle and the ground within the bounding box of the two-wheeled vehicle. Claim 9 A deep learning-based traffic participant detection and location estimation device according to claim 1, wherein when the traffic participant is a three-wheeled vehicle, the first point of the three-wheeled vehicle is defined as the point of contact between the first wheel of the three-wheeled vehicle and the ground within the bounding box of the three-wheeled vehicle, and the second point of the three-wheeled vehicle is defined as the center point of the rear extension line where the three-wheeled vehicle and the ground meet within the bounding box of the three-wheeled vehicle. Claim 10 A method for detecting traffic participants and estimating their location based on deep learning in a device including a processor and memory, comprising: a step of inputting an image into a backbone including a plurality of convolutional layers to extract a feature map for the image; a step of inputting the feature map into a neck to adjust the feature map; a step of inputting the adjusted feature map into a first head to detect the size, type, and location of a traffic participant included in the image; a step of inputting the adjusted feature map into a second head configured to detect first and second points according to the type of traffic participant detected by the first head, wherein the traffic participant includes at least one of a vehicle, a two-wheeled vehicle, a three-wheeled vehicle, and a pedestrian, and is defined differently depending on the type of traffic participant and is set based on the location where the traffic participant contacts the ground, including a sequence for determining the direction of travel of the traffic participant; and a step of estimating the direction of travel of each traffic participant using the first and second points. Claim 11 delete Claim 12 A computer program stored on a computer-readable recording medium that performs the method according to paragraph 10.

Images