Object detection apparatus and method using monocular camera

Abstract

The present embodiments may provide an object distance calculation apparatus and method in which, by means of deep learning, an object on a railroad is recognized from information on an image captured by a monocular camera and the distance to the object can be calculated and corrected.

Description

Object detection device and method using a monocular camera

[0001] The present embodiments relate to an object detection device and method using a monocular camera.

[0002] Implementing autonomous driving technology for railway vehicles requires technology to detect dynamic obstacles on the track and estimate their distance. Methods for estimating distance in autonomous driving technology include utilizing stereo cameras to extract RGB and depth images, calibrating and post-processing them, or using distance sensors such as LiDAR and RADAR to calculate distance.

[0003] However, binocular cameras and distance sensors have the disadvantage of being expensive to install and maintain. Furthermore, while depth images estimated by binocular cameras offer the advantage of estimating relative distances to specific objects, they struggle to estimate the absolute distances required for autonomous driving.

[0004] Accordingly, a specific design for a technology for estimating distance using a monocular camera is required.

[0005] Against this backdrop, an object distance calculation device and method can be provided that recognizes an object using deep learning on an image captured by a monocular camera and then calculates the distance to the object.

[0006] In one aspect, the present embodiments may provide an object distance calculation device including an image acquisition unit that acquires image information captured from a monocular camera configured in a railway vehicle, an object information acquisition unit that recognizes a railway along which the railway vehicle runs, a vanishing line of the railway, and an object existing on the railway from the image information, and acquires object information including coordinates of the vanishing line of the railway and the coordinates of the object existing on the railway, and a distance information calculation unit that calculates distance information for an object existing on the railway based on the object information.

[0007] In another aspect, the present embodiments may provide a method for calculating an object distance, including a step of obtaining image information captured from a monocular camera configured in a railway vehicle, a step of recognizing a railway on which the railway vehicle runs, a vanishing line of the railway, and an object existing on the railway from the image information, a step of obtaining object information including coordinates of the vanishing line of the railway and coordinates of the object existing on the railway, and a step of calculating distance information for an object existing on the railway based on the object information.

[0008] According to the present embodiments, an object distance calculation device and method can be provided that can recognize an object on a railway using image information captured by a monocular camera using deep learning, and calculate and correct the distance to the object.

[0009] Fig. 1 is a schematic diagram for explaining an object distance calculation device according to the present embodiments.

[0010] Figures 2 and 3 are drawings for explaining the vanishing point and vanishing line according to the present embodiments.

[0011] Figure 4 is a drawing for explaining a bounding box according to the present embodiments.

[0012] Figures 5, 6 and 7 are drawings for explaining distance information calculation according to the present embodiments.

[0013] Figures 8 and 9 are drawings for explaining distance information correction according to the present embodiments.

[0014] Fig. 10 is a drawing for explaining an object distance calculation method according to the present embodiments.

[0015] Figures 11 and 12 are flowcharts for explaining distance information correction according to the present embodiments.

[0016] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, identical components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the technical idea of ​​the present invention, the detailed description may be omitted. When "includes," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it may include a case in which the plural is included unless specifically stated otherwise.

[0017] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms.

[0018] In a description of the positional relationship of components, when it is described that two or more components are "connected," "combined," or "connected," it should be understood that the two or more components may be directly "connected," "combined," or "connected," but that the two or more components may also be further "interposed" with another component to be "connected," "combined," or "connected." Here, the other component may be included in one or more of the two or more components that are "connected," "combined," or "connected" to each other.

[0019] In the description of the temporal flow relationship related to components, operation methods, or manufacturing methods, for example, when the temporal or flow relationship is described as “after”, “following”, “next to”, “before”, etc., it may also include cases where it is not continuous, unless “immediately” or “directly” is used.

[0020] Meanwhile, when numerical values ​​or corresponding information (e.g., levels, etc.) for components are mentioned, even without separate explicit description, the numerical values ​​or corresponding information may be interpreted as including an error range that may occur due to various factors (e.g., process factors, internal or external impact, noise, etc.).

[0021] Hereinafter, an object distance calculation device and method according to embodiments of the present disclosure will be described with reference to related drawings.

[0022] Fig. 1 is a schematic diagram illustrating an object distance calculation device according to the present embodiments. Figs. 2 and 3 are diagrams illustrating a vanishing point and a vanishing line according to the present embodiments. Fig. 4 is a diagram illustrating a bounding box according to the present embodiments. Figs. 5, 6, and 7 are diagrams illustrating distance information calculation according to the present embodiments.

[0023] Referring to FIG. 1, an object distance calculation device (100) may include an image acquisition unit (110) that acquires image information (1) captured from a monocular camera installed in a railway vehicle, an object information acquisition unit (120) that recognizes a railway on which a railway vehicle runs, a vanishing line (210) of the railway, and an object existing on the railway from the image information (1), and acquires object information including coordinates of the vanishing line (210) of the railway and the coordinates of the object existing on the railway, and a distance information calculation unit (130) that calculates distance information for an object existing on the railway based on the object information.

[0024] The configuration of the object distance calculation device (100) illustrated in FIG. 1 is an example and is not limited thereto. The object distance calculation device (100) may further include other components as needed, or some components may be omitted. In this case, according to an example, each component of the object distance calculation device (100) may be implemented as one by being combined with one another according to the communication structure design method, or some components may be omitted.

[0025] A monocular camera installed in a railway vehicle may be installed at least in one of the front or rear portions of the railway vehicle in the direction of travel. The image acquisition unit (110) may acquire image information (1) in real time from the monocular camera installed at least in one of the front or rear portions of the railway vehicle in the direction of travel. As an example, the monocular camera installed in the railway vehicle may be a pinhole camera model. The railway on which the railway vehicle runs has the characteristic that there is no significant change in gradient. Therefore, the distance information calculation device may acquire image information (1) using the pinhole camera model, obtain object information from the image information (1), and then calculate distance information about the object. Here, the pinhole camera model is an example of the type of camera installed in the railway vehicle, and is not limited thereto. The type of monocular camera installed in the railway vehicle may be configured in various ways as needed, as long as it does not contradict the technical idea of ​​the present disclosure.

[0026] The image information (1) acquired by the image acquisition unit (110) may be a general 2D photograph, a video, or a frame representing an image at a specific point in time in the video. In addition, the image information (1) may include a railway, a railway vanishing line (210), an object existing on the railway, and an object existing around the railway.

[0027] The object information acquisition unit (120) can acquire image information (1) from the image acquisition unit (110). The object information acquisition unit (120) can analyze the acquired image information (1) using a deep learning-based object detection algorithm. The object information acquisition unit (120) can analyze the image information (1) to recognize a railroad, a railroad vanishing line (210), an object existing on the railroad, and an object existing around the railroad included in the image information (1).

[0028] In one embodiment, the object information acquisition unit (120) can acquire object information by inputting image information (1) into a deep learning-based object detection algorithm including a convolutional neural network. Here, as long as the object detection algorithm can acquire object information by using image information as an input value, various algorithms such as the known Faster R-CNN, R-FCN (Region-based Fully Convolutional Networks), etc. can be used, and as long as the technical idea of ​​the present disclosure can be applied, it is not limited to a specific algorithm model. The object information acquisition unit (120) can be learned in advance based on an algorithm and sample images suitable for a railway vehicle and railway conditions.

[0029] The object information acquisition unit (120) can recognize the railroad, the vanishing line (210) of the railroad, objects existing on the railroad, and objects existing around the railroad included in the image information (1) using an object detection algorithm. The object information acquisition unit (120) can recognize the railroad on which the railroad vehicle is currently running. For example, the object information acquisition unit (120) can compare and analyze the image information (1) over time using a deep learning algorithm. Accordingly, the object information acquisition unit (120) can analyze changes in the railroads over time to classify and recognize the railroad on which the railroad vehicle is running and the surrounding railroads.

[0030] Referring to FIG. 2, the object information acquisition unit (120) can generate the coordinates of the vanishing point (200) of the recognized railroad and recognize the vanishing line (210).

[0031] Referring to FIG. 3, all parallel straight lines in physical space can have the same vanishing point (200) on the captured image. In addition, the vanishing points (200) of straight lines belonging to the same plane in physical space can exist on a straight line on the captured image. That is, different vanishing points (200) belonging to the same plane can form a vanishing line (210) as illustrated in FIG. 3.

[0032] Referring back to FIG. 2, the running railroad and the parallel railroads may have the same vanishing point (200) on the captured image. The object information acquisition unit (120) can recognize the running railroad. The object information acquisition unit (120) can recognize the railroad on which the railroad vehicle runs and the railroad parallel to the running railroad. The object information acquisition unit (120) can recognize the vanishing point (200) of the running railroad and the parallel railroad and generate the coordinates of the vanishing point (200). The coordinates of the vanishing point (200) may be generated as pixel coordinates having a value of (x, y) when the horizontal axis is viewed as the x-axis and the vertical axis is viewed as the y-axis.

[0033] The object information acquisition unit (120) can generate a vanishing line (210) based on the coordinates of the vanishing point (200). For example, the object information acquisition unit (120) can estimate the vanishing line (210) by acquiring the y-coordinate of the vanishing point (200). When the coordinates of the vanishing point (200) are (100, -25), the object information acquisition unit (120) can estimate a straight line formed by the equation Y=-25 on the pixel coordinate system as the vanishing line (210). The above description is an example for explaining the step of the object information acquisition unit (120) generating the coordinates of the vanishing point (200) and estimating the vanishing line (210) based on the vanishing point (200), and is not limited thereto. The method for estimating the vanishing line (210) may vary depending on the internal and external parameters of the camera installed in the railway vehicle, and may be configured in various ways as needed, as long as it does not contradict the technical idea of ​​the present disclosure.

[0034] Referring to FIG. 4, the object information acquisition unit (120) can generate coordinates of objects (400) existing on a running railway and objects existing around the railway. In one embodiment, the object information acquisition unit (120) can identify objects and object locations recognized from image information (1) and express them in the form of a bounding box (410).

[0035] As illustrated in FIG. 4, the bounding box (410) can express the object (400) and the area occupied by the object (position and size of the object) in the image information (1) as a rectangular area. Here, the bounding box (410) can be expressed with four coordinates. The coordinates of the upper left corner of the bounding box (410) can be expressed as (x_min, y_max). In addition, the coordinates of the lower right corner of the bounding box (410) can be expressed as (x_max, y_min). That is, the bounding box can be expressed in the form of coordinate values ​​of (x_min, y_max, x_max, y_min). At this time, the values ​​of x and y can be expressed based on the pixel coordinates of the image.

[0036] For example, a bounding box (410) having coordinates of (50, -50, 150, -150) may represent a rectangular area starting from a pixel coordinate position of the upper left corner (50, -50) of the image and ending at a pixel coordinate position of the lower right corner (150, -150) of the image. The object information acquisition unit (120) may generate object information including the coordinates of the vanishing line (210) of the railroad and the coordinates of an object (400) existing on the railroad.

[0037] Referring to Figure 5, a geometric model for distance estimation is illustrated. Based on this model, the method for calculating distance information by applying a deep learning algorithm to image information acquired with a monocular camera is as follows.

[0038] The distance information calculation unit (130) obtains camera information including external and internal parameter information of a monocular camera and can use the camera information to calculate distance information. That is, the distance information calculation unit (130) can calculate distance information using a deep learning algorithm based on object information and camera information.

[0039] As an example, the distance information generating unit (130) can generate distance information d using the mathematical expression 1 below.

[0040] [Mathematical Formula 1]

[0041]

[0042] Here, h represents the actual installed height of the camera, and f is the focal length that can be obtained through camera calibration. δy is the physical size of the pixel corresponding to the y-axis in the image, and this value can be obtained through camera calibration. u is the coordinate of the lower part of the target object in the image estimated through a deep learning-based object detection algorithm, and v is the coordinate of the vanishing line estimated in the image.

[0043] Among the parameters used in the above mathematical formula, the external parameter information of the camera may include the installation height (h) of the camera, the rotation angle of the camera, the translation value of the camera, and the transformation value between the camera coordinate system and the world coordinate system (real-world coordinate system).

[0044] The internal parameter information of the camera may include the focal length (f), principal point, and asymmetry coefficient of the camera. The focal length (f) of the camera may be the distance between the center of the lens and the image sensor. The focal length (f) of the camera may be expressed in units of pixels of an image. The pixels of the image may correspond to cells of the image sensor. Since the focal length (f) is expressed in units of pixels, the focal length (f) may be expressed as a relative value with respect to the size of the cells of the image sensor. For example, if the size of the cells of the image sensor is 0.1 mm and the focal length (f) of the camera is 500 pixels, the distance from the center of the lens of the camera to the image sensor may be 50 mm, which is 500 times the size of the cells of the image sensor.

[0045] The starting point may be the image coordinate of the foot of the perpendicular line drawn from the center of the camera lens to the image sensor. The asymmetry coefficient may be the degree of inclination of the y-axis of the cell array of the image sensor. The distance information calculation unit (130) may obtain the physical size of the recognized object and the pixel value of the object through calibration based on the internal parameters and object information of the camera. Here, the method of obtaining the physical size of the pixel through camera calibration is based on a known technology, and a detailed description is omitted.

[0046] Referring to FIG. 6, the distance information calculation unit (130) can more precisely calculate the distance from the railway vehicle to the object based on the coordinates of the vanishing line (210) and the coordinates of the object. The greater the difference between the coordinates of the vanishing line (210) and the coordinates of the lower part of the object, the closer the distance between the railway vehicle and the object can be. Similarly, the smaller the difference between the coordinates of the vanishing line (210) and the coordinates of the lower part of the object, the farther the distance between the railway vehicle and the object can be.

[0047] Using this, the distance information calculation unit (130) can compare the coordinates of the vanishing line (210) and the coordinates of the lower part of the object and reflect them in the calculation of distance information. That is, the distance information is inversely proportional to the physical value of the pixel corresponding to the y-axis of the object, proportional to the focal length of the camera and the installation height of the camera, and inversely proportional to the difference value between the coordinates of the vanishing line (210) and the coordinates of the lower part of the object.

[0048] Referring to FIG. 7, the distance information calculation unit (130) can obtain object information from the object information acquisition unit (120) and calculate distance information for objects existing on the railway. The distance information calculation unit (130) can display the calculated distance information using output interfaces such as a display, speaker, and haptic module within the railway vehicle.

[0049] Accordingly, by providing an object distance calculation device (100) that obtains an image captured only by a monocular camera installed in a railway vehicle, recognizes an object on a running railway using deep learning, and obtains distance information between the railway vehicle and the object, the installation and maintenance costs of components for calculating distance information can be reduced.

[0050] Figures 8 and 9 are drawings for explaining distance information correction according to the present embodiments.

[0051] Referring to FIG. 8, the object information acquisition unit (120) can recognize an object located around a railroad sleeper or a railroad, and acquire surrounding object information (800) including the coordinates of the sleeper or the coordinates of an object located around the railroad.

[0052] The content of the object information acquisition unit (120) recognizing a railroad sleeper or an object located around a railroad and acquiring surrounding object information (800) including the coordinates of the sleeper, the type of object located around the railroad, and the coordinates of the object located around the railroad is the same as the content of acquiring object information, so it is omitted to avoid redundant explanation.

[0053] The distance information calculation unit (130) can correct the calculated distance information based on surrounding object information (800) including the coordinates of sleepers installed on the railway. In one embodiment, the distance information calculation unit (130) can obtain surrounding object information (800) regarding sleepers installed on the railway. Sleepers installed on the railway can be installed at regular intervals. The distance information calculation unit (130) can include pre-stored surrounding object information (800) including information about the intervals at which sleepers are installed.

[0054] When the distance information calculation unit (130) obtains surrounding object information (800) for a sleeper installed on a railway, the distance information calculation unit (130) can obtain distance information between sleepers from the pre-stored surrounding object information (800). The distance information calculation unit (130) can compare and correct the obtained surrounding object information (800) and the pre-stored distance information between sleepers.

[0055] For example, let's assume that sleepers are installed at 50cm intervals on a railroad. The distance information calculation unit (130) may include pre-stored surrounding object information (800) including information about the intervals at which sleepers are installed. The distance information calculation unit (130) may calculate the distance between sleepers based on the surrounding object information (800) about the sleepers. For example, the distance between sleepers calculated by the distance information calculation unit (130) based on the surrounding object information (800) may be 60cm. The distance information calculation unit (130) may correct the calculated coordinate values ​​of the sleepers based on the pre-stored interval information between sleepers. That is, since the actual interval of 50cm is calculated as 60cm, the correction may be performed by reflecting the corresponding ratio. Accordingly, the distance information calculation unit (130) may generate a correction value for the object coordinate value corresponding to the correction value for the surrounding object coordinate value.

[0056] The distance information calculation unit (130) can correct the distance information calculated for an object existing on the railroad based on a correction value for the object coordinate value.

[0057] Referring to FIG. 9, the distance information calculation unit (130) may correct the calculated distance information based on surrounding object information (800). In one embodiment, the distance information calculation unit (130) may obtain surrounding object information (800) regarding trees installed around a railroad. For example, trees installed around a railroad may be installed at regular intervals. The distance information calculation unit (130) may include pre-stored surrounding object information (800) including information regarding the intervals at which the trees are installed.

[0058] When the distance information calculation unit (130) obtains surrounding object information (800) about trees installed around a railroad, the distance information calculation unit (130) can obtain distance information between trees from the pre-stored surrounding object information (800). The distance information calculation unit (130) can compare and correct the obtained surrounding object information (800) and the pre-stored distance information between trees.

[0059] For example, let's assume that trees are installed at 50m intervals around a railroad. The distance information calculation unit (130) may include pre-stored surrounding object information (800) including information about the intervals at which the trees are installed. The distance information calculation unit (130) may calculate the distance between trees based on the surrounding object information (800) about the trees. For example, the distance between trees calculated by the distance information calculation unit (130) based on the surrounding object information (800) may be 60m. The distance information calculation unit (130) may correct the calculated coordinate values ​​of the trees based on the pre-stored interval information between trees. Accordingly, the distance information calculation unit (130) may generate a correction value for the object coordinate value corresponding to the correction value for the surrounding object coordinate value.

[0060] The distance information calculation unit (130) can correct the distance information calculated for an object existing on the railroad based on the correction value for the object coordinate value. That is, the distance information calculation unit (130) can refine the distance information calculated for an object existing on the railroad by comparing the acquired surrounding object information (800) with the surrounding object information (800) stored in advance. It goes without saying that the content of correcting the distance information based on the object information described above can be substantially equally applied not only to the tree presented as an example, but also to surrounding object information (800) that can be stored in advance.

[0061] In another embodiment, the object distance calculation device (100) may further include a precision map receiving unit that acquires navigation (GPS) information and receives a precision map of a railway. The object information acquisition unit (120) may acquire railway information including gradient information of a railway where a railway vehicle is located based on the navigation (GPS) information and the precision map.

[0062] The precision map receiver can obtain a precision map from the server. Unlike conventional maps, a precision map can be constructed by scanning the railway and fixed objects adjacent to the railway using Lidar equipment, etc., and using information on the shape of the railway and the location and shape of surrounding facilities. Furthermore, the server collects and processes railway condition information and transmits it to the precision map receiver. If the precision map information or firmware needs to be updated, the server can transmit the relevant information to the precision map receiver for updating. In particular, the precision map can reflect changes in the railway environment, such as accident sections or construction areas.

[0063] The precision map of the present invention can be constructed by including railway gradient information. In this case, the object information acquisition unit (120) transmits navigation information and surrounding object information (800) to the server and receives a corresponding precision map, thereby acquiring railway information including the railway gradient information included in the precision map.

[0064] The distance information calculation unit (130) can correct the calculated distance information based on the railway information. Here, the comparison value of the coordinates of the vanishing line (210) and the coordinates of the bottom of the object may vary depending on the slope of the railway. For example, if the object on the railway is located on an uphill road, the coordinates of the bottom of the object on the railway may be generated higher than if it is located on flat ground. In this case, the distance information calculation unit (130) can determine that the coordinates of the bottom of the object are closer to the coordinates of the vanishing line (210) than if it is located on flat ground. That is, in this case, the distance information calculation unit (130) can determine that the object on the railway is further away than its actual location.

[0065] Accordingly, the distance information calculation unit (130) can correct the calculated distance information based on the acquired railway information. That is, the calculated distance information can be refined by obtaining the gradient of the railway on which the railway vehicle is running and the railway on which the object on the railway is located.

[0066] Accordingly, by providing an object distance calculation device that corrects the distance information obtained from a distance information calculation unit using surrounding object information and railway information, the calculated distance information can be further refined, thereby improving the reliability of the distance information generated from a monocular camera.

[0067] Hereinafter, a method for calculating object distances that can perform some or all of the embodiments described with reference to FIGS. 1 through 9 will be described with reference to the drawings. The above description may be omitted to avoid redundant explanation, and in this case, the omitted content may be substantially equally applied to the following description, as long as it does not contradict the technical spirit of the invention.

[0068] Fig. 10 is a diagram for explaining an object distance calculation method according to the present embodiments. Figs. 11 and 12 are flowcharts for explaining distance information correction according to the present embodiments.

[0069] Referring to FIG. 10, the object distance calculation device can obtain image information captured from a monocular camera configured in a railway vehicle (S1010).

[0070] Referring to FIG. 1, an object distance calculation device may include an image acquisition unit that acquires image information captured from a monocular camera installed in a railway vehicle, an object information acquisition unit that recognizes a railway on which a railway vehicle runs, a vanishing line of the railway, and an object existing on the railway from the image information, and acquires object information including coordinates of the vanishing line of the railway and coordinates of an object existing on the railway, and a distance information calculation unit that calculates distance information for an object existing on the railway based on the object information.

[0071] The configuration of the object distance calculation device illustrated in Fig. 1 is an example and is not limited thereto. The object distance calculation device may include additional components as needed, or some components may be omitted. In this case, as an example, each component of the object distance calculation device may be combined into a single unit, depending on the communication structure design method, or some components may be omitted.

[0072] A monocular camera installed in a railway vehicle may be installed at least in one of the front or rear portions of the railway vehicle in the direction of travel. An image acquisition unit may acquire image information in real time from a monocular camera installed at least in one of the front or rear portions of the railway vehicle in the direction of travel. In one embodiment, the monocular camera installed in the railway vehicle may be a pinhole camera model. The railway on which the railway vehicle runs has the characteristic that there is no significant change in gradient. Therefore, a distance information calculation device may acquire image information using the pinhole camera model, obtain object information from the image information, and then calculate distance information about the object. Here, the pinhole camera model is an example of a type of camera installed in the railway vehicle and is not limited thereto. The type of monocular camera installed in the railway vehicle may be configured in various ways as needed, as long as it does not contradict the technical idea of ​​the present disclosure.

[0073] The image information acquired by the image acquisition unit may be a typical 2D photograph, a video, or a frame representing an image at a specific point in time within the video. Furthermore, the image information may include a railway, a railway vanishing line, objects on the railway, and objects surrounding the railway.

[0074] Again, referring to FIG. 10, the object distance calculation device can recognize a railway on which a railway vehicle runs, a vanishing line of the railway, and an object existing on the railway from image information, and obtain object information including the coordinates of the vanishing line of the railway and the coordinates of the object existing on the railway (S1020).

[0075] The object information acquisition unit can acquire image information from the image acquisition unit. The object information acquisition unit can analyze the acquired image information using deep learning. By analyzing the image information, the object information acquisition unit can recognize the railroad, the railroad's vanishing line, objects on the railroad, and objects around the railroad included in the image information.

[0076] In one embodiment, the object information acquisition unit may acquire object information by inputting image information into a deep learning-based object detection algorithm including a convolutional neural network. The object information acquisition unit may be pre-trained based on an algorithm and sample images suitable for railway vehicles and railway conditions.

[0077] The object information acquisition unit can use an object detection algorithm to recognize a railroad, a vanishing line of the railroad, objects on the railroad, and objects around the railroad included in the image information. The object information acquisition unit can recognize the railroad on which a railroad vehicle is currently running. For example, the object information acquisition unit can use a deep learning algorithm to compare and analyze image information over time. Accordingly, the object information acquisition unit can analyze changes in the railroads over time to classify and recognize the railroad on which the railroad vehicle is running and the surrounding railroads.

[0078] Referring to FIG. 2, the object information acquisition unit can generate coordinates of the vanishing point of the recognized railway and recognize the vanishing line.

[0079] Referring to Figure 3, all parallel straight lines in physical space can have the same vanishing point in the captured image. Furthermore, the vanishing points of straight lines on the same plane in physical space can lie on a straight line in the captured image. In other words, different vanishing points on the same plane can form a vanishing line, as illustrated in Figure 3.

[0080] Referring back to FIG. 2, the running railroad and the parallel railroads may have the same vanishing point in the captured image. The object information acquisition unit may recognize the running railroad. The object information acquisition unit may recognize the railroad on which the railroad vehicle runs and the railroad parallel to the running railroad. The object information acquisition unit may recognize the vanishing points of the running railroad and the parallel railroad and generate coordinates of the vanishing points. The coordinates of the vanishing point may be generated as pixel coordinates having a value of (x, y) when the horizontal axis is viewed as the x-axis and the vertical axis is viewed as the y-axis.

[0081] The object information acquisition unit can generate a vanishing line based on the coordinates of the vanishing point. For example, the object information acquisition unit can estimate the vanishing line by acquiring the y-coordinate of the vanishing point. When the coordinates of the vanishing point are (100, -25), the object information acquisition unit can estimate a straight line formed by the equation Y=-25 on the pixel coordinate system as the vanishing line. The above description is an example for explaining the step of the object information acquisition unit generating the coordinates of the vanishing point and estimating the vanishing line based on the vanishing point, and is not limited thereto. The method for estimating the vanishing line may vary depending on the internal parameters and external parameters of the camera installed in the railway vehicle, and may be configured in various ways as needed as long as it does not contradict the technical idea of ​​the present disclosure.

[0082] Referring to FIG. 4, the object information acquisition unit can generate coordinates of objects existing on a moving railway and objects existing around the railway. In one embodiment, the object information acquisition unit can identify objects and their locations from image information and express them in the form of bounding boxes.

[0083] As illustrated in FIG. 4, a bounding box can represent an area (position and size of an object) occupied by an object or object in image information as a rectangular area. Here, the bounding box can be represented by four coordinates. The coordinates of the upper left corner of the bounding box can be represented as (x_min, y_max). In addition, the coordinates of the lower right corner of the bounding box can be represented as (x_max, y_min). That is, the bounding box can be represented in the form of coordinate values ​​of (x_min, y_max, x_max, y_min). At this time, the values ​​of x and y can be expressed based on the pixel coordinates of the image.

[0084] For example, a bounding box with coordinates of (50, -50, 150, -150) may represent a rectangular area that starts at the pixel coordinate position of the upper left corner (50, -50) of the image and ends at the pixel coordinate position of the lower right corner (150, -150) of the image. The object information acquisition unit may generate object information that includes the coordinates of the vanishing line of the railroad and the coordinates of an object existing on the railroad.

[0085] Referring back to FIG. 10, distance information for an object existing on a railway can be calculated based on object information (S1030). Referring to FIG. 5, a geometric model for distance estimation is illustrated. Based on the model, a method for calculating distance information by applying a deep learning algorithm to image information acquired by a monocular camera is as follows. The distance information calculation unit can obtain camera information including external and internal parameter information of the monocular camera and use the camera information to calculate distance information. That is, the distance information calculation unit can calculate distance information using object information and camera information. As an example, the distance information calculation unit (130) can calculate distance information d using the aforementioned mathematical expression 1.

[0086] The external parameter information of the camera may include the installation height of the camera, the rotation angle of the camera, the translation value of the camera, and the transformation value between the camera coordinate system and the world coordinate system (real-world coordinate system).

[0087] The internal parameter information of a camera may include the camera's focal length, principal point, and asymmetry coefficient. The camera's focal length may be the distance between the center of the lens and the image sensor. The camera's focal length may be expressed in units of pixels of an image. Each pixel of the image may correspond to a cell of the image sensor. Since the focal length is expressed in units of pixels, the focal length may be expressed as a relative value to the size of the cell of the image sensor. For example, if the cell size of the image sensor is 0.1 mm and the camera's focal length is 500 pixels, the distance from the center of the camera's lens to the image sensor may be 50 mm, which is 500 times the size of the cell of the image sensor.

[0088] The starting point can be the image coordinate of the foot of the perpendicular line drawn from the center of the camera lens to the image sensor. The asymmetry coefficient can be the degree of inclination of the y-axis of the cell array of the image sensor. The distance information calculation unit can obtain the physical size of the recognized object and the pixel value of the object through calibration based on the camera's internal parameters and object information. Here, the method of obtaining the physical size of the pixel through camera calibration is a well-known technique, and a detailed description thereof will be omitted.

[0089] Referring to Fig. 6, the distance information calculation unit can more precisely calculate the distance from the railway vehicle to the object based on the coordinates of the vanishing line and the coordinates of the object. The greater the difference between the coordinates of the vanishing line and the coordinates of the lower part of the object, the closer the distance between the railway vehicle and the object may be. Similarly, the smaller the difference between the coordinates of the vanishing line and the coordinates of the lower part of the object, the farther the distance between the railway vehicle and the object may be.

[0090] Using this, the distance information calculation unit can compare the coordinates of the vanishing line and the coordinates of the lower part of the object and reflect them in the distance information calculation. That is, the distance information is inversely proportional to the physical value of the pixel corresponding to the y-axis of the object, proportional to the focal length and installation height of the camera, and inversely proportional to the difference value between the coordinates of the vanishing line and the coordinates of the lower part of the object.

[0091] Referring to FIG. 7, the distance information generation unit can obtain object information from the object information acquisition unit and generate distance information for objects existing on the railway. The distance information generation unit can display the generated distance information using output interfaces such as a display, speaker, and haptic module within the railway vehicle.

[0092] Referring to FIG. 11, the object information acquisition unit can recognize a railroad sleeper or an object located around the railroad, and acquire surrounding object information including the coordinates of the sleeper and the coordinates of an object located around the railroad (S1110). The distance information calculation unit can acquire information on previously stored surrounding objects (S1120). The distance information calculation unit can correct the calculated distance information based on the surrounding object information (S1130).

[0093] Referring to FIG. 8, the object information acquisition unit can recognize an object located on a railroad sleeper or around a railroad, and acquire surrounding object information including the coordinates of the sleeper or the coordinates of an object located around the railroad.

[0094] The content of the object information acquisition unit recognizing a railroad sleeper or an object located around a railroad and acquiring surrounding object information including the coordinates of the sleeper, the type of object located around the railroad, and the coordinates of the object located around the railroad is the same as the content of acquiring object information, so it is omitted to avoid redundant explanation.

[0095] The distance information calculation unit can correct the calculated distance information based on surrounding object information including the coordinates of the railroad sleepers installed on the railroad. In one embodiment, the distance information calculation unit can obtain surrounding object information regarding the railroad sleepers installed on the railroad. The railroad sleepers may be installed at regular intervals. The distance information calculation unit can include pre-stored surrounding object information including information regarding the intervals at which the sleepers are installed.

[0096] When a distance information generation unit acquires surrounding object information about a railroad sleeper, it can acquire distance information between sleepers from the pre-stored surrounding object information. The distance information generation unit can compare and correct the acquired surrounding object information with the pre-stored distance information between sleepers.

[0097] For example, assume that sleepers are installed at 50cm intervals on a railway. The distance information calculation unit may include pre-stored surrounding object information including information about the intervals at which sleepers are installed. The distance information calculation unit may calculate the distance between sleepers based on the surrounding object information about the sleepers. For example, the distance between sleepers calculated by the distance information calculation unit based on the surrounding object information may be 60cm. The distance information calculation unit may correct the coordinate values ​​of the sleepers calculated based on the pre-stored interval information between sleepers. Accordingly, the distance information calculation unit may generate a correction value for the object coordinate value corresponding to the correction value for the surrounding object coordinate value.

[0098] The distance information calculation unit can correct the distance information calculated for an object existing on the railway based on a correction value for the object coordinate value.

[0099] Referring to FIG. 9, the distance information calculation unit can correct the calculated distance information based on surrounding object information. In one embodiment, the distance information calculation unit can obtain surrounding object information regarding trees installed around a railroad. For example, trees installed around a railroad may be installed at regular intervals. The distance information calculation unit can include pre-stored surrounding object information, including information regarding the intervals at which the trees are installed.

[0100] When the distance information generation unit acquires surrounding object information about trees installed around a railway, the distance information generation unit can acquire distance information between trees from the pre-stored surrounding object information. The distance information generation unit can compare the acquired surrounding object information with the pre-stored distance information between trees to make corrections.

[0101] For example, assume that trees are installed at 50m intervals around a railroad. The distance information calculation unit may include pre-stored surrounding object information including information about the spacing between trees. The distance information calculation unit may calculate the distance between trees based on the surrounding object information about the trees. For example, the distance between trees calculated by the distance information calculation unit based on the surrounding object information may be 60m. The distance information calculation unit may correct the coordinate values ​​of the trees calculated based on the pre-stored spacing information between trees. Accordingly, the distance information calculation unit may generate a correction value for the object coordinate value corresponding to the correction value for the surrounding object coordinate value.

[0102] The distance information calculation unit can correct the distance information calculated for an object existing on the railway based on the correction value for the object coordinate value. In other words, the distance information calculation unit can refine the distance information calculated for an object existing on the railway by comparing the acquired surrounding object information with pre-stored surrounding object information. It goes without saying that the content of correcting the distance information based on the object information described above can be applied substantially equally not only to the tree presented as an example, but also to surrounding object information that can be stored in advance.

[0103] Referring to FIG. 11, the object distance calculation device can obtain surrounding object information and navigation information (S1110), receive a precision map (S1120), and correct distance information based on the precision map (S1130).

[0104] The object distance calculation device may further include a precision map receiving unit that acquires navigation (GPS) information and receives a precision map of the railway. The object information acquisition unit may acquire railway information, including gradient information of the railway where the railway vehicle is located, based on the navigation (GPS) information and the precision map.

[0105] The precision map receiver can obtain a precision map from the server. Unlike conventional maps, a precision map can be constructed by scanning the railway and fixed objects adjacent to the railway using Lidar equipment, etc., and using information on the shape of the railway and the location and shape of surrounding facilities. Furthermore, the server collects and processes railway condition information and transmits it to the precision map receiver. If the precision map information or firmware needs to be updated, the server can transmit the relevant information to the precision map receiver for updating. In particular, the precision map can reflect changes in the railway environment, such as accident sections or construction areas.

[0106] The precision map of the present invention can be constructed by including railway gradient information. In this case, the object information acquisition unit transmits navigation information and surrounding object information to the server and receives a corresponding precision map, thereby acquiring railway information including the railway gradient information included in the precision map.

[0107] The distance information calculation unit can correct the calculated distance information based on the railway information. Here, the comparison values ​​between the coordinates of the vanishing line and the coordinates of the bottom of the object may vary depending on the slope of the railway. For example, if an object on the railway is located on an uphill road, the coordinates of the bottom of the object on the railway may be generated higher than if the object is located on flat ground. In this case, the distance information calculation unit can determine that the coordinates of the bottom of the object are closer to the coordinates of the vanishing line than if the object is located on flat ground. In other words, in this case, the distance information calculation unit can determine that the object on the railway is further away than its actual location.

[0108] Accordingly, the distance information generation unit can correct the calculated distance information based on the acquired railway information. That is, the calculated distance information can be refined by acquiring the gradient of the railway on which the railway vehicle is traveling and the railway on which the object on the railway is located.

[0109] Accordingly, by providing an object distance calculation method that obtains an image captured only by a monocular camera installed in a railway vehicle, recognizes an object on a running railway using deep learning, and obtains distance information between the railway vehicle and the object, and an object distance calculation method that corrects the distance information obtained by the distance information calculation unit using surrounding object information and railway information, the cost of installation and maintenance of components for performing the distance information calculation method can be reduced, and the reliability of the distance information generated from the monocular camera can be improved by further elaborating the calculated distance information.

[0110]

[0111] The above-described methods and / or various embodiments may be realized by digital electronic circuits, computer hardware, firmware, software, and / or a combination thereof. Various embodiments of the present disclosure may be implemented as a computer program that is executed by a data processing device, for example, one or more programmable processors and / or one or more computing devices, or stored on a computer-readable recording medium and / or a computer-readable recording medium. The above-described computer program may be written in any form of programming language, including a compiled language or an interpreted language, and may be distributed in any form, such as a standalone program, a module, a subroutine, etc. The computer program may be distributed through a single computing device, multiple computing devices connected through the same network, and / or multiple computing devices distributed to be connected through multiple different networks.

[0112] The methods and / or various embodiments described above may be performed by one or more processors configured to execute one or more computer programs that process, store, and / or manage any function, function, etc. by operating on the basis of input data or generating output data. For example, the methods and / or various embodiments of the present disclosure may be performed by special purpose logic circuits such as Field Programmable Gate Arrays (FPGAs) or Application Specific Integrated Circuits (ASICs), and an apparatus and / or system for performing the methods and / or embodiments of the present disclosure may be implemented as special purpose logic circuits such as FPGAs or ASICs.

[0113] The one or more processors executing the computer program may include a general-purpose or special-purpose microprocessor and / or one or more processors of any type of digital computing device. The processor may receive instructions and / or data from each of read-only memory and random-access memory, or may receive instructions and / or data from the read-only memory and the random-access memory. In the present disclosure, components of a computing device performing the methods and / or embodiments may include one or more processors for executing instructions, and one or more memory devices for storing instructions and / or data.

[0114] According to one embodiment, the computing device can transmit and receive data to and from one or more mass storage devices for storing data. For example, the computing device can receive and / or transfer data from a magnetic disc or an optical disc. A computer-readable storage medium suitable for storing instructions and / or data associated with a computer program may include, but is not limited to, any form of non-volatile memory, including semiconductor memory devices such as Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), flash memory devices, and the like. For example, the computer-readable storage medium may include a magnetic disk such as an internal hard disk or a removable disk, a magneto-optical disk, a CD-ROM, and a DVD-ROM disk.

[0115] To provide interaction with a user, a computing device may include, but is not limited to, a display device (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), etc.) for providing or displaying information to the user, and a pointing device (e.g., a keyboard, a mouse, a trackball, etc.) for allowing the user to provide input and / or commands to the computing device. That is, the computing device may further include any other types of devices for providing interaction with the user. For example, the computing device may provide any form of sensory feedback to the user, including visual feedback, auditory feedback, and / or tactile feedback, for interaction with the user. In this regard, the user may provide input to the computing device through various gestures, such as visual, vocal, or motion.

[0116] In the present disclosure, various embodiments may be implemented in a computing system that includes a backend component (e.g., a data server), a middleware component (e.g., an application server), and / or a front-end component. In this case, the components may be interconnected via any form or medium of digital data communication, such as a communications network. For example, the communications network may include a Local Area Network (LAN), a Wide Area Network (WAN), and the like.

[0117] A computing device based on the present embodiments may be implemented using hardware and / or software configured to interact with a user, including a user device, a user interface (UI) device, a user terminal, or a client device. For example, the computing device may include a portable computing device such as a laptop computer. Additionally or alternatively, the computing device may include, but is not limited to, Personal Digital Assistants (PDAs), tablet PCs, game consoles, wearable devices, Internet of Things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, and the like. The computing device may further include other types of devices configured to interact with a user. In addition, the computing device may include a portable communication device (e.g., a mobile phone, a smart phone, a wireless cellular phone, etc.) suitable for wireless communication over a network such as a mobile communication network. The computing device may be configured to communicate wirelessly with a network server using wireless communication technologies and / or protocols, such as Radio Frequency (RF), Microwave Frequency (MWF), and / or Infrared Ray Frequency (IRF).

[0118] The above description is merely an example of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations can be made without departing from the essential characteristics of the technical idea of ​​the present disclosure. In addition, the present embodiments are not intended to limit the technical idea of ​​the present disclosure but rather to explain it, and therefore the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present disclosure.

[0119]

[0120] CROSS-REFERENCE TO RELATED APPLICATION

[0121] This patent application claims priority under 35 USC § 119(a) of Korean Patent Application No. 10-2023-0183268, filed in Korea on December 15, 2023, the entire contents of which are incorporated herein by reference. Furthermore, this patent application claims priority in countries other than the United States for the same reasons, the entire contents of which are incorporated herein by reference.

Claims

1. An image acquisition unit that acquires image information captured from a monocular camera installed in a railway vehicle; An object information acquisition unit that recognizes a railway on which the railway vehicle runs, a vanishing line of the railway, and an object existing on the railway from the image information, and acquires object information including the coordinates of the vanishing line of the railway and the coordinates of the object existing on the railway; and A distance information calculation unit that calculates distance information for an object existing on the railroad based on the object information; An object distance calculation device including:

2. In paragraph 1, The above monocular camera, An object distance calculating device installed at least on one of the front or rear parts in the driving direction of the above railway vehicle.

3. In paragraph 1, The above object information acquisition unit, An object distance calculation device that obtains object information by inputting the image information into the deep learning-based object detection algorithm including convolutional neural networks.

4. In paragraph 1, The above object information acquisition unit, An object distance calculation device that recognizes a sleeper of the above-mentioned railroad or an object located around the above-mentioned railroad and obtains surrounding object information including coordinates of the sleeper or the coordinates of an object located around the above-mentioned railroad.

5. In paragraph 4, The above distance information generating unit, An object distance calculation device that corrects the distance information based on the above surrounding object information.

6. In paragraph 4, A precision map receiving unit that obtains navigation (GPS) information and receives a precision map of the railway; Including more, The above object information acquisition unit, An object distance calculation device that obtains railway information including gradient information of the railway where the railway vehicle is located based on the above navigation (GPS) information and the above precision map.

7. In paragraph 6, The above distance information generating unit, An object distance calculation device that corrects the distance information based on the above railway information.

8. In paragraph 1, The above distance information generating unit, An object distance calculation device that obtains camera information including external parameter information and internal parameter information of the monocular camera and uses the camera information to calculate the distance information.

9. A step of acquiring image information captured from a monocular camera installed in a railway vehicle; A step of recognizing a railway on which the railway vehicle runs, a vanishing line of the railway, and an object existing on the railway from the image information, and obtaining object information including coordinates of the vanishing line of the railway and coordinates of the object existing on the railway; and A step of calculating distance information for an object existing on the railroad based on the object information; A method for calculating object distance including:

10. In paragraph 9, The above monocular camera, A method for calculating an object distance installed at least on one of the front or rear parts of the driving direction of the above railway vehicle.

11. In paragraph 9, The step of obtaining the above object information is: An object distance calculation method for obtaining object information by inputting the image information into the deep learning-based object detection algorithm including convolutional neural networks.

12. In paragraph 9, The step of obtaining the above object information is: An object distance calculation method for recognizing a sleeper of the above-mentioned railway or an object located around the above-mentioned railway and obtaining surrounding object information including coordinates of the sleeper or the coordinates of an object located around the above-mentioned railway.

13. In paragraph 12, The steps for calculating the above distance information are: An object distance calculation method for correcting the distance information based on the above surrounding object information.

14. In paragraph 12, A step of obtaining navigation (GPS) information and receiving a precise map of the railway; Including more, The step of obtaining the above object information is: An object distance calculation method for obtaining railway information including gradient information of the railway where the railway vehicle is located based on the navigation (GPS) information and the precision map.

15. In paragraph 14, The steps for calculating the above distance information are: An object distance calculation method for correcting the distance information based on the above railway information.

16. In paragraph 9, The steps for calculating the above distance information are: An object distance calculation method comprising: obtaining camera information including external parameter and internal parameter information of the monocular camera, and using the camera information to calculate the distance information.

Images