Apparatus and method for monitoring railway track, and railway vehicle

Abstract

The present embodiments relate to an apparatus and a method for monitoring a railway track, and a railway vehicle. The apparatus for monitoring a railway track may comprise: a preprocessing unit for acquiring a railway track route from an input image, and converting the railway track route into a straight railway track route to acquire a preprocessed image; and a recognition unit for recognizing an atypical object, which may be an obstacle present along the railway track route, from the preprocessed image.

Description

Railway track monitoring device and method and railway vehicle

[0001] The present embodiments relate to a railway track monitoring device and method and a railway vehicle.

[0002] A railway vehicle may be a vehicle manufactured for the purpose of operating on railway tracks. Recently, research and development are actively underway into autonomous railway vehicles incorporating autonomous driving technology.

[0003] For this purpose, development of devices and methods for monitoring railway tracks is essential.

[0004] However, the current railroad tracks are covered with various types of gravel, making it very difficult to recognize irregular objects, which makes monitoring of the railroad tracks very inaccurate.

[0005] The present embodiments can provide a railway track monitoring device and method and a railway vehicle capable of accurately recognizing atypical objects and improving monitoring results for railway tracks.

[0006] In one aspect, the present embodiments can provide a railway track monitoring device including a preprocessing unit that obtains a running railway track from an input image and converts the running railway track into a straight running railway track to obtain a preprocessing image; and a recognition unit that recognizes an irregular object that may be an obstacle on the running railway track from the preprocessing image.

[0007] In another aspect, the present embodiments may provide a method for monitoring a railway track, including the steps of obtaining a running railway track from an input image, converting the running railway track into a straight running railway track to obtain a preprocessed image; and recognizing an irregular object that may be an obstacle on the running railway track from the preprocessed image.

[0008] In another aspect, the present embodiments can provide a railway vehicle including a railway track monitoring device that obtains a running railway track from an input image, converts the running railway track into a straight running railway track to obtain a preprocessed image, recognizes an irregular object that may be an obstacle on the running railway track from the preprocessed image, and displays a state on the running railway as a control signal based on the recognized output; and a control device that controls the running device of the railway vehicle based on the control signal to control the running of the railway vehicle.

[0009] According to the present embodiments, a railway track monitoring device and method and a railway vehicle can be provided that can accurately recognize atypical objects and improve monitoring results for railway tracks.

[0010] Figure 1 is a block diagram for explaining a railway vehicle according to the present embodiments.

[0011] Fig. 2 is a block diagram for explaining a railway track monitoring device according to the present embodiments.

[0012] Figure 3 is a block diagram for explaining a preprocessing unit according to the present embodiments.

[0013] Fig. 4 is a block diagram for explaining a recognition unit according to the present embodiments.

[0014] Fig. 5 is a block diagram for explaining the display unit according to the present embodiments.

[0015] Figure 6 is a flowchart for explaining a railway track monitoring method according to the present embodiments.

[0016] Figure 7 is a flowchart for explaining a preprocessing image acquisition method according to the present embodiments.

[0017] Fig. 8 is a flowchart for explaining a method for recognizing the state of a running railway track according to the present embodiments.

[0018] FIG. 9 is a block diagram of a railway track monitoring device and a railway vehicle computer system according to the present embodiments.

[0019] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, identical components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the technical idea of ​​the present invention, the detailed description may be omitted. When "includes," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it may include a case in which the plural is included unless specifically stated otherwise.

[0020] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms.

[0021] In a description of the positional relationship of components, when it is described that two or more components are "connected," "combined," or "connected," it should be understood that the two or more components may be directly "connected," "combined," or "connected," but that the two or more components may also be further "interposed" with another component to be "connected," "combined," or "connected." Here, the other component may be included in one or more of the two or more components that are "connected," "combined," or "connected" to each other.

[0022] In the description of the temporal flow relationship related to components, operation methods, or manufacturing methods, for example, when the temporal or flow relationship is described as “after”, “following”, “next to”, “before”, etc., it may also include cases where it is not continuous, unless “immediately” or “directly” is used.

[0023] Meanwhile, when numerical values ​​or corresponding information (e.g., levels, etc.) for components are mentioned, even without separate explicit description, the numerical values ​​or corresponding information may be interpreted as including an error range that may occur due to various factors (e.g., process factors, internal or external impact, noise, etc.).

[0024]

[0025] Figure 1 is a block diagram for explaining a railway vehicle according to the present embodiments.

[0026] Referring to FIG. 1, a railway vehicle (1) according to the present embodiments may include at least one of a railway track monitoring device (100), a control device (200), and a driving device (300). The railway track monitoring device (100), the control device (200), and the driving device (300) may be connected to each other.

[0027] For example, a railway vehicle (1) according to the present embodiments may include a railway track monitoring device (100) that obtains a running railway track from an input image, converts the running railway track into a straight running railway track to obtain a preprocessed image, recognizes an irregular object that may be an obstacle on the running railway track from the preprocessed image, and displays a state on the running railway as a control signal based on the recognized output, and a control device (200) that controls a running device (300) of the railway vehicle based on the control signal to control the running of the railway vehicle.

[0028] Here, a railway vehicle may be a vehicle manufactured for the purpose of operating on railway tracks. Accordingly, a railway vehicle may include any vehicle that operates on railway tracks.

[0029] Here, a person having ordinary skill in the art to which the present invention pertains will understand that the term railway vehicle can be replaced with other terms having equivalent meanings, such as train, locomotive, power car, passenger car, freight car, special car, etc.

[0030] In particular, the railway vehicle according to the present embodiments may include a locomotive, and more specifically, the railway vehicle may include a locomotive having a torpedo ladle car (TLC) attached thereto, but is not limited thereto as described above.

[0031] Here, a person having ordinary skill in the art to which the present invention pertains will understand that the term railroad track can be replaced with other terms having equivalent meanings, such as track, railroad, etc.

[0032] Here, a person having ordinary skill in the art to which the present invention pertains will understand that the term “image” can be replaced with other terms having equivalent meanings, such as “picture,” “frame,” or “image.”

[0033] A railway track monitoring device (100) can monitor a running railway track.

[0034] For example, a railway track monitoring device (100) can obtain a running railway track from an input image, convert the running railway track into a straight running railway track to obtain a preprocessed image, recognize an irregular object that may be an obstacle on the running railway track from the preprocessed image, and display the state of the running railway as a control signal based on the recognized output.

[0035] This railway track monitoring device (100) will be described in detail below.

[0036] The control device (200) can control the operation of a railway vehicle based on the monitoring results of the running railway track.

[0037] For example, the control device (200) can control the driving device (300) of the railway vehicle based on a control signal to control the driving (or operation) of the railway vehicle.

[0038] Specifically, the control device (200) can be connected to a railway track monitoring device (100). The control device (200) can receive monitoring results of a running railway track from the railway track monitoring device (100).

[0039] Here, the monitoring result of the running railway track may include, but is not limited to, a control signal, and may include any form of a result of monitoring the running railway track.

[0040] For example, the control device (200) can control the driving device (300) of the railway vehicle based on a control signal that is a result of monitoring the running railway track, thereby controlling the driving (or operation) of the railway vehicle.

[0041] Here, the control signal may include at least one of a control signal capable of controlling a railway vehicle in response to an atypical object on a running railway track, a control signal capable of controlling a railway vehicle in response to an atypical obstacle on a running railway track, and a control signal capable of controlling a railway vehicle in response to a risk of an atypical obstacle on a running railway track.

[0042] Here, the control signal may be generated automatically based on the monitoring results of the running railway track when generated by the railway track monitoring device (100), or may be generated by the driver.

[0043] In one example, the control device (200) can control the running (or operation) of the railway vehicle by controlling the running device (300) of the railway vehicle based on a control signal capable of controlling the railway vehicle in response to an irregular object on the running railway track.

[0044] In another example, the control device (200) can control the running (or operation) of the railway vehicle by controlling the running device (300) of the railway vehicle based on a control signal that can control the railway vehicle in response to an irregular obstacle on the running railway track.

[0045] In another example, the control device (200) can control the running (or operation) of the railway vehicle by controlling the running device (300) of the railway vehicle based on a control signal that can control the railway vehicle in response to the risk of an irregular obstacle on the running railway track.

[0046] Meanwhile, the railway vehicle according to the present embodiments may further include a driving device (300).

[0047] The driving device (300) can drive (or operate) a railway vehicle.

[0048] Here, the driving device (300) is a device (or mechanism) that directly drives (or operates) a railway vehicle, and may include at least one of a wheel, a suspension device, a braking device, a power device, and an auxiliary device, but is not limited thereto, and may include any device (or mechanism) that can drive (or operate) a railway vehicle.

[0049] For example, the driving device (300) can drive (or operate) a railway vehicle by controlling the driving operation by the control device (200).

[0050] In one example, the driving device (300) can drive (or operate) a railway vehicle based on a control signal that can control the railway vehicle in response to an irregular object on a running railway track.

[0051] That is, the driving device (300) can operate the braking device of the railway vehicle to brake the railway vehicle based on a control signal capable of controlling the railway vehicle in response to an irregular object on the running railway track.

[0052] In another example, the driving device (300) can drive (or operate) the railway vehicle based on a control signal that can control the railway vehicle in response to an irregular obstacle on the running railway track.

[0053] That is, the driving device (300) can operate the braking device of the railway vehicle to brake the railway vehicle based on a control signal capable of controlling the railway vehicle in response to an irregular obstacle on the running railway track.

[0054] In another example, the driving device (300) may drive (or operate) the railway vehicle based on a control signal that may control the railway vehicle in response to the risk of an irregular obstacle on the running railway track.

[0055] That is, the driving device (300) can operate the braking device of the railway vehicle to brake the railway vehicle based on a control signal capable of controlling the railway vehicle in response to the risk of an irregular obstacle on the running railway track.

[0056] Fig. 2 is a block diagram for explaining a railway track monitoring device according to the present embodiments.

[0057] Referring to FIG. 2, the railway track monitoring device (100) according to the present embodiments may include at least one of an image acquisition unit (110), a preprocessing unit (120), a recognition unit (130), and a display unit (140). The image acquisition unit (110), the preprocessing unit (120), the recognition unit (130), and the display unit (140) may be connected to each other.

[0058] For example, a railway track monitoring device (100) according to the present embodiments may include a preprocessing unit (120) that obtains a running railway track from an input image, converts the running railway track into a straight running railway track, and obtains a preprocessing image, and a recognition unit (130) that recognizes an irregular object that may be an obstacle on the running railway track from the preprocessing image.

[0059] The preprocessing unit (120) can obtain a preprocessed image.

[0060] For example, the preprocessing unit (120) can obtain a running railroad track from an input image and convert the running railroad track into a straight running railroad track to obtain a preprocessing image.

[0061] Specifically, the preprocessing unit (120) can be connected to the image acquisition unit (110) described later. The preprocessing unit (120) can receive input images from the image acquisition unit (110). The preprocessing unit (120) can acquire a running railway track on which a railway vehicle runs from the input images.

[0062] The preprocessing unit (120) can obtain a preprocessed image by converting (or generalizing) a running railway track. For example, the preprocessing unit (120) can obtain a preprocessed image by converting a running railway track into a running railway track of a preset shape.

[0063] Here, the preset shape may include, but is not limited to, a straight line shape, and may include any shape that can represent a running railway track.

[0064] In particular, a straight line shape can mean a shape close to a straight line. Accordingly, a straight line running railway track can mean a running railway track close to a straight line running railway track.

[0065] The recognition unit (130) can recognize the state of the running railway track.

[0066] Here, the condition on the running railway track may include not only the condition on the running railway track but also the condition around the running railway track.

[0067] Specifically, the recognition unit (130) can recognize an amorphous object.

[0068] For example, the recognition unit (130) can recognize an irregular object that may be an obstacle on a running railroad track in a preprocessed image.

[0069] Specifically, the recognition unit (130) can be connected to the preprocessing unit (120). The recognition unit (130) can receive a preprocessed image from the preprocessing unit (120). The recognition unit (130) can recognize an amorphous object in the preprocessed image.

[0070] Here, the amorphous object may be an object that can be an obstacle existing on a running railway track. That is, the amorphous object may include an amorphous obstacle existing on a running railway track and that can be an obstacle.

[0071] Meanwhile, the railway track monitoring device according to the present embodiments may further include an image acquisition unit (110).

[0072] The image acquisition unit (110) can acquire an input image.

[0073] For example, the image acquisition unit (110) can acquire an input image by photographing the external environment of a railway vehicle.

[0074] Here, the external environment of the railway vehicle may mean the environment surrounding the railway vehicle, and the external environment of the railway vehicle may include, but is not limited to, railway tracks, and may include anything surrounding the railway vehicle.

[0075] Here, the input image may include at least one input image among a front input image obtained by photographing the front of the railway vehicle, a rear input image obtained by photographing the rear of the railway vehicle, and a (left / right) side input image obtained by photographing the side of the railway vehicle.

[0076] The image acquisition unit (110) can acquire an input image by photographing the external environment of the railway vehicle at at least one time (or point in time) among real time and any point in time.

[0077] The image acquisition unit (110) may be located on the upper part of the railway vehicle, but is not limited thereto, and may be located anywhere in the railway vehicle where an input image can be acquired by photographing the external environment of the railway vehicle.

[0078] The image acquisition unit (110) may include a camera, but is not limited thereto, and may include any sensor that can capture the external environment of a railway vehicle and acquire an input image.

[0079] Although the image acquisition unit (110) has been described as being included in the railway track monitoring device according to the present embodiments, it is not limited thereto and may be configured as a separate device from the railway track monitoring device.

[0080] As described above, the image acquisition unit (110) according to the present embodiments includes a camera positioned on top of a railway vehicle (e.g., a locomotive), and can capture the front of the railway vehicle in real time through the camera to acquire an FHD video image as an input image, and can provide the input image to a computer positioned inside the railway vehicle connected by wire or wirelessly. Here, the computer inside the railway vehicle can include a railway monitoring device according to the present embodiments.

[0081] Meanwhile, the railway track monitoring device according to the present embodiments may further include a display unit (140).

[0082] The display unit (140) can display the status of the running railway track.

[0083] For example, the display unit (140) can display the status of the running railway track based on the recognition result of the recognition unit (130).

[0084] Specifically, the display unit (140) can be connected to the recognition unit (130). The display unit (140) can receive the output recognized from the recognition unit (130). The display unit (140) can display the status of the running railway track on which the railway vehicle is running based on the recognized output provided from the recognition unit (130), i.e., whether the recognition unit (130) has recognized it.

[0085] Here, the expression may include at least one of a visual expression (e.g., visualization or visualization), an auditory expression (e.g., generating a notification), and a control signal expression (e.g., generating a control signal capable of controlling a driving device of a railway vehicle), but is not limited thereto, and any method capable of expressing the state of a running railway track may be included.

[0086] Figure 3 is a block diagram for explaining a preprocessing unit according to the present embodiments.

[0087] Referring to FIG. 3, the preprocessing unit (120) according to the present embodiments may include at least one of an image segmentation unit (121), an image detection unit (122), and an image conversion unit (123). The image segmentation unit (121), the image detection unit (122), and the image conversion unit (123) may be connected to each other.

[0088] The image segmentation unit (121) can segment the input image to obtain a railway track area.

[0089] Here, a railway track area may include a single railway track area or multiple railway track areas. For clarity, the following description will refer to a railway track area without distinguishing between a single railway track area and multiple railway track areas.

[0090] Specifically, the image segmentation unit (121) can segment the input image into an area inside the railroad track and an area outside the railroad track, and obtain the area inside the railroad track as the railroad track area.

[0091] For example, the image segmentation unit (121) can segment the input image into an area inside the railroad track and an area outside the railroad track based on a deep learning-based image segmentation algorithm, and obtain the area inside the railroad track as the area outside the railroad track.

[0092] Here, the deep learning-based image segmentation algorithm may include an image segmentation algorithm that utilizes the U-Net++ structure with ResNet34 as the backbone.

[0093] Accordingly, the image segmentation unit (121) segments the input image into an area inside the railroad track and an area outside the railroad track based on an image segmentation algorithm utilizing a U-Net++ structure with ResNet34 as a backbone, and acquires the area inside the railroad track as the railroad track area, but can perform learning by using an image of the railroad track secured in advance as learning data and binary separating the area inside the railroad track and the area outside the railroad track as the correct answer data.

[0094] Here, the pre-secured railway track image may include at least one of the public dataset RailSem19 and the railway track image within the Gwangyang Steelworks, but is not limited thereto, and any railway track image that can be learned may be included.

[0095] To explain this more specifically, the railway track monitoring device (e.g., image segmentation unit (121)) according to the present embodiments can segment and generate only the railway track area from the input image by applying the acquired input image by photographing the front railway track through a camera on top of a railway vehicle (e.g., a locomotive, etc.) to an image segmentation technique utilizing a U-Net++ structure with ResNet34 as a backbone.

[0096] To this end, the railway track monitoring device (e.g., image segmentation unit (121)) according to the present embodiments can utilize at least one of the public dataset RailSem19 and a dataset that learned images of railway tracks in the Gwangyang Steelworks, and in particular, can perform learning using an image obtained by binary separating the internal area of ​​the railway track and the external area of ​​the railway line as correct data.

[0097] Through this, the railway track monitoring device according to the present embodiments can effectively segment and generate a railway track area of ​​a railway vehicle (e.g., a locomotive) that is difficult to detect using existing computer image processing techniques.

[0098] The image detection unit (122) can detect the center point of a running railway track in the railway track area.

[0099] For example, the image detection unit (122) can detect the center point of a running railroad track in the railroad track area based on the center point of the image.

[0100] Here, due to the characteristics of railway tracks of railway vehicles (e.g., locomotives, etc.), multiple railway tracks may overlap each other. The image center point (using the rule) can be used to distinguish the running railway tracks and extract the center point of the railway tracks, which can be used as preprocessing for anomaly detection.

[0101] Specifically, there may be multiple railway track areas. Accordingly, the image detection unit (122) detects the center point of each railway track corresponding to each railway track area in the multiple railway track areas, and can detect the center point of the running railway track based on the center point of the railway track located at a preset distance from the image center point.

[0102] Here, being located at a preset distance may include being located closest to the image center point. Accordingly, the image detection unit (122) may detect the center point of the running railroad track based on the center point of the railroad track located closest to the image center point. That is, the image detection unit (122) may detect the center point of the railroad track located closest to the image center point as the center point of the running railroad track.

[0103] In particular, for this purpose, the image detection unit (122) can detect the center point of the running railway track by clustering pixels in rows at a certain interval in the railway track area.

[0104] For example, the image detection unit (122) can detect the central point of the running railroad track by setting a range based on the middle of a row corresponding to the lower part of the railroad track area, detecting the central point of the row through distance-based clustering based on pixels that fall within the set range, and repeatedly performing clustering while moving to a row above at a certain interval.

[0105] To explain this more specifically, the image detection unit (122) is a center point detection unit of a running railway track, and can perform the role of detecting the center point of a running railway track on which a railway vehicle (e.g., a locomotive, etc.) is running among a plurality of railway track areas segmented and generated from an input image.

[0106] In particular, due to the nature of railway tracks, multiple railway tracks may overlap each other, and the running railway track can be distinguished through logic based on the rule that the track starts from the center point of the image. Accordingly, the image detection unit (122) can also distinguish the running railway track based on the rule that the track starts from the center point of the image.

[0107] In addition, the center point of the running railroad track can be used as the starting point for extracting the image area for dividing the railroad track area into up to seven parts. The method for detecting the center point of the running railroad track through the image detection unit (122) can be as follows. First, a range is set based on the middle of the row corresponding to the bottom of at least one image among the input image, the railroad track area, and the input image from which the railroad track area is divided, and the center point in the row can be detected through distance-based clustering based on the pixels falling within the range. By repeatedly performing clustering while moving to the upper row at a certain interval, the center points of the running railroad track can be detected.

[0108] The image conversion unit (123) can obtain a running railway track based on the railway track area and the center point of the running railway track, and can obtain a preprocessed image by converting the running railway track into a straight running railway track.

[0109] For example, the image conversion unit (123) can acquire a surrounding railroad track area as a running railroad track based on the center point of the running railroad track, and convert the running railroad track into a straight running railroad track to obtain a preprocessed image.

[0110] Specifically, the image conversion unit (123) obtains an area having a preset shape by specifying a surrounding railway track area based on a central point of a running railway track, obtains an area having a preset shape as a running railway track, crops the running railway track according to a preset length, warps the cropped running railway track to obtain a straight running railway track, and obtains a preprocessed image based on the straight running railway track.

[0111] Here, the preset shape may be a diamond shape, but is not limited thereto, and any shape that can represent a running railway track may be included.

[0112] Here, the preset length may be a length that can divide the running railway track into seven parts, but is not limited thereto and may include any length that can divide the running railway track.

[0113] Meanwhile, the image conversion unit (123) can generalize image data.

[0114] That is, the image conversion unit (123) can adjust the image to an image of the same size, i.e., generalize it, so that objects or obstacles can be detected well. In addition, the image conversion unit (123) can generalize the near image area and the far image area based on Gaussian blur so that objects or obstacles can be detected well.

[0115] For example, the image conversion unit (123) may obtain an area having a preset shape by specifying a surrounding railroad track area based on a central point of a running railroad track, obtain an area having a preset shape as a running railroad track, generalize a near area and a far area based on Gaussian blur, crop the Gaussian-blurred running railroad track according to a preset length, warp the cropped running railroad track to obtain a straight-line running railroad track, and obtain a preprocessed image based on the straight-line running railroad track.

[0116] In addition, the image conversion unit (123) obtains an area having a preset shape by specifying a surrounding railroad track area based on a central point of a running railroad track, obtains an area having a preset shape as a running railroad track, crops the running railroad track according to a preset length, adjusts the cropped running railroad track to the same size and generalizes a near area and a far area based on a Gaussian blur, warps the generalized and cropped running railroad track to obtain a straight-line running railroad track, and obtains a preprocessed image based on the straight-line running railroad track.

[0117] Here, weights can be assigned to the near and far areas. Accordingly, the image conversion unit (123) can assign different weights to the near and far areas.

[0118] For example, if the railway vehicle is a locomotive with a TLC (torpedo ladle car) attached, the image conversion unit (123) can assign different weights to the close area and the far area according to the distance depending on whether a 300-ton TLC is attached.

[0119] As described above, the image conversion unit (123) according to the present embodiments can divide the running railroad track into up to 7 equal parts according to the length of the running railroad track detected based on the center point of the running railroad track, and can convert the posture of the image to be close to a straight railroad track through an image processing technique corresponding to warping and cropping. In particular, the image conversion unit (123) according to the present embodiments can adjust the image to the same size by dividing the image into up to 7 equal parts and add a Gaussian blur to generalize the image data so that obstacles can be well detected regardless of whether it is a close area or a far area.

[0120] To be more specific, the image conversion unit (123) according to the present embodiments is an image region extraction and posture conversion unit, and crops and warps the running railway track region from the input image based on the segmented and generated railway track region and the center point of the running railway track, thereby modifying the running railway track into a straight generalized image form. This may be to perform more accurate unsupervised learning by utilizing it as an input image for a deep learning-based anomaly detection technique.

[0121] In addition, the image conversion unit (123) according to the present embodiments can divide the image into segments at close and far distances and assign a weight to each segmented image by distance depending on whether a 300-ton TLC is connected.

[0122] In addition, the image conversion unit (123) according to the present embodiments, for image region extraction, generates a diamond-shaped region for the running railroad track by designating a railroad track region around the center point of the running railroad track as a starting point, adds Gaussian blur to generalize the near and far regions, and divides this into up to 7 equal parts according to the length of the running railroad track to provide it as an input image for a deep learning-based anomaly detection technique.

[0123] In addition, the image conversion unit (123) according to the present embodiments can warp the input image into a bird wide view by performing a perspective transform algorithm to generalize the image into a running railroad track located in a straight line.

[0124] Fig. 4 is a block diagram for explaining a recognition unit according to the present embodiments.

[0125] Referring to FIG. 4, the recognition unit (130) according to the present embodiments may include at least one of an object recognition unit (131), an obstacle recognition unit (132), and an obstacle hazard recognition unit (133). The object recognition unit (131), the obstacle recognition unit (132), and the obstacle hazard recognition unit (133) may be connected to each other.

[0126] The object recognition unit (131) can recognize objects.

[0127] In particular, the object recognition unit (131) can recognize an amorphous object, but is not limited thereto, and can recognize any object.

[0128] For example, the object recognition unit (131) can recognize an irregular object that may be an obstacle on a running railroad track in a preprocessed image.

[0129] In particular, the object recognition unit (131) can recognize irregular objects that may be obstacles on a running railroad track in a preprocessed image based on unsupervised learning.

[0130] For example, when the object recognition unit (131) learns normal data and infers a preprocessed image in which an atypical object exists, it can detect the input image (or preprocessed image) as an abnormality.

[0131] That is, the object recognition unit (131) can judge the input image (or preprocessed image) as abnormal if it learns normal data and infers that an atypical object exists in the preprocessed image, and can judge the input image (or preprocessed image) as normal if it infers that normal data exists in the preprocessed image.

[0132] Here, the object recognition unit (131) learns the FastFlow technique through a preprocessed image of normal data, and learns the heat map obtained as a result of inferring the FastFlow using a Support Vector Machine (SVM), and performs binary machine learning to determine whether a normal or an atypical object is detected using the heat map as input data, so that if the heat map obtained from the normal data is inferred as input data, it is judged to be normal, and if the heat map obtained from the image containing the atypical object is inferred as input data, it is judged to be abnormal.

[0133] Through this, the object recognition unit (131) can determine the input image (or preprocessed image) as abnormal if it infers that an atypical object exists in the preprocessed image, and can determine the input image (or preprocessed image) as normal if it infers that normal data exists in the preprocessed image.

[0134] To explain this more specifically, the object recognition unit (131) can learn the FastFlow technique through a preprocessed image of normal data.

[0135] Here, FastFlow can include, but is not limited to, FastFlow based on Normalizing flow, and can include any learning technique as long as normal data can be learned through preprocessed images.

[0136] Here, the object recognition unit (131) can learn by adding preset layers to the last fully-connected layer in the structure of FastFlow.

[0137] Here, the normal data can utilize at least one of the public dataset RailSem19 and the dataset trained on images of railway tracks within the Gwangyang Steelworks.

[0138] In particular, the reason for adding layers is to increase the accuracy of anomaly detection of the railway track monitoring device (or object recognition unit (131)) according to the present embodiments.

[0139] Additionally, the preset layers may include, but are not limited to, three layers and may include one layer, two layers, or four or more layers.

[0140] For example, to improve the accuracy of anomaly detection, the object recognition unit (131) can learn each preprocessed image divided into up to 7 parts by adding the last fully-connected layer of the FastFlow structure to 3 layers.

[0141] Here, the first layer can learn the closest first region and second region in the preprocessed image divided into up to seven parts, the second layer can learn the third region and fourth region in the preprocessed image divided into up to seven parts, and the third layer can learn the furthest fifth region, sixth region, and seventh region in the preprocessed image divided into up to seven parts.

[0142] The object recognition unit (131) can perform binary machine learning to determine whether a normal or atypical object is detected using the heat map as input data by learning the heat map obtained as a result of inferring FastFlow using a support vector machine (SVM).

[0143] Here, SVM can include, but is not limited to, One-class SVM and can include anything that can learn heat maps.

[0144] That is, the object recognition unit (131) can perform unsupervised learning using a heat map obtained by inferring normal data as input, and can learn to determine that the input data is normal when the heat map obtained from the normal data is inferred as input data, and to determine that the input data is abnormal when the heat map obtained from an image containing an atypical object is inferred as input data.

[0145] Through this, the object recognition unit (131) can determine the input image (or preprocessed image) as abnormal if it infers that an atypical object exists in the preprocessed image, and can determine the input image (or preprocessed image) as normal if it infers that normal data exists in the preprocessed image.

[0146] As described above, the object recognition unit (131) applies the preprocessed image to an image anomaly detection algorithm utilizing a FastFlow technique based on a Normalizing flow to display (or detect) an abnormal part in the input image (or preprocessed image) in the form of a heat map, and learns the displayed (or detected) heat map through a One-class SVM to determine whether an anomaly is detected.

[0147] That is, the object recognition unit (131) can utilize unsupervised learning to detect anomalies corresponding to atypical objects in an input image (or preprocessed image). This can detect anomalies in input images (or preprocessed images) that are judged to be normal by learning a set of input images (or preprocessed images) that are judged to be normal, and when an input image (or preprocessed image) containing an atypical object is inferred, that part can be detected as an anomaly.

[0148] The obstacle recognition unit (132) can recognize obstacles.

[0149] The obstacle recognition unit (132) can recognize irregular obstacles, but is not limited thereto, and can recognize any obstacle.

[0150] For example, the obstacle recognition unit (132) can recognize whether an irregular object is an obstacle.

[0151] That is, the obstacle recognition unit (132) can determine whether an irregular object is an obstacle and, based on the determination result, recognize the irregular object as an irregular obstacle.

[0152] Specifically, the obstacle recognition unit (132) determines whether an amorphous object is an obstacle, and if the amorphous object is determined to be an obstacle as a result of the determination, the amorphous object can be recognized as an amorphous obstacle.

[0153] In addition, the obstacle recognition unit (132) determines whether an amorphous object is an obstacle, and if the determination result determines that the amorphous object is not an obstacle, the amorphous object can be recognized as a final amorphous object.

[0154] Here, the obstacle recognition unit (132) can recognize whether an irregular object is an obstacle based on an obstacle recognition algorithm.

[0155] In particular, the obstacle recognition algorithm may include, but is not limited to, a feature map algorithm, and may include any algorithm capable of recognizing obstacles.

[0156] Here, the obstacle recognition unit (132) can recognize whether an irregular object is an obstacle based on unsupervised learning. Unsupervised learning can be applied by changing an object into an obstacle in the unsupervised learning described in the object recognition unit (131), i.e., by changing the object into an obstacle in the unsupervised learning described in the object recognition unit (131). Therefore, a detailed description will be omitted for clarity.

[0157] The obstacle risk recognition unit (133) can recognize the risk of obstacles.

[0158] The obstacle risk recognition unit (133) can recognize the risk of irregular obstacles, but is not limited thereto, and can recognize the risk of any obstacle.

[0159] For example, the obstacle risk recognition unit (133) can recognize the risk of an irregular obstacle.

[0160] That is, the obstacle risk recognition unit (133) can determine whether an irregular obstacle is located on the running railway track and recognize the risk of the irregular obstacle based on the determination result.

[0161] Specifically, the obstacle risk recognition unit (133) determines whether an irregular obstacle is located on the running railway track, and if the determination result shows that an irregular obstacle is located on the running railway track, the irregular obstacle can be recognized as dangerous.

[0162] In addition, the obstacle risk recognition unit (133) determines whether an irregular obstacle is located on the running railway track, and if the determination result shows that the irregular obstacle is not located on the running railway track, the irregular obstacle can be recognized as not being dangerous.

[0163] Here, the obstacle risk recognition unit (133) can recognize the risk of an irregular obstacle based on an obstacle risk recognition algorithm.

[0164] In particular, the obstacle hazard recognition algorithm may include any algorithm that can determine whether an obstacle is located on the running railway.

[0165] Here, the obstacle risk recognition unit (133) can recognize the risk of irregular obstacles based on unsupervised learning. Unsupervised learning can be applied by changing the object described in the object recognition unit (131) to an obstacle located on a running railway, i.e., by applying the unsupervised learning described in the object recognition unit (131). Therefore, a detailed description will be omitted for clarity.

[0166] Fig. 5 is a block diagram for explaining the display unit according to the present embodiments.

[0167] Referring to FIG. 5, the display unit (140) according to the present embodiments may include at least one of an image inversion unit (141), a display unit (142), a notification generation unit (143), and a control signal generation unit (144). The image inversion unit (141), the display unit (142), the notification generation unit (143), and the control signal generation unit (144) may be connected to each other.

[0168] The image inverse conversion unit (141) can obtain an output image by inversely converting the output recognized by the recognition unit (130).

[0169] For example, the image inverse transformation unit (141) can obtain an output image by inversely transforming and mapping the output recognized by the recognition unit (130) to fit the input image.

[0170] That is, the image inverse transformation unit (141) can inversely transform the output recognized by the recognition unit (130) into an area having a preset shape of a running railway track obtained by the image transformation unit (123), and obtain an output image by mapping it to an input image.

[0171] Specifically, the image inverse transformation unit (141) obtains a vertex corresponding to an area having a preset shape of a running railway track obtained from the image transformation unit (123), obtains a vertex of an area recognized by the recognition unit (e.g., an area where an irregular object exists, an area where an irregular obstacle exists, an area where an irregular obstacle exists on a running railway, etc.), and aligns the vertex of the area recognized by the recognition unit (130) with the vertex corresponding to the area having a preset shape of the running railway track obtained from the image transformation unit (123) to inversely transform the area recognized by the recognition unit into an area having a preset shape of the running railway track obtained from the image transformation unit, and maps it to an input image to obtain an output image.

[0172] Here, the preset shape may include a rhombus shape. Accordingly, the area having the preset shape may include a rhombus-shaped area. Accordingly, the vertex may be a point of 4.

[0173] As described above, the image inverse transformation unit (141) according to the present embodiments is a reverse transformation mapping unit, and can reverse transform and map an image determined to be abnormal by the recognition unit into a diamond-shaped area before perspective transformation by the image transformation unit. This can be structured to store four points corresponding to the vertices of the diamond-shaped area of ​​the image transformation unit (or before area extraction and posture transformation) and then recall the vertices of the area determined to be abnormal by the recognition unit.

[0174] The display unit (142) can visually display the status of the running railway track based on the recognized output of the recognition unit.

[0175] For example, the display unit (142) can visually display the state of the running railway track based on the recognized output of the recognition unit in the form of an input image of the image acquisition unit.

[0176] Specifically, the display unit (142) can visually display the state on the running railway track by visualizing the output image, i.e., the output image acquired from the image inversion unit (141) based on the recognized output of the recognition unit, in the form of an input image, i.e., an input image from the image acquisition unit.

[0177] As described above, the display unit (142) according to the present embodiments can play a role in visualizing a diamond-shaped area, which is determined to be abnormal due to the presence of an irregular obstacle obtained through the image inversion unit (141), in a diamond shape in the area of ​​the running railway track in the input image obtained by the image acquisition unit. Through this, the railway track monitoring device according to the present embodiments can provide a judgment as to whether an irregular obstacle exists at a distance of 50 m or more on the railway track of a locomotive pulling a heavy TLC in an autonomous driving system of the locomotive or in an autonomous driving system of the locomotive, and at which location on the running railway track the irregular obstacle exists.

[0178] The notification generating unit (143) can audibly display the status of the running railway track based on the recognized output of the recognition unit.

[0179] For example, the notification generating unit (143) can audibly display the status on the running railway track based on at least one recognition result among the recognition of an atypical object, recognition of an atypical obstacle, and recognition of the risk of an atypical obstacle of the recognition unit.

[0180] In one example, the notification generating unit (143) can generate a notification that can notify of an atypical object on a running railway track when the recognition unit recognizes an atypical object.

[0181] In another example, the notification generating unit (143) can generate a notification that can notify of an irregular obstacle on a running railway track when the recognition unit recognizes an irregular obstacle.

[0182] In another example, the notification generating unit (143) can generate a notification to notify of the risk of an irregular obstacle on a running railway track when the recognition unit recognizes the risk of an irregular obstacle.

[0183] That is, when the recognition unit recognizes that an irregular obstacle is located on the running railway track and is dangerous, the notification generation unit (143) can generate a notification that an irregular obstacle is located on the running railway track and is dangerous.

[0184] Here, the notifications capable of notifying an atypical object on a running railway track, the notifications capable of notifying an atypical obstacle on a running railway track, and the notifications capable of notifying the danger of an atypical obstacle on a running railway track may use the same notification, or may use different notifications, or may use only some of the same notifications, or may use only some of the different notifications.

[0185] Here, the notification capable of notifying an atypical object on a running railway track, the notification capable of notifying an atypical obstacle on a running railway track, and the notification capable of notifying a risk of an atypical obstacle on a running railway track can generate different notifications based on at least one of the approach distance between the railway vehicle and at least one of the atypical object and the atypical obstacle, and the speed of the railway vehicle.

[0186] The control signal generation unit (144) can display the status of the running railway as a control signal based on the recognized output.

[0187] For example, the control signal generation unit (144) can display the status on the running railway track as a control signal based on at least one recognition result among the recognition of an irregular object, recognition of an irregular obstacle, and recognition of the risk of an irregular obstacle of the recognition unit.

[0188] That is, the control signal generation unit (144) can generate a control signal capable of controlling the railway vehicle based on at least one recognition result among the recognition of an irregular object, recognition of an irregular obstacle, and recognition of the risk of an irregular obstacle of the recognition unit.

[0189] In one example, when the recognition unit recognizes an atypical object, the control signal generation unit (144) can generate a control signal capable of controlling a railway vehicle in response to the atypical object on the running railway track.

[0190] In another example, when the recognition unit recognizes an irregular obstacle, the control signal generation unit (144) can generate a control signal capable of controlling a railway vehicle in response to an irregular obstacle on a running railway track.

[0191] In another example, when the recognition unit recognizes the risk of an irregular obstacle, the control signal generation unit (144) can generate a control signal capable of controlling a railway vehicle in response to the risk of an irregular obstacle on a running railway track.

[0192] That is, when the recognition unit recognizes that an irregular obstacle is located on the running railway track and is in danger, the control signal generation unit (144) can generate a control signal capable of controlling the railway vehicle in response to the danger of the irregular obstacle located on the running railway track.

[0193] Here, the control signal capable of controlling the railway vehicle in response to an irregular object on the running railway track, the control signal capable of controlling the railway vehicle in response to an irregular obstacle on the running railway track, and the control signal capable of controlling the railway vehicle in response to the risk of an irregular obstacle on the running railway track may use the same control signal, or may use different control signals, or may use only some of the same control signals, or may use only some of the different control signals.

[0194] Here, a control signal capable of controlling a railway vehicle in response to an atypical object on a running railway track, a control signal capable of controlling a railway vehicle in response to an atypical obstacle on a running railway track, and a control signal capable of controlling a railway vehicle in response to a risk of an atypical obstacle on a running railway track can generate a control signal differently based on at least one of an approach distance between the railway vehicle and at least one of the atypical object and the atypical obstacle and a speed of the railway vehicle.

[0195] Meanwhile, the control signal generated by the control signal generation unit (144) can be provided to the control device of the railway vehicle. The control device of the railway vehicle can control the driving device of the railway vehicle based on the control signal, thereby controlling the driving (or operation) of the railway vehicle.

[0196] Hereinafter, a railway track monitoring method according to the present embodiments will be described with reference to the attached drawings. The railway track monitoring method according to the present embodiments can be performed using a railway track monitoring device and a railway vehicle. Accordingly, for the sake of clarity, any portions overlapping with the railway track monitoring device and railway vehicle according to the present embodiments described above with reference to FIGS. 1 to 5 will be omitted below.

[0197] Figure 6 is a flowchart for explaining a railway track monitoring method according to the present embodiments.

[0198] Referring to FIG. 6, the railway track monitoring method according to the present embodiments may include at least one step among an input image acquisition step (S100), a preprocessing image acquisition step (S200), a state recognition step on a running railway track (S300), a state display step on a running railway track (S400), and a railway vehicle operation status judgment step (S500).

[0199] For example, a method for monitoring a railway track according to the present embodiments may include a step of obtaining a running railway track from an input image, a step of converting the running railway track into a straight running railway track to obtain a preprocessed image, and a step of recognizing an irregular object that may be an obstacle existing on the running railway track from the preprocessed image.

[0200] To explain specifically, first, a preprocessed image can be obtained (S200).

[0201] For example, in step S100, a running railway track can be obtained from an input image, and a preprocessed image can be obtained by converting the running railway track into a straight running railway track.

[0202] Afterwards, the condition on the running railway track can be recognized (S300).

[0203] Specifically, in step S300, an amorphous object can be recognized.

[0204] For example, in step S300, an irregular object that may be an obstacle on a running railway track can be recognized in a preprocessed image.

[0205] Meanwhile, the railway track monitoring method according to the present embodiments may further include an input image acquisition step (S100). That is, the input image acquisition step (S100) may further be included before the preprocessing image acquisition step (S200).

[0206] In step S100, an input image can be acquired.

[0207] For example, in step S100, an input image can be obtained by photographing the external environment of a railway vehicle.

[0208] Meanwhile, the railway track monitoring method according to the present embodiments may further include a status display step (S400) on the running railway track. That is, after the status recognition step (S200) on the running railway track, the status display step (S400) on the running railway track may further be included.

[0209] In step S400, the status of the running railway track can be displayed.

[0210] For example, in step S400, the status of the running railway track can be displayed based on the recognition result in step S300.

[0211] Specifically, in step S400, the output recognized in step S300 can be reverse-converted to obtain an output image.

[0212] For example, in step S400, the output recognized in step S300 can be mapped inversely to the input image to obtain an output image.

[0213] That is, in step S400, the output recognized in step S300 is reverse-converted into a region having a preset shape of a running railway track obtained in step S200, and mapped to an input image to obtain an output image.

[0214] In step S400, the status on the running railway track can be visually displayed based on the output recognized in step S300.

[0215] For example, in step S400, the state on the running railway track can be visualized and displayed in the form of an input image in step S100 based on the output recognized in step S300.

[0216] Specifically, in step S400, an output image, i.e., an output image obtained by performing image inverse transformation based on the output recognized in S300, is visualized in the form of an input image, i.e., an input image in step S100, so that the state on the running railway track can be visually expressed.

[0217] In step S400, the status on the running railway track can be audibly displayed based on the output recognized in step S300.

[0218] For example, in step S400, the status on the running railway track can be audibly displayed based on the recognition result of at least one of the atypical object recognition, atypical obstacle recognition, and atypical obstacle risk recognition in step S300.

[0219] In step S400, the state of the running railway can be expressed as a control signal based on the output recognized in step S300.

[0220] For example, in step S400, the status on the running railway track can be expressed as a control signal based on the recognition result of at least one of the atypical object recognition, atypical obstacle recognition, and atypical obstacle risk recognition in step S300.

[0221] Here, the control signal generated in step S400 can control the driving device of the railway vehicle, thereby controlling the driving (or operation) of the railway vehicle.

[0222] Meanwhile, the railway track monitoring method according to the present embodiments may further include a railway vehicle operation status determination step (S500). That is, after the status display step (S400) on the running railway track, the railway vehicle operation status determination step (S500) may be further included.

[0223] Step S500 can determine the operating status of railway vehicles.

[0224] For example, in step S500, the operating status of the railway vehicle is determined, and if the determination result shows that the railway vehicle is in operation, an input image can be acquired (S100).

[0225] Additionally, in step S500, the operating status of the railway vehicle is determined, and if the determination result indicates that the railway vehicle has stopped operating, the railway track monitoring can be terminated.

[0226] Meanwhile, the railway vehicle control method according to the present embodiments may include the railway track monitoring step and the railway vehicle control step described above.

[0227] Here, the railway vehicle control stage can control the operation of the railway vehicle based on the results of monitoring the running railway track. For example, the railway vehicle control stage can control the operation of the railway vehicle by controlling the driving device of the railway vehicle based on the control signal from the railway track monitoring stage.

[0228] Figure 7 is a flowchart for explaining a preprocessing image acquisition method according to the present embodiments.

[0229] Referring to FIG. 7, the preprocessing image acquisition method according to the present embodiments may include at least one of a railway track area acquisition step (S210), a central point detection step of a running railway track (S220), and a preprocessing image acquisition step (S230).

[0230] First, the input image can be segmented to obtain a railway track area (S210).

[0231] Specifically, in step S210, the input image is divided into a region inside the railway track and a region outside the railway track, and the region inside the railway track can be acquired as a region outside the railway track.

[0232] For example, in step S210, based on a deep learning-based image segmentation algorithm, the input image can be segmented into a region inside a railroad track and a region outside a railroad track, and the region inside a railroad track can be acquired as a railroad track region.

[0233] Afterwards, the center point of the running railway track can be detected in the railway track area (S220).

[0234] For example, in step S220, the center point of a running railway track can be detected in the railway track area based on the center point of the image.

[0235] Specifically, there may be multiple railway track areas. Accordingly, in step S220, the center point of each railway track corresponding to each railway track area in the multiple railway track areas is detected, and the center point of the running railway track can be detected based on the center point of the railway track located at a preset distance from the image center point.

[0236] Here, the positioning at a preset distance may include the position closest to the image center point. Accordingly, in step S220, the center point of the running railway track can be detected based on the center point of the railway track located closest to the image center point. That is, in step S220, the center point of the railway track located closest to the image center point can be detected as the center point of the running railway track.

[0237] In particular, for this purpose, in step S220, the central point of the running railway track can be detected by clustering pixels in the railway track area into rows at a certain interval.

[0238] Afterwards, a running railway track can be obtained based on the railway track area and the center point of the running railway track, and a preprocessing image can be obtained by converting the running railway track into a straight running railway track (S230).

[0239] For example, in step S230, a preprocessing image can be obtained by acquiring a surrounding railway track area based on a central point of a running railway track as a running railway track, and converting the running railway track into a straight running railway track.

[0240] Specifically, in step S230, a region having a preset shape is obtained by specifying a region of a surrounding railway track based on a central point of a running railway track, an region having a preset shape is obtained as a running railway track, the running railway track is cropped according to a preset length, the cropped running railway track is warped to obtain a straight running railway track, and a preprocessed image can be obtained based on the straight running railway track.

[0241] Fig. 8 is a flowchart for explaining a method for recognizing the state of a running railway track according to the present embodiments.

[0242] Referring to FIG. 8, the method for recognizing the state of a running railway track according to the present embodiments may include at least one of an object recognition step (S310), an obstacle recognition step (S320), and an obstacle risk recognition step (S330).

[0243] First, the object can be recognized (S310).

[0244] For example, in step S310, an irregular object that may be an obstacle on a running railway track can be recognized in a preprocessed image.

[0245] In particular, in step S310, based on unsupervised learning, an irregular object that may be an obstacle on a running railway track can be recognized in a preprocessed image.

[0246] For example, in step S310, when normal data is learned and a preprocessed image in which an atypical object exists is inferred, the input image (or preprocessed image) can be detected as an abnormality.

[0247] That is, in step S310, if it is inferred that an atypical object exists in the preprocessed image by learning normal data, the input image (or preprocessed image) can be judged as abnormal, and if it is inferred that normal data exists in the preprocessed image, the input image (or preprocessed image) can be judged as normal.

[0248] Afterwards, obstacles can be recognized (S320).

[0249] For example, in step S320, it can be recognized whether an irregular object is an obstacle.

[0250] That is, in step S320, it is determined whether the amorphous object is an obstacle, and based on the determination result, the amorphous object can be recognized as an amorphous obstacle.

[0251] Specifically, in step S320, it is determined whether the amorphous object is an obstacle, and if the amorphous object is determined to be an obstacle as a result of the determination, the amorphous object can be recognized as an amorphous obstacle.

[0252] Additionally, in step S320, it is determined whether the amorphous object is an obstacle, and if the amorphous object is not determined to be an obstacle as a result of the determination, the amorphous object can be recognized as a final amorphous object.

[0253] Afterwards, the risk of obstacles can be recognized (S330).

[0254] For example, in step S330, the risk of an irregular obstacle can be recognized.

[0255] That is, in step S330, it is determined whether an irregular obstacle is located on the running railway track, and the risk of the irregular obstacle can be recognized based on the determination result.

[0256] Specifically, step S330 determines whether an irregular obstacle is located on the running railway track. If the determination result indicates that an irregular obstacle is located on the running railway track (Y), the irregular obstacle can be recognized as dangerous. Then, step S400 can be performed.

[0257] Additionally, in step S330, it is determined whether an irregular obstacle is located on the running railway track. If the determination result indicates that the irregular obstacle is not located on the running railway track (N), the irregular obstacle can be recognized as not dangerous. Then, step 500 can be performed.

[0258]

[0259] FIG. 9 is a block diagram of a railway track monitoring device and a railway vehicle computer system according to the present embodiments.

[0260] Referring to FIG. 9, the above-described embodiments may be implemented in a computer system, for example, as a computer-readable recording medium. As illustrated in the drawing, a computer system (1000) such as a railway track monitoring device and a railway vehicle may include at least one or more elements of a processor (1010), a memory (1020), a storage unit (1030), a user interface input unit (1040), and a user interface output unit (1050), which may communicate with each other via a bus (1060). In addition, the computer system (1000) may also include a network interface (1070) for connecting to a network. The processor (1010) may be a CPU or a semiconductor device that executes processing instructions stored in the memory (1020) and / or the storage unit (1030). The memory (1020) and the storage unit (1030) may include various types of volatile / non-volatile storage media. For example, the memory may include ROM (1024) and RAM (1025).

[0261] Accordingly, the present embodiments may be implemented as a computer-implemented method or as a non-volatile computer storage medium storing computer-executable instructions. The instructions, when executed by a processor, may perform a method according to at least one embodiment of the present embodiments.

[0262]

[0263] As described above, the railway track monitoring device and method and the railway vehicle according to the present embodiments are directed to deep learning-based railway track detection and anomaly detection that detects a railway track area through an image and recognizes an atypical object (or obstacle) on the detected railway track to prevent accidents of a locomotive running on a railway track and to recognize an object (or obstacle) for autonomous locomotive driving.

[0264] That is, the railway track monitoring device and method and the railway vehicle according to the present embodiments can acquire 2D RGB images in real time through cameras attached to the front and rear of the locomotive, and recognize an irregular object (or obstacle) on the railway track of a locomotive connected to a TLC (Torpedo Ladle Car) containing a 300-ton molten iron through deep learning-based region segmentation and anomaly detection algorithms.

[0265] Therefore, the railway track monitoring device and method according to the present embodiments and the railway vehicle can detect the railway tracks of the locomotive using a deep learning-based image recognition technique to determine whether a long-distance atypical object (or obstacle) of a locomotive pulling a heavy TLC is detected as an atypical object (or obstacle) on the railway tracks, and detect the detected atypical object (or obstacle) on the railway tracks as an abnormality.

[0266] Through this, the railway track monitoring device and method according to the present embodiments and the railway vehicle can detect railway tracks corresponding to a distance of 50 m or more from a running locomotive through a deep learning-based image processing method proposed based on a camera installed on top of a locomotive, and detect irregular objects (or obstacles) on the detected railway tracks as abnormalities, thereby reducing the risk of accidents of the running locomotive.

[0267] In addition, the railway track monitoring device and method and the railway vehicle according to the present embodiments are a computer vision-based artificial intelligence technology that indicates the area of ​​a specific point in an image, and in particular, a deep learning technology that imitates the cognitive ability of the human brain to compute and process numerous weighted variables of a nonlinear structure, and can bring the automatic image processing ability using a computer close to the level of a human expert.

[0268] In addition, the railway track monitoring device and method and the railway vehicle according to the present embodiments can solve various problems by applying automatic image processing techniques using computer vision to railway fields such as autonomous locomotive driving and track abnormality detection. In particular, the railway track monitoring device and method and the railway vehicle according to the present embodiments can reduce reasons such as carelessness and judgment errors of locomotive drivers running on railway tracks by applying deep learning-based image processing techniques to accurately detect obstacles on railway tracks, thereby preventing major accidents in advance.

[0269] In addition, the railway track monitoring device and method and the railway vehicle according to the present embodiments can accurately recognize a flat, irregularly shaped object (or obstacle) near the railway track.

[0270] In addition, the railway track monitoring device and method and the railway vehicle according to the present embodiments can accurately distinguish and detect a background of various gravel existing on the railway track of a locomotive and a dark-colored railway track located thereon.

[0271] In particular, when the railway track monitoring device and method and railway vehicle according to the present embodiments are applied to a locomotive loaded with a heavy load, such as a TLC loaded with 300 tons of coal, the braking distance is detected to be longer in proportion to the weight, thereby enabling accurate detection of objects (or obstacles) at a distance, thereby further improving the technological capabilities of recognition technology.

[0272]

[0273] The above description is merely an illustrative example of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations can be made without departing from the essential characteristics of the technical idea of ​​the present disclosure. In addition, the present embodiments are not intended to limit the technical idea of ​​the present disclosure but rather to explain it, and therefore the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included within the scope of the rights of the present disclosure.

[0274]

[0275] CROSS-REFERENCE TO RELATED APPLICATION

[0276] This patent application claims priority under 35 USC § 119(a) to Korean Patent Application No. 10-2023-0183157, filed December 15, 2023, the entire contents of which are incorporated herein by reference. Furthermore, this patent application claims priority in countries other than the United States for the same reasons, the entire contents of which are incorporated herein by reference.

Claims

1. A preprocessing unit that obtains a running railroad track from an input image and converts the running railroad track into a straight running railroad track to obtain a preprocessing image; and A railway track monitoring device including a recognition unit that recognizes an irregular object that may be an obstacle on a running railway track in the above preprocessed image.

2. In paragraph 1, The above preprocessing unit, An image segmentation unit for segmenting the above input image to obtain a railway track area; An image detection unit for detecting the center point of a running railway track in the above railway track area; and A railway track monitoring device including an image conversion unit that obtains the running railway track based on the railway track area and the center point of the running railway track, and converts the running railway track into the straight-line running railway track to obtain the preprocessed image.

3. In paragraph 2, The above video segmentation part, A railway track monitoring device that divides the above input image into a railway track interior region and a railway track exterior region, and acquires the railway track interior region as the railway track region.

4. In paragraph 2, The above image detection unit, A railway track monitoring device that detects the center point of each railway track corresponding to each railway track area in a plurality of railway track areas, and detects the center point of the running railway track based on the center point of the railway track located at a preset distance from the image center point.

5. In paragraph 4, The above image conversion unit, A railway track monitoring device that acquires a railway track area surrounding a central point of the above-mentioned running railway track as the above-mentioned running railway track, converts the above-mentioned running railway track into a straight-line running railway track, and acquires a preprocessed image.

6. In paragraph 5, The above image conversion unit, A railway track monitoring device that obtains an area having a preset shape by specifying a railway track area surrounding a central point of the above-described running railway track as a reference point, obtains the area having the preset shape as a running railway track, crops the running railway track according to a preset length, warps the cropped running railway track to obtain a straight running railway track, and obtains the preprocessed image based on the straight running railway track.

7. In paragraph 1, The above recognition unit, A railway track monitoring device including an object recognition unit that recognizes an atypical object that may be an obstacle on a railway track in the preprocessed image based on unsupervised learning.

8. In paragraph 7, The above object recognition unit, A railway track monitoring device that detects a preprocessed image as an abnormality when a preprocessed image in which an atypical object exists is inferred by learning normal data.

9. In paragraph 8, The above object recognition unit, A railway track monitoring device that learns the FastFlow technique through the above-mentioned preprocessed image, learns the heat map obtained as a result of inferring the FastFlow using a Support Vector Machine (SVM), and performs binary machine learning to determine whether the heat map is normal or an atypical object is detected as input data, so that if the heat map obtained from the normal data is inferred as input data, it is judged to be normal, and if the heat map obtained from the image containing the atypical object is inferred as input data, it is judged to be abnormal.

10. In paragraph 7, The above recognition unit, A railway track monitoring device further comprising an obstacle recognition unit that determines whether an amorphous object is an obstacle and recognizes the amorphous object as an amorphous obstacle based on the determination result.

11. In Article 10, The above recognition unit, A railway track monitoring device further comprising an obstacle hazard recognition unit that determines whether an irregular obstacle is located on a running railway track and recognizes the hazard of the irregular obstacle based on the determination result.

12. In paragraph 1, A railway track monitoring device further comprising a display unit that displays the status of a running railway track based on the recognition result of the above recognition unit.

13. In paragraph 12, The above display part is, An image inverse transformation unit that obtains an output image by inversely transforming the output recognized by the above recognition unit; and A railway track monitoring device including a display unit that visualizes the output image in the form of the input image to visually display the status of the running railway track.

14. A step of obtaining a running railway track from an input image and converting the running railway track into a straight running railway track to obtain a preprocessing image; and A railway track monitoring method comprising a step of recognizing an atypical object that may be an obstacle existing on a running railway track in the above preprocessed image.

15. In paragraph 14, The step of obtaining the above preprocessed image is: A step of dividing the above input image to obtain a railway track area; Step of detecting the center point of the running railway track in the above railway track area: and A railway track monitoring method comprising the steps of obtaining the running railway track based on the railway track area and the center point of the running railway track, and converting the running railway track into the straight-line running railway track to obtain the preprocessed image.

16. In paragraph 15, The step of dividing the above input image to obtain the railway track area is as follows. A railway track monitoring method comprising: dividing the above input image into a railway track interior region and a railway track exterior region, and obtaining the railway track interior region as the railway track region.

17. In paragraph 15, The step of detecting the center point of the running railway track in the above railway track area is: A railway track monitoring method comprising: detecting the center point of each railway track corresponding to each railway track area in a plurality of railway track areas; and detecting the center point of the running railway track based on the center point of the railway track located at a preset distance from the image center point.

18. In paragraph 15, The step of obtaining the above preprocessed image is: A railway track monitoring method comprising: acquiring a railway track area surrounding a central point of the above-mentioned running railway track as the above-mentioned running railway track; and converting the above-mentioned running railway track into a straight-line running railway track to acquire a preprocessed image.

19. In paragraph 14, The step of recognizing the above amorphous object is: A railway track monitoring method for recognizing an atypical object that may be an obstacle on a running railway track in the preprocessed image based on unsupervised learning.

20. A railway track monitoring device that obtains a running railway track from an input image, converts the running railway track into a straight running railway track to obtain a preprocessed image, recognizes an irregular object that may be an obstacle on the running railway track from the preprocessed image, and displays the state of the running railway as a control signal based on the recognized output; and A railway vehicle including a control device that controls the running of the railway vehicle by controlling the running device of the railway vehicle based on the above control signal.

Images