Apparatus and method for detecting obstacle

Abstract

The present disclosure relates to a railway vehicle autonomous driving technology, and provides an apparatus and a method for detecting an obstacle by using multiple camera sensors, the apparatus comprising: a reception unit for receiving two or more pieces of image information generated at the same time from two or more camera sensors, respectively; an image integration unit for converting the two or more pieces of image information into one integrated image; a region of interest setting unit for setting a region of interest in the integrated image, by using location information and control information of a railway vehicle; and an obstacle detection unit for detecting an obstacle in the region of interest.

Description

Obstacle detection device and method thereof

[0001] The present disclosure relates to autonomous driving technology for railway vehicles. More specifically, it relates to obstacle detection technology for autonomous driving of railway vehicles.

[0002] Traditionally, railroad cars were operated by engineers directly, following signals displayed on signals. While the central control center assists the engineer by operating the track switches when necessary, this conventional method relies entirely on the engineer's perception and judgment of the surroundings, raising the risk of accidents.

[0003] Accordingly, recently, alarm devices have been installed inside railway vehicles and devices that detect emergency situations and automatically stop the railway vehicle have been developed and are being used.

[0004] In particular, in the case of railway vehicles used for transporting molten iron within steel mills, the risk may be higher than that of operating general railway vehicles on separate tracks, as they operate within a specific business site and there are numerous unrefined risks.

[0005] Furthermore, the size and shape of a railway vehicle limits the detection range of camera sensors and other devices. This issue can lead to limitations in the detection range of multiple camera sensors, even when configured with multiple sensors. Furthermore, when integrating and processing image information from multiple camera sensors, increased computational speeds limit the timely detection of potential dangerous situations in real time.

[0006] Therefore, conventional technology has limitations in preventing risks caused by human and unexpected factors, and its ability to detect obstacles in advance through high computational speeds is also limited. A fast and accurate obstacle detection technology that can address these issues is needed.

[0007] The present disclosure aims to provide autonomous driving technology for railway vehicles.

[0008] In one aspect, the present embodiments provide an obstacle detection device using a multi-camera sensor, comprising a receiving unit that receives two or more image information generated at the same time from two or more camera sensors, an image integration unit that converts the two or more image information into one integrated image, a region of interest setting unit that sets a region of interest in the integrated image using location information and control information of a railway vehicle, and an obstacle detection unit that detects an obstacle in the region of interest.

[0009] In another aspect, the present embodiments provide an obstacle detection method using a multi-camera sensor, comprising a receiving step of receiving two or more image information generated at the same time from two or more camera sensors, an image integration step of converting the two or more image information into one integrated image, a region of interest setting step of setting a region of interest in the integrated image using location information and control information of a railway vehicle, and an obstacle detection step of detecting an obstacle in the region of interest.

[0010] According to the present disclosure, autonomous driving technology for railway vehicles can be provided.

[0011] FIG. 1 is a schematic diagram illustrating a railway vehicle autonomous driving system according to one embodiment.

[0012] FIG. 2 is a diagram illustrating a configuration of an obstacle detection device according to one embodiment.

[0013] FIG. 3 is a diagram for explaining an operation of receiving two or more image information according to one embodiment.

[0014] FIG. 4 is a diagram for explaining an integrated image generation operation according to one embodiment.

[0015] FIG. 5 is a diagram for explaining an operation of setting a region of interest on an integrated image according to one embodiment.

[0016] FIG. 6 is a drawing for explaining an operation of setting a region of interest on an integrated image according to another embodiment.

[0017] FIG. 7 is a drawing for explaining an obstacle detection method according to one embodiment.

[0018] FIG. 8 is a drawing for explaining an obstacle detection method according to another embodiment.

[0019] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, identical components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the technical idea of ​​the present invention, the detailed description may be omitted. When "includes," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it may include a case in which the plural is included unless specifically stated otherwise.

[0020] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms.

[0021] In a description of the positional relationship of components, when it is described that two or more components are "connected," "combined," or "connected," it should be understood that the two or more components may be directly "connected," "combined," or "connected," but that the two or more components may also be further "interposed" with another component to be "connected," "combined," or "connected." Here, the other component may be included in one or more of the two or more components that are "connected," "combined," or "connected" to each other.

[0022] In the description of the temporal flow relationship related to components, operation methods, or manufacturing methods, for example, when the temporal or flow relationship is described as “after”, “following”, “next to”, “before”, etc., it may also include cases where it is not continuous, unless “immediately” or “directly” is used.

[0023] Meanwhile, when numerical values ​​or corresponding information (e.g., levels, etc.) for components are mentioned, even without separate explicit description, the numerical values ​​or corresponding information may be interpreted as including an error range that may occur due to various factors (e.g., process factors, internal or external impact, noise, etc.).

[0024] The embodiments are described in detail with reference to the drawings below.

[0025]

[0026] When transporting large quantities of materials, such as steel mills, larger vehicles like trains may be required rather than smaller vehicles like cars. In particular, for multi-ton vessels, the weight and characteristics of the vessels pose limitations on transporting them using smaller vehicles.

[0027] Furthermore, when transporting materials within a specific business location, such as a factory, the material production and processing plants are designated, and the material transport routes are relatively fixed. Therefore, considering these characteristics, the use of rail vehicles capable of transporting large quantities of materials via rail is essential.

[0028] In the case of actual iron transport railways used in steel mills, they are used to transport iron produced in blast furnaces to the steel mills. Traditionally, the dispatching and operation of railway vehicles were controlled by the locomotive operator and the controller in the control room.

[0029] However, the plant is littered with numerous small transport vehicles, workers, and cargo, and rail vehicles directly enter the plant, limiting the ability to close the tracks. Therefore, access restrictions, like on regular railways, are difficult to enforce, and the risk of accidents remains constant.

[0030] Furthermore, controllers lack detailed information about the surrounding areas where actual trains operate, and locomotive engineers face challenges in quickly recognizing and addressing potential accident risks due to the nature of open tracks. Furthermore, both controllers and locomotive engineers face the potential for human error.

[0031] In the event of an accident such as a collision with a railway vehicle carrying hundreds of tons of coal, there is a possibility of a large-scale accident occurring due to spillage of cargo.

[0032] In order to solve the problem in this situation, this disclosure aims to provide a system that quickly recognizes obstacles and dangerous situations in the surroundings when a railway vehicle is running autonomously by linking a control server and a locomotive.

[0033] In particular, a customized autonomous driving system that takes into account open track conditions, the presence of multiple moving obstacles, and the characteristics of the load is required. For each operation described below, the results of a known artificial intelligence model can be utilized through a learning process. There are no restrictions on the AI ​​model, and the AI ​​model can be used for detailed operations such as image integration parameter generation, surrounding obstacle detection and classification, obstacle collision risk assessment, precise location assessment, and track environment assessment. In addition, an AI model optimized for each situation can be used to derive autonomous driving control values ​​and obstacle recognition values. The AI ​​model can be trained using specialized factors for each situation as learning data, and can be installed on the control server or the railway vehicle depending on the configuration location of the autonomous driving control device or obstacle detection device of the railway vehicle.

[0034]

[0035] The autonomous railway vehicle technology to which the present disclosure can be applied is a technology that successfully executes operation instructions given by the control center through multi-sensor-based recognition and judgment, thereby automatically moving the railway vehicle to the destination, instead of the repetitive task of the locomotive driver recognizing the driving environment and performing operations through railway control instructions.

[0036] Meanwhile, autonomous trains operate on fixed tracks, requiring control technology in a relatively limited environment compared to cars, which can freely change lanes. However, when a train is equipped with cargo and passenger cars, the weight of the vehicle increases braking distances. Therefore, high accuracy and optimized computational speed are essential for track recognition and risk assessment.

[0037] Various sensors, such as cameras, lidar, and radar, are used for obstacle detection and recognition. However, for omnidirectional sensing, including long-distance sensing, sensors other than cameras suffer from significant cost and computational overhead. Furthermore, because lidar and radar primarily acquire distance information, they are unable to address recognition issues such as traffic signal recognition, sign recognition, and object recognition.

[0038] The rapid advancement of deep learning-based obstacle detection algorithms using cameras has enabled autonomous driving systems to assess hazardous situations using a variety of technologies, including camera-based pedestrian and vehicle detection, traffic light and track recognition, and driving environment segmentation. Furthermore, the accuracy and speed of single-image processing must be improved to a level suitable for practical use.

[0039]

[0040] In the aforementioned context, the present embodiments aim to provide a technology for ensuring the safety of autonomous railway vehicles through obstacle detection using only camera sensors. In particular, the present invention proposes a technology for installing multiple cameras with various settings to acquire image information in the direction of travel. Furthermore, the present invention proposes a detection device technology that integrates images received from multiple cameras based on the track into a single image, and accurately extracts the obstacle detection area necessary for assessing dangerous situations based on railway control and location information, thereby achieving high detection accuracy and high computational speed.

[0041] In this disclosure, a railway vehicle refers to a moving vehicle that runs on rails. For example, a railway vehicle may refer to a locomotive. Furthermore, the power source of a railway vehicle may be not only an internal combustion engine but also electric power, and there are no limitations regarding the power source or power transmission system.

[0042]

[0043] FIG. 1 is a schematic diagram illustrating a railway vehicle autonomous driving system according to one embodiment.

[0044] Referring to FIG. 1, one or more railway vehicles (110) can be connected to a control server (100) via a network. The control server (100) receives information from the railway vehicles (110) and performs a function of transmitting related control information to the railway vehicles (110). In addition, the control server (100) can perform a function of controlling the operation of the railway vehicles (110) by analyzing and generating information for operation control of the railway vehicles (110) and transmitting the information to the railway vehicles (110).

[0045] Railway vehicles (110) can transmit information detected from various sensors configured in the railway vehicles (110) while running on the tracks to the control server (100). In addition, the control server (100) can transmit various railway control information, such as work instruction information, route information, track control information, and TLC status information, to the railway vehicles (110).

[0046] Railway vehicles (110) can perform driving preparations using railway control information and perform autonomous driving to perform tasks set according to work instruction information.

[0047] Information exchange between railway vehicles (110) and the control server (100) can be performed at preset intervals or when a specific event occurs, and when an emergency situation occurs in the railway vehicles (110), relevant information can be immediately transmitted to the control server (100).

[0048] Wireless communication technologies can be applied to networks. Examples include mobile communication technologies, short-range communication technologies like WiFi and Bluetooth, and beacon technology. Two or more technologies can also be used in conjunction.

[0049] For example, the control server (100) may transmit downlink control information on a physical control channel to transmit information to railway vehicles (110). The communication device configured in the railway vehicles (110) may specify the frequency / time resources of the downlink data information including the railway control information based on the downlink control information and decode the corresponding frequency / time resources to obtain the railway control information. If the railway control information is normally decoded, the communication device configured in the railway vehicles (110) transmits an Ack message to the control server (100) through the uplink control channel or the uplink data channel. If the Ack message is not received, the control server (100) performs a procedure for retransmitting the railway control information.

[0050] Hereinafter, an obstacle detection device that can be configured in a railway vehicle (110) or a control server (100) in the aforementioned system will be described with reference to the drawings.

[0051]

[0052] FIG. 2 is a diagram illustrating a configuration of an obstacle detection device according to one embodiment.

[0053] Referring to FIG. 2, an obstacle detection device (200) using multiple camera sensors may include a receiving unit (210) that receives two or more image information generated at the same time from two or more camera sensors.

[0054] For example, the receiving unit (210) may receive captured images from two or more camera sensors configured in a railway vehicle. Alternatively, the receiving unit (210) may also receive video images from two or more camera sensors. The two or more camera sensors may have different FoVs, resolutions, installation locations, etc. Accordingly, the image information received by the receiving unit (210) may be images captured from different areas, and there may be overlapping areas between each piece of image information.

[0055] Additionally, the receiver (210) can map and receive images captured at the same time for two or more image information received from two or more camera sensors. For example, multiple camera sensors can generate image information at preset intervals or points in time. In this case, multiple camera sensors can generate image information by including mapping information indicating the generation point in time in each image information.

[0056] The receiving unit (210) can identify image information having the same creation time when receiving two or more pieces of image information. This is because image information having the same time must be created as a single integrated image.

[0057] The obstacle detection device (200) may include an image integration unit (220) that converts two or more image pieces of information into one integrated image.

[0058] For example, the image integration unit (220) can convert image information generated at the same time from two or more pieces of image information into a single integrated image. To this end, the image integration unit (220) can group images generated at the same time by checking the mapping information mapped to the two or more pieces of image information. Image information generated at the same time can be converted into a single integrated image.

[0059] For example, the image integration unit (220) can create an integrated image by matching feature points extracted from each of two or more pieces of image information based on the running track of a railway vehicle.

[0060] For example, the image integration unit (220) may extract primary feature points using a preset algorithm from each of two or more image information, delete feature points that do not correspond to the driving track from among the primary feature points, and then generate an integrated image using the remaining secondary feature points. For example, the image integration unit (220) may extract feature points by applying a feature point extraction algorithm to each of the image information acquired at the same time. Various preset algorithms for feature point extraction may be applied. For example, the image information may be converted into an integral image using a preset formula. The integral image refers to a form in which pixel brightness values ​​are accumulated. Thereafter, feature points may be detected from the integral image using a Hessian detector. The Hessian detector is a feature point detection algorithm based on a Hessian matrix and exhibits high accuracy. In addition, the Hessian detector is used when detecting a blob at a location where the determinant is the maximum value. When using the Hessian detector, the performance can be improved by using the FAST Hessian Detector. In addition, the image integration unit (220) can extract feature points from an image using various algorithms.

[0061] In addition, the image integration unit (220) can match secondary feature points generated from each of two or more image pieces, remove outliers using an outlier removal algorithm, and then generate an integrated image using a homography matrix. For example, secondary feature points are matched based on their mutual distances, and a transformation relationship (homography) between the images is calculated to project the images onto a single plane. At this time, an outlier removal algorithm such as the RANSAC algorithm can be used to establish an accurate transformation relationship from outliers that interfere with the prediction of model parameters.

[0062] Once the transformation relationship between images is calculated, the image integration unit (220) can rotate and transform different images using the image transformation relationship to create an integrated image.

[0063] As described above, the image integration unit (220) extracts feature points from each image, but only extracts and matches feature points in the track area that can be included in each image, and removes outliers to generate an integrated image. This enables high-speed integrated image generation even in situations where a large number of feature points are extracted.

[0064] Meanwhile, pre-trained AI algorithms can be used to select feature points in the track area. Alternatively, model parameters can be set to extract feature points in the track area based on the track characteristics and provided as parameters to each pre-set algorithm.

[0065] The obstacle detection device (200) may include an area of ​​interest setting unit (230) that sets an area of ​​interest in an integrated image using location information and control information of a railway vehicle.

[0066] For example, the region of interest setting unit (230) can set the region of interest by setting offset information based on the driving track included in the integrated image. The region of interest setting unit (230) can recognize the driving track portion on the integrated image and apply certain offset information to the left and right sides of the driving track to set the region of interest based on the driving track.

[0067] In addition, when multiple tracks are included in the integrated image, the region of interest setting unit (230) may select a running track using control information and location information, and may set a region of interest by applying offset information based on the selected running track. Here, the control information may include at least one of track information, destination information, dangerous area setting information, freight information, trainset information, and width information of the railway vehicle. In addition, the control information may include work instruction information for the railway vehicle, track information on which the railway vehicle must run, track control information on the corresponding track, and TLC status information of the corresponding railway vehicle. The region of interest setting unit (230) may use the control information to check information on the track on which the railway vehicle must run, and may use location information to check information on a track to be used among multiple tracks at the current location of the railway vehicle. When information on the track on which the railway vehicle will run is confirmed, the region of interest setting unit (230) may apply offset information based on the corresponding running track to set a region of interest based on the planned driving range.

[0068] For example, offset information may be set based on railway vehicle information, and may be set to increase or decrease based on control information and location information based on the standard offset information.

[0069] For example, offset information can be set based on the width of the TLC attached to the railway vehicle using the freight information, formation quantity information, and width information of the railway vehicle. The area of ​​interest setting unit (230) can set the offset information by storing the degree of increase or decrease in offset information for setting the area of ​​interest in a mapping table based on control information.

[0070] For another example, offset information can be determined based on the location where the train is operating. If there is a work area ahead of the train's track or a hazardous area, such as a level crossing, the offset information can be set to increase to detect obstacles in that hazardous area.

[0071] The obstacle detection device (200) may include an obstacle detection unit (240) that detects obstacles in an area of ​​interest.

[0072] For example, if an obstacle is detected within the region of interest, the obstacle detection unit (240) can determine a risk level based on at least one of obstacle information, location information, and control information. The obstacle detection unit (240) can detect obstacles using a preset deep learning-based object detection algorithm or an abnormal obstacle detection algorithm. The algorithm can be trained based on data from an environment within a business site where actual railway vehicles operate.

[0073] The obstacle detection unit (240) can not only detect the presence of an obstacle using the aforementioned algorithm, but can also determine whether the obstacle is a moving object or a fixed object. Alternatively, the type of obstacle, direction of movement, and speed can be calculated using an object classification algorithm.

[0074] The obstacle detection unit (240) can determine the risk level of an obstacle using the type, direction of movement, and speed information of the obstacle. For example, the obstacle detection unit (240) can set the risk level high when the obstacle is moving toward the railway vehicle, and can set the risk level low when the obstacle is moving away from the railway vehicle. Alternatively, the obstacle detection unit (240) can set the risk level of the obstacle low when the obstacle is a fixed object and there is no overlapping area considering the width of the running track and the railway vehicle, considering the size of the obstacle.

[0075] Alternatively, the obstacle detection unit (240) may set the risk level for the obstacle to be high if the location where the obstacle is detected is in a risk area according to the control information.

[0076] The risk level can be set in various ways, such as as a continuous number or grade. The obstacle detection unit (240) can issue a warning through a notification when an obstacle that poses a risk of collision with a railway vehicle exists, depending on the risk level, or can generate a control signal to control the movement of the railway vehicle.

[0077] Meanwhile, the obstacle detection device (200) may further include an error determination unit (250) that determines whether an error has occurred in the integrated image by using the location information of the railway vehicle, the integrated image, and a preset reference image.

[0078] For example, the error determination unit (250) can determine that an error has occurred in the integrated image conversion if the track location in the preset reference image and the track location derived from the integrated image are judged to have an error greater than the preset reference value by matching the location information of the railway vehicle.

[0079] A reference image can refer to images collected in advance based on the operation of railway vehicles on each track. Specifically, given the characteristics of railway vehicles, the similarity between the captured images can be very high when railway vehicles of the same size and shape operate on the track. This similarity can be utilized to store a preset reference image and compare it with the integrated image actually generated from each railway vehicle. By comparing the position of the track within the image, any errors in the integrated image can be determined. If an error is determined to have occurred, the parameters of the algorithm used by the railway vehicle to generate the integrated image can be modified and applied.

[0080] Alternatively, the reference image can be generated from multiple integrated images captured at the same location. As mentioned above, for images captured by identically shaped railway vehicles at the same location, the location of the running tracks must be identical or similar. Using this, the reference image can be used to determine the location of the running tracks by averaging them and comparing them with the integrated image to determine whether an error exceeds a threshold value.

[0081] In addition, the error determination unit (250) can determine whether an error occurred in the creation of an integrated image using various algorithms when the railway vehicle is in a stationary state and not in operation.

[0082] Through the above-described operation, the obstacle detection device for autonomous railway vehicles detects hazards in the direction of travel by creating a single image using only the camera, rather than integrating detection results from multiple cameras and other sensors. This allows the device to quickly produce detection results and reduce errors arising from integrating detection results from multiple, diverse sensors, enabling more accurate assessment of risk situations.

[0083] Furthermore, the obstacle detection device can prevent malfunctions caused by obstacles detected outside the direction of travel by integrating the track segmentation algorithm and control information to specify the detection area where obstacle detection is required. In addition, the obstacle detection device can utilize the characteristic that railway vehicles only run on the track to derive the track segmentation results in advance in a domain similar to the learning environment, and then use the pre-derived segmentation results based on the location. This enables accurate derivation of the detection area even in environments where the track segmentation results are not good, such as at night or in rainy weather, thereby ensuring safety.

[0084] In addition, the obstacle detection device compares the results of the integrated image and track detection calculated in advance while stopped or waiting for work with the values ​​calculated based on the current situation. If the error is large, it can identify various problems such as sensor contamination, change in sensing direction, and algorithm error, thereby ensuring the stability of the system.

[0085]

[0086] Below, the operation of each component described in FIG. 2 is described in more detail, applying various embodiments. Each embodiment described below can be performed in any combination or in its entirety by the aforementioned obstacle detection device. For ease of understanding, the following description focuses on the obstacle detection device, which may be configured in a railway vehicle as described above, or in a control server or railway vehicle control server.

[0087]

[0088] FIG. 3 is a diagram for explaining an operation of receiving two or more image information according to one embodiment.

[0089] Camera sensors can capture different images depending on their installation location, lens settings, and other factors. To capture long-distance information, the camera sensor's field of view (FoV) can be narrowed and mounted on the top of the train. To capture close-range information, the camera sensor's field of view can be widened and mounted on the middle or bottom of the train.

[0090] For example, four or more camera sensors can be installed to detect the front, rear, and sides of a railway vehicle. Furthermore, since autonomous driving must successfully reach its destination based on the current position of the railway vehicle upon receiving instructions from the control server, GPS or lidar sensors capable of acquiring and predicting location information may be attached to the upper portion of the railway vehicle.

[0091] Referring to FIG. 3, four images can be acquired at the same time depending on the FoV and installation location set for each camera sensor. For example, a first image (310), which is a long-distance shooting image, and a second image (320), which is a short-distance shooting image, can be acquired. In addition, a third image (330) and a fourth image (340), which are configured on the left and right sides of the railway vehicle (300) and generate side images in the direction of movement, can be acquired. Images cannot be acquired for the location of the railway vehicle (300).

[0092] When the first image (310), the second image (320), the third image (330), and the fourth image (340) are acquired in this way, they can be mapped as images at the same time and generated as an integrated image through an image integration unit. Each image may have overlapping and non-overlapping areas. However, in the case of a driving track, it may be included in all images. In addition, each camera sensor may be configured so that all images of the driving track are included. At this time, there may be parts where the track branches or bends differently from FIG. 3, such as track switching or curved sections, but parameters for image integration can be configured in advance and used for image integration.

[0093] FIG. 4 is a diagram for explaining an integrated image generation operation according to one embodiment.

[0094] Referring to FIG. 4, the image integration unit can convert two or more image information into one integrated image.

[0095] For example, the image integration unit can acquire multiple images for the same time period (S400). As described above, the camera sensor generates images from various angles and positions, and the images generated at the same time can be mapped and recognized by the image integration unit. The image integration unit can identify, classify, and acquire images of the same time period to integrate multiple images generated at the same time into a single image.

[0096] The image integration unit can select and match feature points for each of two or more images (S410). Predefined algorithms can be used to represent and compare image regions and similarity invariants, and various fast and robust algorithms can be used. For example, various feature extraction algorithms such as SURF and SIFT can be used.

[0097] For example, the image integration unit can extract primary features using a preset algorithm and generate secondary features by selecting only features related to the driving track.

[0098] For example, the image integration unit can detect blob features using a Hessian Detector and generate descriptors around the features. This enables feature extraction that is robust to image blur, rotation, and scale.

[0099] The image integration unit generates an integral image based on image information. An integral image is a form in which pixel brightness values ​​are accumulated.

[0100] Once the integral image is generated, the Hessian matrix is ​​used to extract interest points from the image. The Hessian matrix is ​​used to determine whether a variable function has an extremum value, whether it is a maximum or minimum. For example, when a real function is given as f(x1, x2, x3,..., xn), the Hessian matrix is ​​expressed as follows.

[0101]

[0102] If the determinant of the Hessian matrix is ​​found and the value of the determinant is positive and both eigenvalues ​​are negative or positive, it can be judged as a point of interest.

[0103] The image integration unit uses determinant data found with the Hayesian matrix to detect feature points for each filter size. In other words, feature points can be extracted by scaling the box filter area. This has the advantage of eliminating aliasing. Alternatively, feature points can also be extracted by scaling the image.

[0104] To explain feature extraction in detail, Hessian Determinant data is obtained for each pixel and compared to a threshold. If the value is greater than the threshold, the Hessian Determinants of the eight adjacent pixels are compared to see if the value is larger. If the value is large, the nine Hessian Determinants, each in a 3x3 size corresponding to the upper and lower box filter sizes, are compared, and the largest one is detected as a feature point.

[0105] When a feature point is detected, the Haar Wavelet Response dx and dy are obtained in the x and y directions for pixels within a circle with a radius of 6s, using the scale information s at which the feature point is detected centered on the feature point. Afterwards, the window is rotated at 5-degree intervals, and the magnitudes of the vectors within the 60-degree window are added to determine the direction with the largest magnitude as the main direction of the feature point.

[0106] Once the dominant direction is determined, a descriptor is generated. The descriptor represents the brightness change of neighboring pixels within a certain area around the feature point. Using the dominant direction and scale information s, a window of size 20s is created centered on the feature point, and the created window is divided into 4x4 areas, thereby dividing it into 16 areas. Each area is again divided into 5x5 areas, and each pixel is filtered using a Haar wavelet filter to calculate the sum of dx, the sum of the absolute value of dx, the sum of the absolute value of dy, and the sum of the absolute value of dy. Through this, 4 descriptors are created for each of the 16 areas, for a total of 64 descriptors.

[0107] Except for the feature points in the driving track area, the remaining feature points are removed from the generated feature points. The feature points in the driving track area are matched between images.

[0108] The image integration unit can perform an outlier removal operation to remove outliers (S420).

[0109] The image integration unit removes outliers from the feature matching results. For example, the image integration unit can remove outliers from the dataset using an algorithm that removes noise and predicts the model. This is because removing outliers is necessary to obtain the homography matrix, which allows different images to be integrated into a single image.

[0110] For example, the RANdom SAmple Consensus algorithm can be used as an outlier removal algorithm. The outlier removal algorithm selects the model with the largest consensus, i.e., the model supported by the largest number of data points. To achieve this, the outlier removal algorithm randomly selects several sample data points and derives model parameters that satisfy these sample data points. It then counts the number of data points close to the derived model, and if the number is large, it memorizes the model. This process is repeated N times, and the model with the largest number of supported data points is returned as the final result.

[0111] For example, feature point pairs (matched feature points) are randomly extracted from image information. Homography is then calculated using the data. By multiplying each feature point by the calculated homography matrix, the location of the feature point in another image can be predicted. The difference (e.g., residual) between the expected location and the actual feature point match location is calculated. At this time, the difference value for each feature point location is set as an error value, and the sum of the squares of each error value is calculated to generate a model value. The above random selection operation is repeated a preset number of times or until a model value below a threshold value is generated. If a low model value is generated, the model is set as the optimal model and saved.

[0112] Afterwards, the image integration unit can calculate the image transformation relationship (S430). For example, the image integration unit can generate an integrated image using a homography matrix. For example, the homography matrix is ​​calculated to calculate a transform value, and the corresponding transform value is applied to each image to generate an integrated image. Alternatively, the transform parameters can be calculated and stored in advance, and the corresponding parameters can be used.

[0113] FIG. 5 is a diagram for explaining an operation of setting a region of interest on an integrated image according to one embodiment.

[0114] Referring to FIG. 5, a plurality of images can be integrated to generate an integrated image (500). In the integrated image, a portion (530) not captured by the camera sensor does not appear.

[0115] The region of interest setting unit can set a region of interest (510) by applying offset information based on the driving track in the integrated image (500). Zero masking is performed on regions (520) other than the region of interest (510) to prevent obstacle detection. This prevents waste of time and computational power for unnecessary obstacle detection and analysis.

[0116] Here, the offset can be set based on the left and right outer boundaries of the running track. For example, the offset information can be set based on railway vehicle information. In other words, the reference offset information can be set based on the width of the railway vehicle.

[0117] Alternatively, offset information may be set to increase or decrease based on control and location information from the reference offset information. For example, if a railway vehicle is carrying a TLC with a width greater than that associated with the reference offset, offset information may be set to increase from the reference offset based on the control information to reflect the width information of the corresponding TLC.

[0118] Alternatively, when a railway vehicle is located in a section designated as a danger zone, offset information related to the danger zone may be set to increase from the reference offset, taking control information into account. Offset information may be set differently for the left and right sides.

[0119] FIG. 6 is a drawing for explaining an operation of setting a region of interest on an integrated image according to another embodiment.

[0120] Referring to Fig. 6, this illustrates a case where offset information is set differently on the left and right, excluding the uncaptured area (630) in the integrated image (600). The area of ​​interest (610) is set by applying offset 1 and offset 2, and the area of ​​no interest (620) is subject to zero masking.

[0121] For example, if work is in progress on the left front of a track, the area may be designated as a danger zone in the control information. The area of ​​interest setting unit verifies the danger zone information in the control information and, if it determines that the forward portion of the integrated image contains a danger zone using the position information of the railway vehicle, sets the obstacle detection area by increasing Offset 1 for that portion to be greater than Offset 2.

[0122] If a train passes through a hazardous area, the reference offset can be reapplied. This allows for dynamic control of obstacle detection in hazardous areas, such as work areas or crossings, while also applying the obstacle detection algorithm only to certain areas of interest, resulting in faster computational speeds.

[0123] Once the region of interest is established, the obstacle detection unit can detect obstacles within the region of interest. Obstacle detection can be performed using various deep learning algorithms capable of camera image-based obstacle recognition, such as object detection, anomaly detection, and scene classification, using a masked integrated image.

[0124] When an obstacle is detected within the region of interest, the obstacle detection unit can determine a risk level for the obstacle based on at least one of obstacle information, location information, and control information. The obstacle information may include information on whether the obstacle is moving, the type of obstacle, the direction of movement of the obstacle, and the speed of the obstacle. The location information may include information on the location of the railway vehicle, and the control information may include information on the width of the railway vehicle, TLC information, information on the running track, and weight information including TLC.

[0125] The obstacle detection unit can determine the risk of collision using the location information, route information, TLC status information, and driving environment information (e.g., obstacle information) of the railway vehicle, and can determine the risk level based on the braking distance of the railway vehicle.

[0126] For example, an obstacle detection unit can determine an occupied area of ​​a railway vehicle based on track information and TLC status information within a running track, and monitor the presence and movement of obstacles within a surveillance area set based on the occupied area.

[0127] For example, when a railway vehicle is in operation, the width of the railway vehicle can be determined based on the track information within the railway vehicle's track. Furthermore, when the railway vehicle operates by connecting TLCs, the width of the TLCs needs to be considered. Therefore, the obstacle detection unit primarily determines the occupied area occupied by the railway vehicle on the track using track information and TLC status information. Once the occupied area is determined, a monitoring area can be determined by applying a certain offset based on the occupied area. The monitoring area can be set to be wider than the occupied area, and depending on the direction of operation of the railway vehicle, the front can be set to be wider than the rear.

[0128] The obstacle detection unit can analyze the type and movement path of an obstacle when detecting it. Obstacle types can be identified using an AI classification model based on sensor data. Obstacle movement paths can be analyzed using time-series movement and type information about the obstacle.

[0129] The obstacle detection unit can determine whether there is a risk of collision between an obstacle and the occupied area of ​​a railway vehicle. For example, a collision risk can be determined if there is a possibility that an obstacle could enter the occupied area. For example, the obstacle detection unit can calculate the collision risk by calculating the Time to Collision (TTC) between the target obstacle and the occupied area.

[0130] The obstacle detection unit continues to perform obstacle detection operations when it determines that there is no risk of collision. If the obstacle detection unit determines that there is a risk of collision, it can determine the collision risk level.

[0131] For example, the obstacle detection unit may calculate a collision risk level based on estimated braking distance information calculated based on at least one of TLC number information, TLC capacity information, and TLC type information for the speed information of the railway vehicle, and information on the expected collision distance with the obstacle. Alternatively, the collision risk level may be determined through the TTC calculation described above.

[0132] For example, the total weight of a railway vehicle can be determined based on the number of TLCs connected to the railway vehicle, the amount of coal loaded in the TLCs, and the type of TLCs. Therefore, the estimated braking distance can be affected by the corresponding weight. Furthermore, the estimated braking distance may vary depending on the characteristics of the coal. Considering these points, the obstacle detection unit calculates the estimated braking distance and compares the expected collision distance with an obstacle that poses a collision risk to determine the collision risk level. Depending on the collision risk level, deceleration driving, emergency braking, and normal driving operations can be performed separately.

[0133] This example illustrates a method for setting a collision risk level based on a comparison of the estimated braking distance and the expected collision distance. Each value described below is exemplary and subject to change. Furthermore, when the collision risk level is determined by the TTC, the collision risk level can be determined based on the TTC value and the presence or absence of a preset TTC value for each section.

[0134] For example, the obstacle detection unit can control the vehicle to immediately initiate emergency braking by setting the maximum collision risk level when the estimated braking distance is greater than the expected collision distance (the available driving distance before collision). In this case, emergency braking is performed to minimize the impact, as collision avoidance is difficult.

[0135] As another example, the obstacle detection unit can control the vehicle to perform deceleration driving by setting the collision risk level to an intermediate level when the expected collision distance (driving distance before collision) is greater than the estimated braking distance and less than the estimated braking distance + X m. Here, X can be set in advance. In this case, the collision can be avoided if emergency braking is performed immediately, but since there is a possibility that the collision risk can be resolved, such as when the obstacle stops, deceleration driving is initiated and the collision risk level is continuously re-evaluated.

[0136] As another example, if the expected collision distance (driving distance before collision) is greater than the estimated braking distance + X m, the obstacle detection unit sets the collision risk level low and drives normally at the target speed. However, even in this case, the collision risk may still exist, so the collision risk judgment cycle is reduced to perform a process of more frequently re-evaluating the collision risk level for the target obstacle.

[0137] Meanwhile, the obstacle detection unit may consider additional factors affecting braking distance, such as weather and temperature, when calculating the estimated braking distance. The formula for calculating the braking distance, which increases or decreases depending on weather and temperature on the track, can be used in a publicly available format.

[0138] Additionally, the collision risk level assessment cycle can be set to vary based on TLC status information. For example, if multiple TLCs are connected to a railway vehicle and the amount of fuel injected into the TLCs exceeds a certain level, the estimated braking distance may increase, and thus the collision risk level assessment cycle can be set to shorten.

[0139] Meanwhile, the obstacle detection device may further include an error determination unit that determines whether an error has occurred in the integrated image using the railway vehicle's location information, the integrated image, and a preset reference image. For example, the error determination unit may determine that an error has occurred in the integrated image conversion if the track location in the preset reference image, which matches the railway vehicle's location information, and the track location derived from the integrated image are determined to have an error greater than a preset reference value.

[0140] If an error is determined to have occurred, the obstacle detection device can change the parameters of each algorithm used to generate the integrated image. For example, the error can be corrected by changing the transform parameter values ​​used to generate the integrated image.

[0141] Through the above operations, the obstacle detection device can quickly generate an integrated image while the railway vehicle is in operation and quickly detect obstacles within the region of interest by removing unnecessary elements through zero masking. This can provide fast computational speeds and stable autonomous railway vehicle operation.

[0142] Below, we re-examine the obstacle detection method for the aforementioned operations. While some details may be omitted to avoid unnecessary redundancy, all operations included in this disclosure can be performed using the methods described below.

[0143] FIG. 7 is a drawing for explaining an obstacle detection method according to one embodiment.

[0144] Referring to FIG. 7, an obstacle detection method using multiple camera sensors may include a receiving step (S700) of receiving two or more image information generated at the same time from two or more camera sensors.

[0145] For example, the receiving stage may receive captured images from two or more camera sensors configured on a railway vehicle. Alternatively, the receiving stage may receive video images from two or more camera sensors. The two or more camera sensors may have different fields of view (FoV), resolutions, installation locations, etc. Accordingly, the image information received by the receiving stage may be images captured from different areas, and there may be overlapping areas between the image information.

[0146] Additionally, the receiving step can map and receive images captured at the same time for two or more image information received from two or more camera sensors. For example, multiple camera sensors can generate image information at preset intervals or points in time. In this case, the multiple camera sensors can generate each image information by including mapping information indicating the generation point in time.

[0147] The obstacle detection method may include an image integration step of converting two or more image information into one integrated image (S710).

[0148] For example, the image integration step can convert image information generated at the same time from two or more image information into a single integrated image. To achieve this, the image integration step can identify the mapping information mapped to the two or more image information and group the images generated at the same time. Image information generated at the same time can be converted into a single integrated image.

[0149] For example, the image integration step can create an integrated image by matching feature points extracted from each of two or more image pieces based on the running track of a railway vehicle.

[0150] For example, the image integration step can extract primary feature points from two or more image information using a preset algorithm, delete feature points that do not correspond to the driving track among the primary feature points, and then use the remaining secondary feature points to create an integrated image. For example, the image integration step can extract feature points by applying a feature extraction algorithm to each image information acquired at the same time. Various preset algorithms can be applied for feature point extraction. For example, the image information can be converted into an integral image using a preset formula. An integral image refers to a form in which pixel brightness values ​​are accumulated. Afterwards, a Hessian detector can be used to detect feature points from the integral image. When using a Hessian detector, performance can be improved by using the FAST Hessian Detector. In addition, the image integration step can extract feature points from the image using various algorithms.

[0151] Additionally, the image integration step can match secondary features generated from each of two or more images, remove outliers using an outlier removal algorithm, and then generate an integrated image using a homography matrix. For example, secondary features are matched based on their mutual distances, and a transformation relationship (homography) between the images is calculated to project them onto a single plane. At this time, an outlier removal algorithm such as the RANSAC algorithm can be used to establish an accurate transformation relationship from outliers that interfere with the prediction of model parameters.

[0152] Once the transformation relationship between images is calculated, the image integration step can rotate and transform different images using the image transformation relationship to create an integrated image.

[0153] As described above, the image integration step extracts feature points from each image. However, only those feature points within the track region that can be included in each image are extracted and matched, and outliers are removed to create a unified image. This enables high-speed unified image generation even when a large number of feature points are extracted.

[0154] Meanwhile, pre-trained AI algorithms can be used to select feature points in the track area. Alternatively, model parameters can be set to extract feature points in the track area based on the track characteristics and provided as parameters to each pre-set algorithm.

[0155] The obstacle detection method may include an area of ​​interest setting step of setting an area of ​​interest in an integrated image using location information and control information of a railway vehicle (S720).

[0156] For example, the region of interest setting step can set the region of interest by setting offset information based on the driving track included in the integrated image. The region of interest setting step can recognize the driving track portion in the integrated image and apply certain offset information to the left and right sides of the driving track to set the region of interest based on the driving track.

[0157] In addition, in the region of interest setting step, when multiple tracks are included in the integrated image, the driving track can be selected using control information and location information, and the region of interest can be set by applying offset information based on the selected driving track. The region of interest setting step can use control information to confirm information on the track on which the railway vehicle must drive, and can use location information to confirm information on which of multiple tracks to use from the current location of the railway vehicle. In the region of interest setting step, once the track information on which the railway vehicle will drive is confirmed, the region of interest can be set based on the planned driving range by applying offset information based on the corresponding driving track.

[0158] For example, offset information may be set based on railway vehicle information, and may be set to increase or decrease based on control information and location information based on the standard offset information.

[0159] For example, offset information can be set based on the width of the TLC attached to the railway vehicle, using freight information, formation quantity information, and width information of the railway vehicle. The region of interest setting step can set the offset information by storing the degree of increase or decrease in offset information for setting the region of interest in a mapping table based on control information.

[0160] For another example, offset information can be determined based on the location where the train is operating. If there is a work area ahead of the train's track or a hazardous area, such as a level crossing, the offset information can be set to increase to detect obstacles in that hazardous area.

[0161] The obstacle detection method may include an obstacle detection step of detecting an obstacle in an area of ​​interest (S730).

[0162] For example, the obstacle detection step can determine the risk level of an obstacle based on at least one of obstacle information, location information, and control information when an obstacle is detected within the region of interest. The obstacle detection step can detect obstacles using a preset deep learning-based object detection algorithm or an abnormal obstacle detection algorithm. These algorithms can be trained based on data from actual environments within a business site where railway vehicles operate.

[0163] The obstacle detection step not only detects the presence of an obstacle using the aforementioned algorithm, but can also determine whether the obstacle is a moving or stationary object. Alternatively, an object classification algorithm can be used to determine the type of obstacle, its direction of movement, and its speed.

[0164] The obstacle detection step can determine the risk level of an obstacle using information about the obstacle's type, direction of movement, and speed. For example, the obstacle detection step can set a high risk level if the obstacle is moving toward the train, and a low risk level if it is moving away from the train. Alternatively, the obstacle detection step can consider the size of the obstacle, and if the obstacle is a fixed object and there is no overlapping area between the track and the train, the risk level can be set low.

[0165] Alternatively, the obstacle detection step may set the risk level for the obstacle to a high level if the location where the obstacle is detected is in a risk area according to the control information.

[0166] Risk levels can be set in a variety of ways, including as continuous numbers or grades. The obstacle detection stage can issue a warning or generate a control signal to control the behavior of a railway vehicle when an obstacle with a risk of collision exists, depending on the risk level.

[0167] In addition, the obstacle detection method may include an error determination step for determining whether an error has occurred in the integrated image by using the location information of the railway vehicle, the integrated image, and a preset reference image (S740).

[0168] For example, the error determination step can determine that an error has occurred in the integrated image conversion if the track location in the preset reference image and the track location derived within the integrated image are judged to have an error greater than the preset reference value by matching the location information of the railway vehicle.

[0169] A reference image can refer to images collected in advance based on the operation of railway vehicles on each track. Specifically, given the characteristics of railway vehicles, the similarity between the captured images can be very high when railway vehicles of the same size and shape operate on the track. This similarity can be utilized to store a preset reference image and compare it with the integrated image actually generated from each railway vehicle. By comparing the position of the track within the image, any errors in the integrated image can be determined. If an error is determined to have occurred, the parameters of the algorithm used by the railway vehicle to generate the integrated image can be modified and applied.

[0170] Alternatively, the reference image can be generated from multiple integrated images captured at the same location. As mentioned above, for images captured by identically shaped railway vehicles at the same location, the location of the running tracks must be identical or similar. Using this, the reference image can be used to determine the location of the running tracks by averaging them and comparing them with the integrated image to determine whether an error exceeds a threshold value.

[0171] In addition, the error determination step can use various algorithms to determine whether an error occurred in the creation of the integrated image when the railway vehicle is in a stationary state and not in operation.

[0172] Each of the aforementioned steps may perform some or all of the operations described with reference to FIGS. 1 through 6, and the execution steps may involve two or more steps being performed simultaneously or in a partially changed order. Furthermore, specific steps may be combined into a single step, or specific steps may be separated into two or more steps.

[0173] FIG. 8 is a drawing for explaining an obstacle detection method according to another embodiment.

[0174] Referring to Figure 8, track areas can be pre-defined in the integrated image to achieve faster computational speeds. This is possible because pre-learned track areas can be applied, taking into account the characteristics of railway vehicles operating on fixed tracks. The information described in Figure 7 is given the same identification number and is briefly described below.

[0175] At step S700, two or more images captured at the same time can be acquired. At step S710, an integrated image can be generated using the image integration scheme described above. In this case, a railway vehicle's travel path can be generated based on the railway vehicle's positioning information and control information (S805). Once the travel path is generated, the travel path information can be used to generate the integrated image.

[0176] If the track area is pre-defined within the integrated image (S810 YES), the obstacle detection method determines the track area using the pre-defined parameters. The track area can be pre-defined based on images captured at each location. Alternatively, if the track area is not pre-defined (S810 NO), the track area can be extracted from the integrated image using a track area extraction algorithm (S820).

[0177] Once the track area is set within the integrated image, a region of interest (ROI) is set based on the track area, and zero-masking is performed on the remaining area (S720). Subsequently, obstacle detection is performed within the ROI (S730). Route information can be applied during the ROI setting operation. For example, if the integrated image includes points where multiple tracks branch off, the route corresponding to the route along which the railway vehicle will travel can be determined, and the ROI can be set based on the corresponding route.

[0178] Through the aforementioned actions, rapid and accurate obstacle detection can be performed by railway vehicles. This enables safe autonomous railway vehicle operation.

[0179] The above description is merely an illustrative example of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations can be made without departing from the essential characteristics of the technical idea of ​​the present disclosure. In addition, the present embodiments are not intended to limit the technical idea of ​​the present disclosure but rather to explain it, and therefore the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included within the scope of the rights of the present disclosure.

[0180]

[0181] CROSS-REFERENCE TO RELATED APPLICATION

[0182] This patent application claims priority under 35 USC § 119(a) to Korean Patent Application No. 10-2023-0182619, filed December 15, 2023, the entire contents of which are incorporated herein by reference. Furthermore, this patent application claims priority in countries other than the United States for the same reasons, the entire contents of which are incorporated herein by reference.

Claims

1. In an obstacle detection device using a multi-camera sensor, A receiving unit that receives two or more image information generated at the same time from two or more camera sensors; An image integration unit that converts two or more image pieces of information into one integrated image; An area of ​​interest setting unit that sets an area of ​​interest in the integrated image using the location information and control information of the railway vehicle; and An obstacle detection device including an obstacle detection unit that detects an obstacle in the above region of interest.

2. In paragraph 1, The above image integration part is, An obstacle detection device that creates the integrated image by matching feature points extracted from each of the two or more image information based on the running track of the railway vehicle.

3. In paragraph 2, The above image integration part is, An obstacle detection device that extracts primary feature points using a preset algorithm from each of the two or more pieces of image information, deletes feature points that do not correspond to the driving track among the primary feature points, and then creates the integrated image using the remaining secondary feature points.

4. In paragraph 3, The above image integration part is, An obstacle detection device that matches the secondary feature points generated from each of the two or more image pieces of information, removes outliers using an outlier removal algorithm, and then generates the integrated image using a homography matrix.

5. In paragraph 1, The above area of ​​interest setting section is, An obstacle detection device that sets the region of interest by setting offset information based on the driving track included in the above integrated image.

6. In paragraph 5, The above area of ​​interest setting section is, An obstacle detection device that selects the driving track using the control information and the location information when the integrated image includes multiple tracks, and sets the region of interest by applying offset information based on the selected driving track.

7. In paragraph 6, The above control information is, An obstacle detection device including at least one of the railway vehicle's route information, destination information, danger zone setting information, freight information, formation quantity information, and width information.

8. In paragraph 6, The above offset information is, An obstacle detection device in which reference offset information is set based on railway vehicle information, and is set to increase or decrease according to the control information and the location information based on the reference offset information.

9. In paragraph 1, The above obstacle detection unit, An obstacle detection device that determines a risk level according to the obstacle based on at least one of the obstacle information, the location information, and the control information when the obstacle is detected within the area of ​​interest.

10. In paragraph 1, An obstacle detection device further comprising an error determination unit that determines whether an error has occurred in the integrated image using the location information of the railway vehicle, the integrated image, and a preset reference image.

11. In Article 10, The above error determination unit is, An obstacle detection device that determines that an error has occurred in the integrated image conversion when the track location in the preset reference image and the track location derived from the integrated image are judged to have an error greater than a preset reference value by matching the location information of the above railway vehicle.

12. In a method for detecting obstacles using multiple camera sensors, A receiving step for receiving two or more image information generated at the same time from two or more camera sensors; An image integration step for converting two or more image pieces of information into one integrated image; A region of interest setting step for setting a region of interest in the integrated image using the location information and control information of the railway vehicle; and An obstacle detection method comprising an obstacle detection step of detecting an obstacle in the above region of interest.

13. In paragraph 12, The above image integration part is, An obstacle detection method for generating an integrated image by matching feature points extracted from each of two or more image information based on the running track of the railway vehicle.

14. In paragraph 13, The above image integration step is, An obstacle detection method comprising: extracting primary feature points using a preset algorithm from each of the two or more pieces of image information; deleting feature points that do not correspond to the driving track among the primary feature points; and then generating the integrated image using the remaining secondary feature points.

15. In paragraph 12, The above area of ​​interest setting step is, An obstacle detection method for setting the region of interest by setting offset information based on the driving track included in the above integrated image.

16. In paragraph 15, The above area of ​​interest setting step is, An obstacle detection method for selecting a driving track using the control information and the location information when the integrated image includes a plurality of tracks, and setting the region of interest by applying offset information based on the selected driving track.

17. In paragraph 16, The above offset information is, An obstacle detection method in which reference offset information is set based on railway vehicle information, and the reference offset information is set to increase or decrease based on the control information and the location information.

18. In paragraph 12, An obstacle detection method further comprising an error determination step of determining whether an error has occurred in the integrated image using the position information of the railway vehicle, the integrated image, and a preset reference image.

19. In paragraph 18, The above error determination step is, An obstacle detection method for determining that an error has occurred in the integrated image conversion when the track location in the preset reference image and the track location derived from the integrated image are judged to have an error greater than a preset reference value by matching the location information of the above railway vehicle.

Images