Railway detection model learning device, railway detection model learning method, and railway detection device

Abstract

To achieve highly accurate learning and construction of a railway detection model dedicated to a low-illuminance image, capable of performing accurate railway detection even when using a captured image under a low-illuminance environment.SOLUTION: A railway detection model learning device includes: a low-illuminance image selection unit that selects a low-illuminance image for learning from image correspondence information in which each of a plurality of high-illuminance images obtained by imaging a periphery of a railway vehicle traveling under a high-illuminance situation by an imaging device is associated with one of a plurality of low-illuminance images; a high-illuminance image selection unit that selects the high-illuminance image corresponding to the low-illuminance image for learning from the image correspondence information; a railway detection unit that inputs the selected high-illuminance image to a high-illuminance image railway detection model to detect a railway region in the high-illuminance image; a teacher data generation unit that generates information on the detected railway region in the high-illuminance image as teacher data; and a low-illuminance image railway detection model construction unit that learns and constructs a low-illuminance image railway detection model from the low-illuminance image for learning and the teacher data using a region division algorithm for the low-illuminance image for learning.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to a track detection model learning device, a track detection model learning method, and a track detection device.

Background Art

[0002] In order to realize the automatic operation of railway vehicles, the surroundings such as in front of the running railway vehicle are monitored, and the track area, that is, the area of the left and right rails, is detected in real time from the captured image captured in a low-illumination environment with low illuminance such as during bad weather in the daytime or at night. In the captured image in such a low-illumination environment, since the rail area of the track is unclear, it is difficult to extract the edge of the rail by image processing technology. For this reason, it is necessary to construct a track detection model dedicated to low-illumination images using a region segmentation algorithm of deep learning or the like.

[0003] In order to construct a track detection model dedicated to low-illumination images, it is necessary to train the track detection model by teaching the track area as correct data by human eyes for the training images taken in a low-illumination environment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, even when viewing the captured image in a low-illumination environment with the human eye, it is difficult to accurately indicate the track area, resulting in incomplete instructions, and thus making it difficult to learn and construct a track detection model dedicated to low-illumination images.

[0006] An object of the embodiment is to provide a track detection model learning device, a track detection model learning method, and a track detection device that can realize high-precision learning and construction of a track detection model dedicated to low-illumination images capable of accurately performing track detection even using captured images in a low-illumination environment.

Means for Solving the Problem

[0007] The track detection model learning device of the embodiment includes: each of a plurality of high-illumination images captured by an imaging device around a railway vehicle running in a high-illumination situation higher than a predetermined illuminance, and each of a plurality of low-illumination images captured by the imaging device at the same location as each imaging location of the plurality of high-illumination images around the railway vehicle running in a low-illumination situation lower than the predetermined illuminance; a storage unit that stores image correspondence information associating them; a low-illumination image selection unit that selects a low-illumination image for learning from the image correspondence information; a high-illumination image selection unit that selects the high-illumination image corresponding to the low-illumination image for learning from the image correspondence information; a track detection unit that inputs the selected high-illumination image into a track detection model for high-illumination images, which is a learned model that inputs the high-illumination image and outputs a track area in the high-illumination image, to detect the track area in the high-illumination image; a teacher data generation unit that generates information on the track area in the detected high-illumination image as teacher data; and a low-illumination image track detection model construction unit that learns and constructs a track detection model for low-illumination images, which is a learned model that inputs the low-illumination image and outputs a track area in the low-illumination image, using a region segmentation algorithm for the low-illumination image from the low-illumination image for learning and the teacher data.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Mode for Carrying Out the Invention

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0010] (Embodiment) FIG. 1 is a schematic configuration block diagram of the track detection model learning system according to the embodiment. The track detection model learning system 10 according to the present embodiment is mounted on the railway vehicle 1, and mainly includes an antenna 300, a positioning device 310, a camera 400, a track detection model learning device 100, an inertial sensor 600, a storage device 500, a display device 700, and a track detection device 200. Here, the railway vehicle may be referred to as a vehicle. Also, in the present embodiment, the vehicle is configured as a single - car formation, but it is not limited to this, and a formation of two or more cars may be used.

[0011] The antenna 300 is installed near the center in the width direction of the railway vehicle 1 and receives a radio wave signal transmitted from a positioning artificial satellite. The radio wave signal is a signal containing information for measuring the position of the railway vehicle 1.

[0012] The positioning device 310 is mounted on the railway vehicle 1 and can measure the position of the railway vehicle 1 based on the radio wave signal from the artificial satellite received by the antenna 300. The position of the railway vehicle 1 is the position information near the center of the left and right rails of the track. The positioning device 310 constitutes, for example, a GNSS (Global Navigation Satellite System), performs positioning (satellite positioning) of the railway vehicle 1 from the radio wave received by the antenna 300, and outputs the positioning information as the measurement result to the track detection model learning device 100 and the track detection device 200.

[0013] Here, GNSS is a general term for satellite positioning systems such as GPS in the United States, Quasi-Zenith Satellite System (QZSS) in Japan, GLONASS (GLO) in Russia, Galileo (GAL) in the European Union, and Beidou (BDS) in China. Therefore, the receiver constituting the antenna 300 and the positioning device 310 is configured to be able to perform positioning regardless of which of those satellite positioning systems it is.

[0014] The inertial sensor 600 is, for example, an autonomous sensor such as a three-axis acceleration sensor, a three-axis gyro sensor, and a three-axis geomagnetic sensor. The inertial sensor 600 outputs the output data of each sensor to the track detection model learning device 100.

[0015] The camera 400 is provided on the railway vehicle 1 and images the surroundings of the railway vehicle 1. In the present embodiment, as shown in FIG. 1, the camera 400 is provided on the front side of the railway vehicle 1 and images the front of the railway vehicle 1 as an example of the surroundings of the railway vehicle 1. The camera 400 sends the captured image information to the track detection model learning device 100 and the track detection device 200. The camera 400 is an example of an imaging device. Note that the installation position of the camera 400 is not limited to the front side. Also, the camera 400 is not limited to imaging the front of the railway vehicle 1, and may be configured to image the surroundings such as the side or the rear of the railway vehicle 1.

[0016] The storage device 500 is a storage medium such as an HDD (Hard Disc Drive) or an SSD (Solid State Drive), for example. In the storage device 500, a line detection model 501 for daytime images and a line detection model 502 for nighttime images are stored. Details of the line detection model 501 for daytime images and the line detection model 502 for nighttime images will be described later. Additionally, the storage device 500 may be configured to store a map database (map DB) in which the travel route of the railway vehicle 1 is registered.

[0017] The display device 700 displays various data from the line detection model learning device 100. The display device 700 is, for example, a monitor or the like.

[0018] The line detection model learning device 100 inputs the current position of the railway vehicle 1 measured by the positioning device 310 and the captured image in front of the railway vehicle 1 captured by the camera 400, and learns and constructs a line detection model 501 for daytime images and a line detection model 502 for nighttime images. Details of the line detection model learning device 100 will be described later.

[0019] The line detection device 200 inputs the current position of the railway vehicle 1 measured by the positioning device 310 and the captured image in front of the railway vehicle 1 captured by the camera 400, and uses the line detection model 501 for daytime images and the line detection model 502 for nighttime images stored in the storage device 500 to detect the line area on which the railway vehicle 1 travels from the captured image. Here, the line area is an area in the captured image where a pair of left and right rails exist. Details of the line detection device 200 will be described later.

[0020] Next, details of the line detection model learning device 100 will be described. FIG. 2 is a block diagram showing an example of the functional configuration of the line detection model learning device 100 according to the embodiment. FIG. 2 also shows an inertial sensor 600, a positioning device 301, a camera 400, and a storage device 500.

[0021] As shown in FIG. 2, the line detection model learning device 100 according to the present embodiment mainly includes a location-based position information acquisition unit 101, a location-based image information acquisition unit 102, a same location determination unit 103, an image association unit 104, a daytime image selection unit 105, a line detection unit 106, a nighttime image selection unit 107, a teacher data generation unit 108, a nighttime image line detection model construction unit 109, a daytime image acquisition unit 110, a line instruction unit 111, a daytime image line detection model construction unit 112, and a storage unit 150.

[0022] The location-based image information acquisition unit 102 acquires a plurality of high-illumination images and low-illumination images obtained by imaging the front of the railway vehicle 1 with the camera 400 during the running of the railway vehicle 1 from the camera 400. Here, the high-illumination image is an imaging image obtained by imaging the front of the railway vehicle 1 running under a high-illumination situation where the illuminance is higher than a predetermined illuminance. Also, the low-illumination image is an imaging image obtained by imaging the front of the railway vehicle 1 running under a low-illumination situation where the illuminance is lower than a predetermined illuminance.

[0023] In the present embodiment, as an example of the predetermined illuminance, the illuminance between the illuminance during the day and the illuminance at night is used. That is, in the present embodiment, the high-illumination image is an image captured from the railway vehicle 1 during the daytime hours, and hereinafter, this high-illumination image is referred to as a daytime image. Here, the daytime hours are, for example, the time period from sunrise to sunset.

[0024] Also, the low-illumination image is an image captured from the railway vehicle 1 during the nighttime hours, and hereinafter, this low-illumination image is referred to as a nighttime image. Here, the nighttime hours are, for example, the time period from sunset to sunrise of the next day.

[0025] The location-based image information acquisition unit 102 stores the acquired plurality of daytime images and the acquired plurality of nighttime images in the storage unit 150.

[0026] The location-specific position information acquisition unit 101 sequentially acquires the current position information of the railway vehicle 1 from the positioning device 310 in synchronization with the acquisition of daytime images and nighttime images, and stores the acquired plurality of position information in the storage unit 150. Note that the location-specific position information acquisition unit 101 may further correct the position information based on the detection signal from the inertial sensor 600.

[0027] The storage unit 150 is a storage medium such as an HDD or an SSD. In the storage unit 150 of the present embodiment, a plurality of daytime images, a plurality of nighttime images, and a plurality of position information are stored. Further, as shown in FIG. 2, the daytime image, the nighttime image, and the position information determined to be images of the same location are associated (in other words, linked) as image correspondence information 151. Note that in the example of the image correspondence information 151 in the present embodiment, the position information is associated with the daytime image and the nighttime image determined to be images of the same location. However, as long as at least the daytime image and the nighttime image are associated, the association of the position information is not necessary.

[0028] Here, the daytime image track detection model 501 stored in the storage device 500 will be described. The daytime image track detection model 501 is a learned model that inputs a daytime image and outputs a track area where a pair of left and right rails exists in the daytime image. The daytime image track detection model 501 is configured by, for example, a neural network and is learned and constructed by machine learning such as deep learning. Specifically, the daytime image track detection model 501 is generated in advance by the daytime image acquisition unit 110, the track teaching unit 111, and the daytime image track detection model construction unit 112 described below.

[0029] The daytime image acquisition unit 110 acquires a plurality of daytime images from the camera 400. These plurality of daytime images are used for learning.

[0030] The track instruction unit 111 receives an input of a track area instruction from the user for each of a plurality of daytime images for learning acquired by the daytime image acquisition unit 110. Specifically, the track instruction unit 111 displays each of the plurality of daytime images acquired on the display device 700, allows the user to instruct and specify the track area from each of the displayed daytime images, and receives the input of the instruction.

[0031] The daytime image track detection model construction unit 112 uses a region division algorithm for the daytime images for learning and machine learning such as deep learning to learn and construct a daytime image track detection model 501 from each of the plurality of daytime images for learning and each of the track area instructions received by the track instruction unit 111. Here, in the present embodiment, as an example of the region division algorithm, a semantic segmentation algorithm is used. However, the region division algorithm is not limited to this.

[0032] If this daytime image track detection model 501 is used, the detection of the track area from the daytime image is performed with high accuracy. However, in the case of an image captured in a low illuminance environment such as a nighttime image, it is difficult to accurately detect the track area from the nighttime image even if the daytime image track detection model 501 is used. Thus, when the illuminance conditions are different, the appearance of the track in the image changes greatly, and it becomes difficult to learn the track area at night.

[0033] FIG. 3 is an explanatory diagram showing an example of the detection result of the track area from the captured image using different daytime image track detection models 501. FIGS. 3(a), (b), and (c) show examples using different daytime image track detection models 501, respectively. Also, the upper images in FIGS. 3(a), (b), and (c) show daytime images, and the middle and lower images show nighttime images.

[0034] As shown in FIG. 3, in the upper daytime image, the track area is detected well. However, in the case of the nighttime images captured in the low illuminance environment shown in the middle and lower parts, over-detection or non-detection occurs, and it becomes difficult to accurately detect the track area.

[0035] FIG. 4 is an explanatory diagram showing an example when teaching a track area in a night image. As shown in FIG. 4, in a night image captured in a low-illumination environment, for example, a distant rail portion such as the portion indicated by reference numeral 5001 is unclear, and it becomes difficult to accurately teach the track area to the daytime image track detection model 501.

[0036] Therefore, in the present embodiment, by adopting the configuration shown below, such problems are solved.

[0037] Returning to FIG. 2, the same location determination unit 103 determines whether each of a plurality of daytime images and each of a plurality of night images stored in the storage unit 150 are images captured at the same location based on the location information stored in the storage unit 150 and the image features of the daytime images and night images stored in the storage unit 150. Details of such determination will be described later.

[0038] The image association unit 104 associates the daytime image and the night image determined by the same location determination unit 103 to be captured at the same location, and stores the result as image association information in the storage unit 150.

[0039] Next, the track detection model 502 for night images stored in the storage device 500 will be described. The track detection model 502 for night images is a learned model that inputs a night image and outputs a track area where a pair of left and right rails exist in the night image. The track detection model 502 for night images is configured by, for example, a neural network and is learned and constructed by machine learning such as deep learning. Here, the track detection model 502 for night images of the present embodiment is learned using, as teacher data, the track area in the daytime image output by inputting the daytime image at the same location corresponding to the learning night image into the track detection model 501 for daytime images. Since the teacher data is an accurate track area and such accurate teacher data is used, according to the present embodiment, it is possible to construct a track detection model 502 for night images that can accurately output the track area even for night images.

[0040] Specifically, the nighttime image line detection model 502 is generated by the nighttime image selection unit 107, the daytime image selection unit 105, the line detection unit 106, the teacher data generation unit 108, and the nighttime image line detection model construction unit 109, which will be described below.

[0041] The nighttime image selection unit 107 selects nighttime images for learning from the image correspondence information 151 in the storage unit 150. The nighttime image selection unit 107 selects a plurality of nighttime images for learning. The daytime image selection unit 105 selects daytime images corresponding to each of the nighttime images for learning selected by the nighttime image selection unit 107 from the image correspondence information 151 in the storage unit 150.

[0042] The line detection unit 106 inputs the daytime image selected by the daytime image selection unit 105 into the daytime image line detection model 501, and detects the line area output from the daytime image line detection model 501 as the line area in the daytime image. The teacher data generation unit 108 sets (or generates) the information on the line area in the daytime image detected by the line detection unit 106 as teacher data.

[0043] The nighttime image line detection model construction unit 109 uses a region segmentation algorithm for the nighttime images for learning and machine learning such as deep learning to learn and construct the nighttime image line detection model from each of the plurality of nighttime images for learning selected by the nighttime image selection unit 107 and each of the teacher data (that is, the line area of the daytime image output from the daytime image line detection model 501 by inputting the daytime image corresponding to the nighttime image for learning). Here, in this embodiment, as an example of the region segmentation algorithm, a semantic segmentation algorithm is used.

[0044] FIG. 5 is a diagram showing an example of an image to which a region division algorithm is applied to a daytime image for learning and a nighttime image for learning according to an embodiment. FIGS. 5(a) and 5(b) are examples of daytime images to which the region division algorithm in the daytime image line detection model construction unit 112 is applied. FIG. 5(c) is an example of a nighttime image to which the region division algorithm in the nighttime image line detection model construction unit 109 is applied.

[0045] As shown in FIG. 5(a), the daytime image line detection model construction unit 112 labels a learning image with a line rail, an area outside the line, etc. Similarly, the nighttime image line detection model construction unit 109 labels the learning nighttime image in FIG. 5(c) with a line rail, an area outside the line, etc. Then, a region division algorithm such as semantic segmentation is executed.

[0046] Next, details of the determination by the same location determination unit 103 as to whether the daytime image and the nighttime image are images taken at the same location will be described.

[0047] FIG. 6 is a diagram showing an example of the association between a daytime image and a nighttime image taken at the same location in the embodiment. The same location determination unit 103 associates a daytime image and a nighttime image taken at the same location as shown in the example of FIG. 6 based on image features.

[0048] As a first determination method, the same location determination unit 103 extracts edges as image features from each of a plurality of daytime images and each of a plurality of nighttime images, and based on the position information and the similarity between the extracted edges, determines whether each of the plurality of daytime images and each of the plurality of nighttime images are images taken at the same location.

[0049] Specifically, the same location determination unit 103 first generates edge images obtained by edge extraction for each of the daytime image and the nighttime image.

[0050] FIG. 7 is a diagram for explaining an example of a first determination method by the same location determination unit 103 according to the embodiment. FIG. 7(a) is an example of the acquired night image (i.e., the original image). FIG. 7(b) shows an image obtained by performing edge extraction on the image of FIG. 7(a). In the example of the edge image of FIG. 7(b), a vertical Sobel operator is applied. An edge image is similarly generated for the daytime image.

[0051] Next, the same location determination unit 103 calculates the similarity between the two edge images. More specifically, the same location determination unit 103 selects a plurality of daytime images that are candidate associations in the vicinity of the position information at the time of imaging for a certain night image. Then, the same location determination unit 103 calculates the similarity between the edge image of the night image and the edge images of the plurality of candidate daytime images according to the following formula (1).

[0052] [Equation] Here, in formula (1), for the two edge images, the absolute value of the difference in pixel values is calculated between the pixels of the same two-dimensional coordinates. The value obtained by dividing the sum of the absolute differences of all pixels by the total number of pixels in the image is calculated. The more similar the two edge images are, the smaller the value becomes, and the less similar they are, the larger the value becomes.

[0053] Then, the same location determination unit 103 determines that the daytime image of the edge image for which the similarity value calculated by formula (1) is the minimum is an image captured at the same location as the night image. By repeating such processing, the same location determination unit 103 determines whether each of the plurality of night images and each of the plurality of daytime images are at the same location.

[0054] As a second determination method, the same location determination unit 103 calculates a feature map, which is a map of feature amounts as image features, from each of the plurality of daytime images and each of the plurality of night images, and based on the position information and the similarity between the calculated feature maps, determines whether each of the plurality of daytime images and each of the plurality of night images are images captured at the same location.

[0055] Specifically, the same-location discrimination unit 103 first calculates the feature map of the nighttime image and the feature map of the daytime image. FIG. 8 is a diagram for explaining an example of a second determination method by the same-location discrimination unit 103 according to the embodiment. FIG. 8(a) shows an example of calculating a feature map from a nighttime image, and FIG. 8(b) shows an example of calculating a feature map from a daytime image.

[0056] The same-location discrimination unit 103 can use a tensor as the feature map. FIG. 9 is a diagram showing an example of a tensor used as the feature map generated by the same-location discrimination unit 103 according to the embodiment. As shown in FIG. 9, a two-dimensional tensor can be used as the feature map, but it is not limited thereto. For example, a feature vector (one-dimensional tensor) and a three-dimensional tensor can be used. However, the method for generating the feature map is not limited to this.

[0057] FIG. 10 is a diagram for explaining an example of calculating the similarity of the feature maps according to the embodiment. As shown in FIG. 10, the same-location discrimination unit 103 can use the precision rate, recall rate, F value, etc. as evaluation indices to calculate the similarity of the two feature maps.

[0058] Note that the similarity of the feature maps is not limited to these. For example, the same-location discrimination unit 103 can be configured to use the Euclidean distance between two feature maps as the similarity between the feature maps.

[0059] In FIG. 8, as a result of the same location determination unit 103 calculating the similarity of the feature maps in this way, it shows that the similarity between the feature map of the nighttime image and the feature maps of the three daytime images N-1, N, and N+1 is 60%, 90%, and 80% respectively. Therefore, the same location determination unit 103 determines that the daytime image N with a feature map similarity of 90% is the daytime image at the same location as the nighttime image. According to this second determination method, the images at the same location can be determined with higher accuracy than the first determination method.

[0060] As a third determination method, the same location determination unit 103 detects a track area from each of a plurality of daytime images and each of a plurality of nighttime images, and uses the detected track area as a feature map to determine whether each of the plurality of daytime images and each of the plurality of nighttime images are images taken at the same location based on the position information and the similarity between the calculated feature maps.

[0061] Specifically, the same location determination unit 103 detects a track area from each of a plurality of daytime images and each of a plurality of nighttime images. FIG. 11 is a diagram for explaining an example of a third determination method by the same location determination unit 103 according to the embodiment. FIG. 11(a) shows an example of a track area detected from a nighttime image, and FIG. 11(b) shows an example of a track area detected from a daytime image.

[0062] Next, the same location determination unit 103 calculates each feature map. The method for calculating the feature map is the same as that of the second determination method. Then, the same location determination unit 103 calculates the similarity between the nighttime image and the daytime image, that is, the similarity between the feature maps of the two images including the track detection results, and determines that the daytime image with the highest similarity among the plurality of candidate daytime images for a certain nighttime image is the daytime image at the same location as the nighttime image.

[0063] Here, in order to absorb the sway of the railway vehicle 1, the same location determination unit 103 also uses the nighttime image and the daytime image created by correcting the lateral displacement and rotation as comparison targets. In addition, in order to address the motion blur of the camera 400 caused by the acceleration of the railway vehicle 1, the same-location discrimination unit 103 also uses blurred images of night-time images and daytime images as comparison targets. Furthermore, in order to absorb the influence of weather, lighting, etc., the same-location discrimination unit 103 also uses images obtained by adjusting the luminance values, contrast, etc. of night-time images and daytime images as comparison targets. In addition, the same-location discrimination unit 103 also uses images generated by adjustment methods other than the above as comparison targets.

[0064] As described above, the daytime image and the night-time image determined to be captured at the same location by the same-location discrimination unit 103 are associated as image association information 151 by the above-described image association unit 104 and stored in the storage unit 150.

[0065] FIG. 12 is a diagram showing an example of a daytime image and a night-time image associated in the embodiment. FIG. 12(a) is an example of an image in which a track area that is a straight portion of the rail is shown, and FIG. 12(b) is an example of an image in which a track area that is a curved portion of the rail is shown.

[0066] Next, the details of the track detection device 200 will be described. FIG. 13 is a block diagram showing an example of the functional configuration of the track detection device 200 according to the embodiment. As shown in FIG. 13, the track detection device 200 according to the present embodiment mainly includes an image acquisition unit 201, a track detection unit 202, and a control unit 203.

[0067] The image acquisition unit 201 acquires daytime images and night-time images of the running railway vehicle 1 from the camera 400. The track detection unit 202 inputs the daytime image acquired by the image acquisition unit 201 into the daytime-image track detection model 501, and acquires the track area in the daytime image by obtaining the track area in the daytime image output from the daytime-image track detection model 501, thereby detecting the track area in the daytime image.

[0068] In addition, in the present embodiment, the track detection unit 202 inputs the nighttime image acquired by the image acquisition unit 201 into the track detection model 502 for nighttime images, and acquires the track region in the nighttime image output from the track detection model 502 for nighttime images, thereby detecting the track region in the nighttime image.

[0069] Based on the track region detected by the track detection unit 202, the control unit 203 performs various controls such as the running control of the railway vehicle 1.

[0070] Next, the track detection model learning process by the track detection model learning system 10 of the present embodiment configured as described above will be described.

[0071] FIG. 14 is a flowchart showing an example of the procedure of the overall process of track detection model learning according to the embodiment. First, the location-based position information acquisition unit 101, the location-based image information acquisition unit 102, the same location determination unit 103, and the image association unit 104 of the track detection model learning device 100 perform day image / night image association processing (S101).

[0072] Next, the daytime image acquisition unit 110, the track teaching unit 111, and the track detection model construction unit 112 for daytime images execute the learning process of the track detection model 501 for daytime images (S102). Then, the daytime image selection unit 105, the nighttime image selection unit 107, the track detection unit 106, the teacher data generation unit 108, and the track detection model construction unit 109 for nighttime images execute the learning process of the track detection model 502 for nighttime images (S103). As described above, the track detection model 502 for nighttime images is constructed and learned in the storage device 500.

[0073] Next, the details of the day image / night image association processing in S101 will be described. FIG. 15 is a flowchart showing an example of the procedure of the day image / night image association processing according to the embodiment.

[0074] First, while the railway vehicle 1 is running during the daytime, the following processes are performed. The location-specific image information acquisition unit 102 acquires a daytime image from the camera 400 (S201). In synchronization with this, the location-specific position information acquisition unit 101 acquires the position information of the current position of the railway vehicle 1 from the positioning device 310 (S202). Then, the location-specific image information acquisition unit 102 stores the acquired daytime image in the storage unit 150, and the location-specific position information acquisition unit 101 stores the acquired position information in the storage unit 150 (S203).

[0075] The processes from S201 to S203 are repeatedly executed while the railway vehicle 1 is running during the daytime.

[0076] Next, while the railway vehicle 1 is running during the nighttime, the following processes are performed. The location-specific image information acquisition unit 102 acquires a nighttime image from the camera 400 (S204). In synchronization with this, the location-specific position information acquisition unit 101 acquires the position information of the current position of the railway vehicle 1 from the positioning device 310 (S205). Then, the location-specific image information acquisition unit 102 stores the acquired nighttime image in the storage unit 150, and the location-specific position information acquisition unit 101 stores the acquired position information in the storage unit 150 (S206).

[0077] The processes from S204 to S206 are repeatedly executed while the railway vehicle 1 is running during the nighttime.

[0078] Next, the same-location determination unit 103 determines, based on the image features and position information of each of the plurality of daytime images and the plurality of nighttime images stored in the storage unit 150, whether the images are taken at the same location by the above-described method (S207).

[0079] Next, the image association unit 104 associates the daytime image and the nighttime image determined to be taken at the same location by the same-location determination unit 103 (S208). Then, the image association unit 104 further associates the position information with the associated daytime image and nighttime image, and stores it in the storage unit 150 as image association information 151 (S209). After that, the process returns to the calling source.

[0080] Next, the details of the daytime image line detection model learning process in the overall process of FIG. 14 will be described. FIG. 16 is a flowchart showing an example of the procedure of the daytime image line detection model learning process according to the embodiment.

[0081] First, the daytime image acquisition unit 110 acquires a daytime image for learning from the camera 400 (S401). Next, the line instruction unit 111 instructs the user or the like to indicate the line area in the daytime image for learning (S402). Then, the daytime image line detection model construction unit 112 inputs the instructed line area as teacher data, which is correct data, together with the daytime image for learning into the daytime image line detection model 501, and uses the semantic segmentation algorithm to learn and construct the daytime image line detection model 501 (S403).

[0082] The processes from S401 to S403 are repeatedly executed until a predetermined end condition is satisfied (S404, S404: No). When the predetermined end condition is satisfied in S404 (S404: Yes), the process returns to the caller.

[0083] Next, the details of the nighttime image line detection model learning process in the overall process of FIG. 14 will be described. FIG. 17 is a flowchart showing an example of the procedure of the nighttime image line detection model learning process according to the embodiment.

[0084] First, the nighttime image selection unit 107 selects a nighttime image for learning from the storage unit 150 (S501). Next, the daytime image selection unit 105 selects a daytime image of the same location associated with the nighttime image selected in S501 from the image correspondence information 151 in the storage unit 150 (S502).

[0085] Next, the line detection unit 106 inputs the daytime image selected in S502 into the daytime image line detection model 501, and obtains the line area output from the daytime image line detection model 501, thereby detecting the line area (S503). Next, the teacher data generation unit 108 sets the line area in the daytime image detected in S503 as teacher data (S504).

[0086] Then, the nighttime image line detection model construction unit 109 learns and constructs the nighttime image line detection model 502 from the learning nighttime images acquired in S501 and the teacher data which is the line area as the correct answer data set in S504 (S505). The nighttime image selection unit 107 determines whether the processing has been completed for all the nighttime images in the image correspondence information 151 of the storage unit 150 (S506). When the nighttime image selection unit 107 determines that the processing has not been completed for all the nighttime images yet (S506: No), the processing from S501 to S505 is repeatedly executed.

[0087] On the other hand, in S506, when the nighttime image selection unit 107 determines that the processing has been completed for all the nighttime images (S506: Yes), the processing returns to the calling source. Through the above processing, the nighttime image line detection model 502 is learned and constructed.

[0088] Next, the line detection processing from the nighttime image by the line detection device 200 will be described. FIG. 18 is a flowchart showing an example of the procedure of the line detection processing from the nighttime image according to the embodiment.

[0089] First, the image acquisition unit 201 acquires a nighttime image in front of the railway vehicle 1 running during the nighttime time zone from the camera 400 (S601). Next, the track detection unit 202 inputs the nighttime image acquired in S601 into the nighttime image track detection model 502 stored in the storage device 500 (S602). Then, the track detection unit 202 detects the track area by acquiring the track area output from the nighttime image track detection model 502 (S603). Next, the control unit 203 executes various driving controls based on the track area detected in S603 (S604).

[0090] As described above, in this embodiment, the track detection model learning device 100 includes a storage unit 150 that stores image correspondence information 151 associating each of a plurality of daytime images captured by the camera 400 in front of the railway vehicle 1 running during the daytime time zone with a higher illuminance than a predetermined illuminance, and a nighttime image captured by the camera 400 in the same imaging area as the imaging area of the daytime image at the same location as the imaging location of each of the plurality of daytime images in front of the railway vehicle 1 running during the nighttime time zone with a lower illuminance than the predetermined illuminance; a nighttime image selection unit 107 that selects a nighttime image for learning from the image correspondence information 151; a daytime image selection unit 105 that selects a daytime image corresponding to the nighttime image for learning from the image correspondence information 151; a track detection unit 106 that inputs the selected daytime image into a learned model, the daytime image track detection model 501, which inputs a daytime image and outputs a track area in the daytime image, to detect the track area in the daytime image; a teacher data generation unit 108 that generates information on the track area in the detected daytime image as teacher data; and a nighttime image track detection model construction unit 109 that learns and constructs a learned model, the nighttime image track detection model 502, which inputs a nighttime image and outputs a track area in the nighttime image, using a region segmentation algorithm for the nighttime image for learning from the nighttime image for learning and the teacher data.

[0091] That is, in this embodiment, an accurate line region detected using the daytime image line detection model 501 from a clear daytime image captured under high illuminance conditions is used as teacher data, and the nighttime image line detection model 502 is learned and constructed from this teacher data which is the accurate line region and a nighttime image captured at the same location as the above daytime image. Therefore, according to this embodiment, even when using a nighttime image as a low-illuminance image captured under a low-illuminance environment where the line region is unclear and difficult to teach, the nighttime image line detection model 502 for detecting an accurate line region from the nighttime image can be learned and constructed with high precision. Also, in this embodiment, by using the nighttime image line detection model 502 learned in this way, line detection can be accurately performed even on a nighttime image as a low-illuminance image.

[0092] Also, in this embodiment, the line detection model learning device 100 acquires a plurality of daytime images and a plurality of nighttime images from the camera 400, and a location-specific image information acquisition unit 102 that stores the acquired plurality of daytime images and the acquired plurality of nighttime images in the storage unit 150, and synchronously with the acquisition of the daytime images and the acquisition of the nighttime images, acquires position information of the railway vehicle 1 from the positioning device 310 that measures the current position of the railway vehicle 1, and a location-specific position information acquisition unit 101 that stores the acquired plurality of position information in the storage unit 150, and based on the position information, the image features of the daytime images and the nighttime images, a same location determination unit 103 that determines whether each of the plurality of daytime images stored in the storage unit 150 and each of the plurality of nighttime images stored in the storage unit are images captured at the same location, and an image association unit 104 that associates the daytime image and the nighttime image determined to be captured at the same location and stores them in the storage unit 150 as image correspondence information 151. Therefore, according to this embodiment, the daytime image as a high-illuminance image and the nighttime image as a low-illuminance image captured at the same location can be associated and held, and the daytime image at the same location can be easily acquired from the nighttime image for learning. Therefore, according to this embodiment, the nighttime image line detection model 502 for detecting an accurate line region from the nighttime image can be learned and constructed with high precision.

[0093] Also, in this embodiment, the same location determination unit 103 of the line detection model learning device 100 extracts edges as image features from each of a plurality of daytime images and each of a plurality of nighttime images, and determines whether each of the plurality of daytime images and each of the plurality of nighttime images are images taken at the same location based on the position information and the similarity between the extracted edges. Therefore, according to this embodiment, the daytime images and nighttime images at the same location can be associated with high accuracy. Accordingly, according to this embodiment, the line detection model 502 for nighttime images for detecting an accurate line region from nighttime images can be learned and constructed with higher accuracy.

[0094] Also, in this embodiment, the same location determination unit 103 of the line detection model learning device 100 calculates a feature map, which is a map of feature amounts as image features, from each of a plurality of high-brightness daytime images and each of a plurality of nighttime images, and determines whether each of the plurality of daytime images and each of the plurality of nighttime images are images taken at the same location based on the position information and the similarity between the calculated feature maps. Therefore, according to this embodiment, the daytime images and nighttime images at the same location can be associated with higher accuracy. Accordingly, according to this embodiment, the line detection model 502 for nighttime images for detecting an accurate line region from nighttime images can be learned and constructed with higher accuracy.

[0095] Also, in this embodiment, the same location determination unit 103 of the line detection model learning device 100 detects a line region from each of a plurality of daytime images and each of a plurality of nighttime images, and uses the detected line region as a feature map to determine whether each of the plurality of daytime images and each of the plurality of nighttime images are images taken at the same location based on the position information and the similarity between the calculated feature maps. Therefore, according to this embodiment, the daytime images and nighttime images at the same location can be associated with higher accuracy. Accordingly, according to this embodiment, the line detection model 502 for nighttime images for detecting an accurate line region from nighttime images can be learned and constructed with higher accuracy.

[0096] In addition, in the present embodiment, the line detection model learning device 100 further includes a daytime image acquisition unit 110 that acquires daytime images, a line teaching unit 111 that receives an input of line area teaching in the acquired daytime images, and a daytime image line detection model construction unit 112 that learns and constructs a daytime image line detection model 501 using a region division algorithm for the acquired daytime images from the acquired daytime images and the received teaching. Therefore, according to the present embodiment, by using the learned and constructed daytime image line detection model 501, even when using a nighttime image taken in a low illumination environment and being a low illumination image with an unclear line area and difficult to teach, a nighttime image line detection model 502 for accurately detecting the line area from the nighttime image can be learned and constructed with high precision.

[0097] In addition, in the present embodiment, the region division algorithm is a semantic segmentation algorithm. Therefore, in the present embodiment, the nighttime image line detection model 502 can be learned and constructed with higher precision.

[0098] In addition, the line detection device 200 of the present embodiment includes an image acquisition unit 201 that acquires a nighttime image as a low illumination image obtained by imaging the front of the railway vehicle 1 running during a nighttime time zone with a low illumination lower than a predetermined illumination by a camera 400, and a line detection unit 202 that inputs the acquired nighttime image into the nighttime image line detection model 502 to detect a line area. The nighttime image line detection model 502 is a learned model that inputs a nighttime image and outputs a line area in the nighttime image. The daytime image line detection model 501, which is a learned model that inputs a daytime image taken by the camera 400 of the front of the railway vehicle 1 running during a daytime time zone with a high illumination higher than a predetermined illumination and outputs a line area in the daytime image, is input with the detected line area in the daytime image as teacher data, and is learned using a region division algorithm from the learning nighttime image and the teacher data.

[0099] That is, in the present embodiment, even when using a nighttime image as a low-illumination image that is captured in a low-illumination environment and has an unclear line area and is difficult to teach, line detection is performed using a line detection model 502 for nighttime images that can detect an accurate line area from the nighttime image. Therefore, according to the present embodiment, by using the line detection model 502 for nighttime images learned in this way, line detection can be accurately performed even on a nighttime image as a low-illumination image.

[0100] (Modification example) In the above embodiment, a line detection model 502 for nighttime images is provided separately from the line detection model 501 for daytime images, and the line detection model 502 for nighttime images is learned, but it is not limited to this.

[0101] For example, from a learning nighttime image and the teacher data of the line area output from the line detection model 501 for daytime images, using a region segmentation algorithm for the learning nighttime image, the line detection model 501 for daytime images is learned, and the learned line detection model 501 for daytime images is configured as the line detection model 502 for nighttime images for the line detection model construction unit 109. Thereby, since the line detection model 501 for daytime images is learned and constructed as the line detection model 502 for nighttime images, the processing efficiency of learning can be improved.

[0102] Also, in the above embodiment, the line detection model 501 for daytime images and the line detection model 502 for nighttime images are stored in a storage device 500 separate from the line detection model learning device 100 and the line detection device 200, but it is not limited to this. For example, the line detection model 501 for daytime images and the line detection model 502 for nighttime images can be provided in the storage unit 150 etc. of the line detection model learning device 100, or can be provided in the line detection device 200. Thereby, the device configuration can be simplified.

[0103] In the above-described embodiment, a daytime image was used as the high-luminance image and a nighttime image was used as the low-luminance image, but it is not limited thereto. For example, as the high-luminance image, an image captured during a daytime time zone when the weather is sunny may be used, and as the low-luminance image, an image captured during a daytime (and nighttime) time zone under bad weather may be used. Even in this case, the same operational effects as those of the above-described embodiment are achieved.

[0104] The line detection model learning device 100 of the above-described embodiment includes a control device such as a CPU, a storage device such as a ROM (Read Only Memory) and a RAM, an external storage device such as an HDD, an SSD, and a CD drive device, a display device such as a display device, and an input device such as a keyboard and a mouse, and has a hardware configuration using a normal computer.

[0105] The line detection model learning program executed by the line detection model learning device 100 of the above-described embodiment is provided by being pre-embedded in a ROM or the like.

[0106] The line detection model learning program executed by the line detection model learning device 100 of the above-described embodiment may be configured to be recorded and provided on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, and a DVD (Digital Versatile Disk) in an installable format or an executable format file.

[0107] Furthermore, the line detection model learning program executed by the line detection model learning device 100 of the above-described embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the line detection model learning program executed by the line detection model learning device 100 of the above-described embodiment may be configured to be provided or distributed via a network such as the Internet.

[0108] The line detection model learning program executed by the line detection model learning apparatus 100 of the above-described embodiment has a module configuration including each of the above-described functional units. As actual hardware, the CPU (processor) reads the line detection model learning program from the above-described ROM and executes it, so that each of the above-described functional units (location-specific position information acquisition unit 101, location-specific image information acquisition unit 102, same location determination unit 103, image association unit 104, daytime image selection unit 105, line detection unit 106, nighttime image selection unit 107, teacher data generation unit 108, nighttime image line detection model construction unit 109, daytime image acquisition unit 110, line instruction unit 111, daytime image line detection model construction unit 112) is loaded onto the main storage device, and each functional unit is generated on the main storage device.

[0109] The line detection apparatus 200 of the above-described embodiment includes a control device such as a CPU, a storage device such as a ROM and a RAM, an external storage device such as an HDD, an SSD, and a CD drive device, a display device such as a display device, and an input device such as a keyboard and a mouse, and has a hardware configuration using an ordinary computer.

[0110] The line detection program executed by the line detection apparatus 200 of the above-described embodiment is provided by being pre-embedded in a ROM or the like.

[0111] The line detection program executed by the line detection apparatus 200 of the above-described embodiment may be configured to be recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, and a DVD in an installable format or an executable format file and provided.

[0112] Furthermore, the line detection program executed by the line detection apparatus 200 of the above-described embodiment may be configured to be stored on a computer connected to a network such as the Internet and downloaded via the network for providing. Also, the line detection program executed by the line detection apparatus 200 of the above-described embodiment may be configured to be provided or distributed via a network such as the Internet.

[0113] The line detection program executed by the line detection device 200 of the above-described embodiment has a module configuration including the above-described respective functional units. As actual hardware, the CPU (processor) reads the line detection model learning program from the above ROM and executes it, whereby the above respective functional units (image acquisition unit 201, line detection unit 202, control unit 203) are loaded onto the main storage device, and each functional unit is generated on the main storage device.

[0114] Although some embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0115] 1 Railway vehicle 10 Line detection model learning system 100 Line detection model learning device 101 Location-specific position information acquisition unit 102 Location-specific image information acquisition unit 103 Same location discrimination unit 104 Image association unit 105 Daytime image selection unit 106 Line detection unit 107 Nighttime image selection unit 108 Teacher data generation unit 109 Nighttime image line detection model construction unit 110 Daytime image acquisition unit 111 Line instruction unit 112 Daytime image line detection model construction unit 150 Storage unit 151 Image correspondence information 200 Line detection device 201 Image acquisition unit 202 Line Detection Unit 203 Control Unit 300 Antenna 310 Positioning Device 400 Camera 500 Storage Device 501 Line Detection Model for Daytime Images 502 Line Detection Model for Nighttime Images 600 Inertial Sensor 700 Display Device

Claims

1. a storage unit that stores image correspondence information that associates each of a plurality of high-illuminance images captured by an imaging device of the surroundings of a traveling railway vehicle under a high-illuminance condition higher than a predetermined illuminance with each of a plurality of low-illuminance images captured by the imaging device at the same locations as the imaging locations of each of the plurality of high-illuminance images, the images being of the surroundings of the traveling railway vehicle under a low-illuminance condition lower than the predetermined illuminance; a low-illumination image selection unit that selects a low-illumination image for learning from the image correspondence information; a high-illuminance image selection unit that selects the high-illuminance image corresponding to the low-illuminance image for learning from the image correspondence information; a track detection unit that inputs the selected high-illumination image into a track detection model for high-illumination images, the track detection model being a trained model that inputs the high-illumination image and outputs a track area in the high-illumination image, and detects a track area in the high-illumination image; A teacher data generating unit that generates information on a railroad area in the detected high-illuminance image as teacher data; a low-illuminance image track detection model construction unit that learns and constructs a low-illuminance image track detection model, which is a trained model that inputs the low-illuminance image and outputs a track area in the low-illuminance image, from the low-illuminance image for training and the teacher data, using a region segmentation algorithm for the low-illuminance image for training; A track detection model learning device comprising:

2. a location-specific image information acquisition unit that acquires the plurality of high-illuminance images and the plurality of low-illuminance images from the imaging device and stores the plurality of high-illuminance images and the plurality of low-illuminance images in the storage unit; a location-specific position information acquisition unit that acquires position information of the railway vehicle from a positioning device that measures a current position of the railway vehicle in synchronization with acquisition of the high-illuminance image and the low-illuminance image, and stores the acquired pieces of position information in the storage unit; an identical location determination unit that determines whether each of the plurality of high-illuminance images stored in the storage unit and each of the plurality of low-illuminance images saved in the storage unit are images captured at the same location based on the location information and image features of the high-illuminance image and the low-illuminance image; an image matching unit that matches the high illuminance image and the low illuminance image determined to have been captured at the same location and stores the matched image information in the storage unit; The railroad detection model learning device according to claim 1 , further comprising:

3. the same-location determination unit extracts edges as the image features from each of the plurality of high-illuminance images and each of the plurality of low-illuminance images, and determines whether each of the plurality of high-illuminance images and each of the plurality of low-illuminance images are images captured at the same location based on the position information and a similarity between the extracted edges. The railroad detection model learning device according to claim 2 .

4. the same location determination unit calculates a feature map, which is a map of feature amounts as the image features, from each of the plurality of high-illuminance images and each of the plurality of low-illuminance images, and determines whether each of the plurality of high-illuminance images and each of the plurality of low-illuminance images are images captured at the same location based on the position information and a similarity between the calculated feature maps. The railroad detection model learning device according to claim 2 .

5. the same-location determination unit detects a railroad area from each of the plurality of high-illuminance images and each of the plurality of low-illuminance images, and determines whether or not each of the plurality of high-illuminance images and each of the plurality of low-illuminance images are images captured at the same location based on the position information and the calculated similarity between the feature maps, using the detected railroad area as the feature map. The railroad detection model learning device according to claim 4 .

6. a high-illuminance image acquisition unit for acquiring the high-illuminance image; A track teaching unit that receives input of a track area teaching in the acquired high illumination image; a high illuminance image track detection model constructing unit that learns and constructs the high illuminance image track detection model from the acquired high illuminance image and a received instruction by using the region segmentation algorithm for the acquired high illuminance image; The railroad detection model learning device according to claim 1 , further comprising:

7. The segmentation algorithm is a semantic segmentation algorithm. The railroad detection model learning device according to claim 1 .

8. the low-illuminance image track detection model construction unit learns the high-illuminance image track detection model from the learning low-illuminance image and the teacher data by using a region segmentation algorithm for the learning low-illuminance image, and constructs the learned high-illuminance image track detection model as the low-illuminance image track detection model. The railroad detection model learning device according to claim 1 .

9. the high illuminance image is a daytime image captured of the surroundings of the railroad vehicle traveling during daytime hours, The low-illumination image is a nighttime image captured around the railroad vehicle traveling during a nighttime period. The railroad detection model learning device according to claim 1 .

10. A railroad detection model learning method executed by a railroad detection model learning device, comprising: The railroad detection model learning device includes: a storage unit that stores image correspondence information that associates each of a plurality of high-illuminance images captured by an imaging device of the surroundings of a traveling railway vehicle under a high-illuminance condition higher than a predetermined illuminance with each of a plurality of low-illuminance images captured by the imaging device at the same locations as the respective high-illuminance images of the surroundings of the traveling railway vehicle under a low-illuminance condition lower than the predetermined illuminance, selecting a low-illumination image for learning from the image correspondence information; selecting the high illuminance image corresponding to the low illuminance image for learning from the image correspondence information; inputting the selected high-illumination image into a track detection model for high-illumination images, which is a trained model that inputs the high-illumination image and outputs a track area in the high-illumination image, to detect a track area in the high-illumination image; generating information on a railroad area in the detected high-illumination image as training data; learning and constructing a track detection model for low-illumination images, which is a trained model that inputs the low-illumination image and outputs a track area in the low-illumination image, using a region segmentation algorithm for the low-illumination image for training from the training low-illumination image and the teacher data; A method for training a railroad detection model comprising:

11. an image acquisition unit that acquires a low-illumination image of the surroundings of a traveling railway vehicle captured by an imaging device under a low-illumination condition lower than a predetermined illuminance; a track detection unit that detects a track area by inputting the acquired low-illumination image into a track detection model for low-illumination images; The track detection model for low-illumination images is a trained model that inputs the low-illumination image and outputs a railroad area in the low-illumination image; a high-illuminance image of the surroundings of the railway vehicle traveling under a high-illuminance condition higher than the predetermined illuminance is taken by an imaging device, and the high-illuminance image is input to a track detection model for high-illuminance images, which is a trained model that inputs the high-illuminance image and outputs a track area in the high-illuminance image; and the detected track area in the high-illuminance image is used as training data, and training is performed using a region segmentation algorithm from the low-illuminance image for training and the training data. Track detection device.

Images