Judgment system and judgment method

Abstract

To efficiently perform maintenance work on objects at railway facilities. [Solution] The judgment system 1 is mounted on a railway vehicle running on rail R1 and comprises an imaging unit 21 that captures images of the surroundings of the running railway vehicle and outputs the captured images showing the object, a detection unit 42 that inputs the captured images into a first trained model and detects at least the object from the captured images, an information acquisition unit 43 that acquires object information related to the object detected by the detection unit 42, and a judgment unit 44 that determines whether or not there is an abnormality in the object based on the object information.

Description

[Technical Field]

[0001] The present disclosure relates to a determination system and a determination method. [Background technology]

[0002] Patent Document 1 describes an abnormality monitoring device that monitors abnormalities in a circuit breaker based on a circuit breaker bar image, which is an image of the circuit breaker bar. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-41173 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned abnormality monitoring device, the portion of the barrier rod is extracted from the photographed image of the barrier rod, and an abnormality in the barrier is detected. In railway facilities, it is desirable to perform maintenance work on objects to be maintained efficiently.

[0005] The present disclosure aims to provide a determination system and a determination method that enable efficient maintenance work on objects in railway facilities. [Means for solving the problem]

[0006] The judgment system disclosed herein is [1] "a judgment system comprising: an imaging unit mounted on a railway vehicle running on rails, which captures images of the surroundings of the running railway vehicle and outputs captured images of objects; a detection unit which inputs the captured images into a first trained model and detects at least the objects from the captured images; an information acquisition unit which acquires object information regarding the objects detected by the detection unit; and a judgment unit which determines whether or not there is an abnormality in the object based on the object information."

[0007] In this determination system, an imaging unit mounted on a railway vehicle traveling on rails captures images of the area around the railway vehicle, thereby enabling efficient imaging of objects. An information acquisition unit then acquires object information about the object detected by the detection unit, and a determination unit determines whether or not there is an abnormality in the object based on the object information. This allows for highly accurate determination of whether or not there is an abnormality in the object from the captured image. As a result, maintenance work on objects in railway facilities can be performed efficiently.

[0008] The determination system of the present disclosure may be [2] "the determination system described in [1], wherein the imaging unit captures an image of an overhead line as the object located above the rail and a tree located around the overhead line, the detection unit detects a first portion corresponding to the overhead line and a second portion corresponding to the tree in the captured image, the information acquisition unit acquires the number of pixels included in both the first portion and the second portion in the captured image as the object information, and the determination unit determines whether the overhead line and the tree are in close proximity based on the number of pixels." In this case, by the information acquisition unit acquiring the number of pixels included in both the first portion corresponding to the overhead line and the second portion corresponding to the tree, the determination unit can accurately determine whether the overhead line and the tree are in close proximity based on the number of pixels.

[0009] The determination system of the present disclosure may be [3] "the determination system described in [2], wherein the imaging unit has an upward-facing camera that captures an image diagonally upward relative to the horizontal direction, and the detection unit detects the first portion and the second portion from the captured image captured by the upward-facing camera." For example, when the imaging unit captures an image directly ahead, an overhead wire in the foreground and a tree in the background may appear to overlap in the captured image even if they are not actually close to each other, making it difficult to accurately determine whether the overhead wire and the tree are close to each other. However, by using an image captured by an upward-facing camera, the influence of perspective can be reduced, and it is possible to more accurately determine whether the overhead wire and the tree are close to each other.

[0010] The determination system of the present disclosure may be [4] "the determination system described in [1], wherein the rail is provided on a track bed, the imaging unit captures an image of a reinforcement located outside the rail and formed by a rise in the track bed as the object, the detection unit detects a portion of the captured image that captures the reinforcement, the information acquisition unit acquires height information on the height of the reinforcement as the object information, and the determination unit determines whether the reinforcement is low or not based on the height information." In this case, by the information acquisition unit acquiring the height information on the height of the reinforcement, the determination unit can more accurately determine whether the reinforcement is low or not based on the height information.

[0011] The determination system of the present disclosure may be [5] "the determination system according to [4], wherein the information acquisition unit inputs the captured image in which the excess buildup is detected into a second trained model to acquire the height information." In this case, excess buildup height information can be acquired accurately and easily.

[0012] The determination system of the present disclosure may be [6] "the determination system according to [4] or [5], wherein the information acquisition unit acquires depth information for the portion that captures the excess buildup, and acquires the height information based on the depth information." In this case, the height information of the excess buildup can be acquired with high accuracy based on the depth information.

[0013] The determination method of the present disclosure is [7] "a determination method executed by a determination system, comprising: an imaging step of capturing an image of the periphery of a railway vehicle traveling on rails and outputting a captured image showing an object; a detection step of inputting the captured image into a first trained model and detecting at least the object from the captured image; an information acquisition step of acquiring object information related to the object detected by the detection step; and a determination step of determining whether or not there is an abnormality in the object based on the object information." According to this determination method, for the reasons described above, maintenance work on objects in railway facilities can be performed efficiently.

[0014] The determination method of the present disclosure may be [8] "the determination method described in [7], wherein in the imaging step, an overhead wire as the object located above the rail and trees located around the overhead wire are imaged; in the detection step, a first portion corresponding to the overhead wire and a second portion corresponding to the tree are detected in the captured image; in the information acquisition step, the number of pixels included in both the first portion and the second portion in the captured image are acquired as the object information; and in the determination step, it is determined whether or not the overhead wire and the tree are in proximity to each other based on the number of pixels." According to this determination method, for the reasons described above, it is possible to accurately determine whether or not the overhead wire and the tree are in proximity to each other.

[0015] The determination method of the present disclosure may be "9" "a determination method described in [7], in which the rail is provided on a trackbed, and in the imaging step, an image of a reinforcement located outside the rail and formed by a rise in the trackbed is taken as the object, in the detection step, a portion of the captured image that shows the reinforcement is detected, in the information acquisition step, height information on the height of the reinforcement is acquired as the object information, and in the determination step, it is determined whether or not the reinforcement is low based on the height information." According to this determination method, for the reasons described above, it is possible to accurately determine whether or not the reinforcement is low. [Effects of the Invention]

[0016] According to the present disclosure, maintenance work can be efficiently performed on objects in railway facilities. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing the configuration of a determination system. [Figure 2] FIG. 2 is a block diagram showing the configuration of the imaging device. [Figure 3] FIG. 3 is a block diagram showing a general hardware configuration of a processing server. [Figure 4]FIG. 4 is a block diagram showing the configuration of the processing server. [Figure 5] FIG. 5 is a diagram schematically showing the positional relationship between the imaging device and the target object in the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of a determination method in the determination system. [Figure 7] FIG. 7 is a diagram showing an example of a method for processing a captured image by the determination system. [Figure 8] FIG. 8 is a diagram illustrating an example of a determination method performed by the determination system. [Figure 9] FIG. 9 is a flowchart showing an example of a learning method for the first learned model. [Figure 10] FIG. 10 is a diagram schematically showing the positional relationship between the imaging device and the target object in the second embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of a determination method in the determination system. [Figure 12] FIG. 12 is a diagram showing an example of a method for processing a captured image by the determination system. [Figure 13] FIG. 13 is a diagram showing an image of a method for acquiring depth information. [Figure 14] FIG. 14 is a flowchart showing an example of a learning method for the first learned model. [Figure 15] FIG. 15 is a flowchart showing an example of a learning method for the second learned model. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, preferred embodiments of the determination system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are designated by the same reference numerals, and duplicated explanations will be omitted.

[0019] [Configuration of the judgment system] First, the overall configuration of a determination system 1 according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the configuration of the determination system 1. The determination system 1 is a system that determines whether or not an object in a railway facility has an abnormality in order to perform maintenance on the object. In this example, the determination system 1 makes the above determination from an image of the object. The railway facility is a facility used for railways, such as a railway track or electric circuit equipment (power transmission, distribution, and feeder lines). The object is an object that is subject to maintenance in the railway facility, such as excess bank on a railway track or an overhead wire in electric circuit equipment.

[0020] The determination system 1 includes an imaging device 2, an intermediate server 3, a processing server 4, a database 5, and a display device 6. The imaging device 2 is connected to the intermediate server 3 via a network N. The intermediate server 3 is connected to the processing server 4 via the network N. Images from the imaging device 2 are transmitted to the processing server 4 via the intermediate server 3. The number of intermediate servers 3 may be any number equal to or greater than one. The processing server 4 processes images from the imaging device 2. The processing server 4 is connected to a database 5 and a display device 6 via the network N. Images from the imaging device 2 may be transmitted to the processing server 4 without passing through the intermediate server 3. Alternatively, images from the imaging device 2 may be transmitted to the database 5 without passing through both the intermediate server 3 and the processing server 4. In this case, the processing server 4 acquires images from the database 5 and processes the images. The database 5 stores images processed by the processing server 4. The display device 6 outputs the processing results of the processing server 4. The processing server 4 may be connected to the database 5 and the display device 6 via a local cable. The imaging device 2 is mounted on a railway vehicle. On the other hand, the intermediate server 3, processing server 4, database 5, and display device 6 do not have to be mounted on a railway vehicle, but may be placed, for example, in facilities (such as stations) along the railway line.

[0021] The imaging device 2 captures images of objects by capturing images of the surroundings of a traveling railway vehicle. A railway vehicle is a vehicle that travels on railroad tracks, such as a locomotive or a train. The imaging device 2 is mounted, for example, on the lead car of the railway vehicle. As shown in FIG. 2, the imaging device 2 includes an imaging unit 21, a computer 22, a sensor 23, a communication device 24, an input device 25, a display device 26, and a power supply 27. FIG. 2 is a block diagram showing the configuration of the imaging device 2. A user inputs operating conditions to the above devices into the input device 25. The display device 26 outputs images captured by the imaging unit 21. The power supply 27 supplies power to the devices that make up the imaging device 2.

[0022] The imaging unit 21 includes an upward-facing camera 21a, a front-facing camera 21b, and a downward-facing camera 21c. The cameras 21a to 21c are, for example, monocular cameras, stereo cameras, or TOF (Time Of Flight) cameras. When the cameras 21a to 21c are monocular cameras, the portability of the imaging device 2 and the ease of installing the imaging device 2 on a railway vehicle are improved. When the cameras 21a to 21c are stereo cameras or TOF cameras, depth information relative to the target object can be easily obtained. Each of the cameras 21a to 21c includes an imaging lens 21d, a control unit 21e, and a status display unit 21f.

[0023] Upward-facing camera 21a captures an image diagonally upward with respect to the horizontal direction. Upward-facing camera 21a is mounted on the railway vehicle so that imaging lens 21d faces diagonally upward with respect to the horizontal direction. Front-facing camera 21b captures an image directly ahead in the direction of travel of the railway vehicle. Front-facing camera 21b is mounted on the railway vehicle so that imaging lens 21d faces horizontally. Downward-facing camera 21c captures an image diagonally downward with respect to the horizontal direction. By using cameras 21a to 21c that are appropriate for the position of the object, the object can be suitably imaged.

[0024] The cameras 21a to 21c output a plurality of captured images, including a captured image of an object, to the computer 22. The plurality of captured images may include captured images that do not capture the object in addition to captured images that capture the object. The cameras 21a to 21c may output only captured images that capture the object, without outputting captured images that do not capture the object. Each of the plurality of captured images may be a video or a still image. The frame rate of each of the cameras 21a to 21c is, for example, 5 fps to 60 fps, and is, for example, 10 fps. The shutter speed of each of the cameras 21a to 21c is, for example, 1 ms to 5 ms, and is, for example, 2 ms. The frame rate and shutter speed may be appropriately adjusted by the control unit 21e depending on the traveling speed of the railway vehicle. The imaging unit 21 may include at least one of the cameras 21a to 21c.

[0025] The computer 22 is connected to the imaging unit 21, the sensor 23, the communication device 24, the input device 25, and the display device 26, and controls these devices. The computer 22 also stores the captured images from the imaging unit 21. The sensor 23 acquires position information or acceleration information of the railway vehicle and outputs it to the computer 22. The sensor 23 is, for example, a GPS sensor or an acceleration sensor. The computer 22 transmits the position information or acceleration information of the railway vehicle in addition to the captured images to the intermediate server 3 via the communication device 24.

[0026] 3 is a block diagram showing a general hardware configuration of the processing server 4. The processing server 4 includes a CPU (processor) 101 that executes an operating system, application programs, etc., a main memory unit 102 consisting of ROM and RAM, an auxiliary memory unit 103 consisting of a hard disk, flash memory, etc., a communication control unit 104 consisting of a network card or wireless communication module, input devices 105 such as a keyboard and a mouse, and output devices 106 such as a display and a printer.

[0027] Each functional element of the processing server 4, which will be described later, is realized by loading predetermined software onto the CPU 101 or main memory unit 102, operating the communication control unit 104, input device 105, output device 106, display device 6, etc. under the control of the CPU 101, and reading and writing data from and to the main memory unit 102 or auxiliary memory unit 103. Data and databases required for processing are stored in the main memory unit 102 or auxiliary memory unit 103.

[0028] When a moving image is transmitted from the imaging device 2, the processing server 4 divides the moving image into frames to generate still images. The processing server 4 adds position information and acceleration information of the railway vehicle to the still images, and stores the still images and the information in a predetermined storage device. The processing server 4 performs image processing on the generated still images.

[0029] 4 is a block diagram showing the configuration of the processing server 4. The processing server 4 includes a learning unit 41, a detection unit 42, an information acquisition unit 43, and a determination unit 44 as functional components.

[0030] The learning unit 41 trains a learning model by machine learning using teacher data to learn (construct) a first learned model and a second learned model. The learning model is, for example, a convolutional neural network (CNN). The detection unit 42 inputs a captured image to the first learned model and detects at least an object from the captured image. The information acquisition unit 43 acquires object information related to the object detected by the detection unit 42. The object information is information that can be acquired from a portion of the captured image that corresponds to the object. The information acquisition unit 43 may input the captured image in which the object is detected to the second learned model to acquire the object information. The determination unit 44 determines whether or not there is an abnormality in the object based on the object information. "There is an abnormality in the object" means, for example, that the object is in a state in which repair, maintenance, etc. is required.

[0031] [Determination method according to the first embodiment] Next, a determination method according to the first embodiment using the determination system 1 will be described. FIG. 5 is a diagram schematically illustrating the positional relationship between the imaging device 2 and an object. In this example, the object is an overhead wire 30, and the determination system 1 determines whether the overhead wire 30 and a tree 40 are close to each other. If it is determined that the overhead wire 30 and the tree 40 are close to each other, the user adjusts the positional relationship between the overhead wire 30 and the tree 40. For example, the user adjusts the positional relationship between the overhead wire 30 and the tree 40 by cutting down the tree 40. The overhead wire 30 is an electric wire for supplying power to railway vehicles and is stretched above a railway track (rail R1). The railway track includes rails R1 and sleepers R2 installed on the trackbed. The tree 40 is located along the railway line, and a portion of the tree 40 is located around the overhead wire 30. In this example, the imaging device 2 uses an upward-facing camera 21a as the imaging unit 21. The upward-facing camera 21a captures images at an angle of 45 degrees diagonally upward from the horizontal, for example.

[0032] Fig. 6 is a flowchart showing, as flow S1, an example of a determination method in the determination system 1. Fig. 7 is a diagram showing an example of a method of processing a captured image by the determination system 1. Fig. 8 is a diagram showing an example of a determination method by the determination system 1.

[0033] In step S11 (imaging step), the imaging unit 21 captures an image of the area around the railway vehicle and outputs a captured image 50 showing the overhead wires 30 and trees 40. The imaging device 2 transmits the captured image 50 to the processing server 4. In this example, the captured image 50 shows two rows of overhead wires 31 and 32.

[0034] In step S12, the detection unit 42 cuts out a partial image 51 from the captured image 50, depicting the overhead wires 30 and the trees 40, as part of the captured image 50. The range of the partial image 51 is set based on the angle of the upward-facing camera 21a. The detection unit 42 cuts out, for example, the upper half of the captured image 50 as the partial image 51. Alternatively, if the horizontal and vertical lengths of the captured image 50 are set to 100%, the detection unit 42 may generate the partial image 51 by excluding the portion from the bottom to 50% of the captured image 50, the portion from the left to 15% of the captured image 50, and the portion from the right to 35% of the captured image 50. In this case, the partial image 51 is an image located in the upper half of the captured image 50 and in the leftmost region. The range of the partial image 51 may be changed as appropriate depending on the positions of the overhead wires 30 and the trees 40.

[0035] In step S13 (detection step), the detection unit 42 inputs the partial image 51 into a first trained model to detect a first portion 52 corresponding to the overhead wire 30 and a second portion 53 corresponding to the tree 40. In this example, the first trained model is for performing segmentation. The first trained model classifies each pixel of the partial image 51 into one of the overhead wire 30, the tree 40, and other background. A set of pixels classified as the overhead wire 30 corresponds to the first portion 52, and a set of pixels classified as the tree 40 corresponds to the second portion 53. In this example, two rows of first portions 52 corresponding to the two rows of overhead wires 31, 32 are detected. The detection unit 42 classifies the first portions 52 of one row into a first portion 52a corresponding to the overhead wire 31, and classifies the first portions 52 of the other row into a first portion 52b corresponding to the overhead wire 32. The detection unit 42 may generate an image 54 including the first portion 52 and an image 55 including the second portion 53.

[0036] The detection unit 42 may interpolate a portion of the partial image 51 in which the overhead wire 31 is not captured from a portion in which the overhead wire 31 is captured. For example, as shown in FIG. 7 , in the partial image 51, a portion of the overhead wire 31 is hidden by a tree 40. In this case, the detection unit 42 may interpolate pixels corresponding to the portion of the overhead wire 31 that is hidden by the tree 40 from the portion in which the overhead wire 31 is captured in the partial image 51, and classify the pixels as the overhead wire 31. That is, in this example, the first portion 52a includes not only the portion in which the overhead wire 31 is actually captured, but also the portion of the overhead wire 31 that is hidden by the tree 40. Similarly, when a portion of the tree 40 is hidden by the overhead wire 30, the detection unit 42 may interpolate pixels corresponding to the portion of the tree 40 and classify the pixels as the tree 40. Note that the detection unit 42 may input the captured image 50 to the first trained model without generating the partial image 51.

[0037] In step S14 (information acquisition step), the information acquisition unit 43 acquires object information related to the overhead wire 30 detected by the detection unit 42. In this example, the object information is information indicating the degree of overlap between the overhead wire 30 and the tree 40. The information acquisition unit 43 acquires, as the object information, the number of pixels included in both the first portion 52a and the second portion 53 in the captured image 50 (partial image 51), and the number of pixels included in both the first portion 52b and the second portion 53. For example, the information acquisition unit 43 determines whether each pixel in the captured image 50 is included in the first portion 52a and the second portion 53, and acquires the number M1 of pixels determined to be included in both the first portion 52a and the second portion 53. Similarly, the information acquisition unit 43 acquires the number M2 of pixels determined to be included in both the first portion 52b and the second portion 53.

[0038] In step S13, the detection unit 42 interpolates pixels corresponding to parts of the overhead wires 31 hidden by the trees 40, but this interpolation may be performed by the information acquisition unit 43 in step S14. In this case, in step S14, the information acquisition unit 43 acquires the number of interpolated pixels as object information.

[0039] In step S15 (determination step), the determination unit 44 determines whether the overhead wire 31 and the tree 40 are close to each other based on the number of pixels M1, and determines whether the overhead wire 32 and the tree 40 are close to each other based on the number of pixels M2. For example, if the number of pixels M1 exceeds a threshold, the determination unit 44 determines that the overhead wire 31 and the tree 40 are close to each other, and if the number of pixels M2 exceeds a threshold, the determination unit 44 determines that the overhead wire 32 and the tree 40 are close to each other. The threshold is, for example, the number of pixels input by the user.

[0040] 8, for the first portion 52a, pixels of a portion 52c of the first portion 52a are included in the second portion 53. If the number of pixels of the portion 52c (number of pixels M1) exceeds a threshold, the determination unit 44 determines that the overhead wire 31 and the tree 40 are close to each other. On the other hand, for the first portion 52b, the number of pixels M2 is 0. Therefore, the determination unit 44 determines that the overhead wire 32 and the tree 40 are not close to each other.

[0041] In step S16, the determination unit 44 outputs the determination result. The determination unit 44 may display the processing result on the display device 6, may store the processing result in a predetermined storage device such as a memory or a database, or may transmit the processing result to another computer system.

[0042] [Learning method in the first embodiment] Next, an example of a training method for the first trained model used in the determination method according to the first embodiment will be described. Fig. 9 is a flowchart showing an example of the training method in the determination system 1 as flow S2. Flow S2 corresponds to the training phase.

[0043] In step S21, the detection unit 42 acquires an image showing an overhead line and an image showing a tree from the database 5. These images are, for example, images captured by the upward-facing camera 21a. In step S22, the detection unit 42 cuts out a partial image from the acquired image. For example, the detection unit 42 cuts out the partial image using a method similar to that used in step S12. In step S23, the detection unit 42 adds annotation information indicating an overhead line or a tree to the partial image, and stores the partial image and the annotation information as training data in a predetermined storage device. This annotation information may be information stored in the database 5 or information input by a user. The detection unit 42 may increase the training data by adjusting the brightness of the partial image. The detection unit 42 may use the captured image from the imaging device 2 as training data.

[0044] In step S24, the learning unit 41 performs training of the learning model using the teacher data. The learning unit 41 adjusts the parameters of the learning model until a predetermined termination condition is met. In step S25, the learning unit 41 trains the learning model at the time when the training is completed as a first trained model for performing segmentation of overhead wires and trees. The learning unit 41 stores the first trained model in a predetermined storage device. This first trained model is used in step S13.

[0045] [Action and effect] According to the above-described determination system 1 and determination method, the imaging unit 21 mounted on a railway vehicle traveling on rail R1 captures images of the surroundings of the railway vehicle, thereby enabling efficient imaging of objects. The information acquisition unit 43 then acquires object information about the object detected by the detection unit 42, and the determination unit 44 determines whether or not there is an abnormality in the object based on the object information. This makes it possible to determine with high accuracy whether or not there is an abnormality in the object from the captured image. As a result, maintenance work on objects in railway facilities can be performed efficiently.

[0046] In the determination system 1 and the determination method, the imaging unit 21 captures an image of the overhead wire 30 located above the rail R1 and the tree 40 located around the overhead wire 30, the detection unit 42 detects a first portion 52 corresponding to the overhead wire 30 and a second portion 53 corresponding to the tree 40 in the captured image 50, the information acquisition unit 43 acquires the number of pixels included in both the first portion 52 and the second portion 53 in the captured image 50 as object information, and the determination unit 44 determines whether the overhead wire 30 and the tree 40 are in close proximity based on the number of pixels. By the information acquisition unit 43 acquiring the number of pixels included in both the first portion 52 corresponding to the overhead wire 30 and the second portion 53 corresponding to the tree 40, the determination unit 44 can accurately determine whether the overhead wire 30 and the tree 40 are in close proximity based on the number of pixels.

[0047] In the determination system 1 and the determination method, the imaging unit 21 has an upward-facing camera 21a that captures an image diagonally upward relative to the horizontal direction, and the detection unit 42 detects a first portion 52 and a second portion 53 from a captured image 50 captured by the upward-facing camera 21a. For example, when the imaging unit 21 captures an image of the area directly ahead, the overhead wire 30 on the near side and the tree 40 on the far side in the captured image 50 may appear to overlap in the captured image 50 even if they are not actually close to each other, which may make it difficult to make an accurate determination. However, by using the image captured by the upward-facing camera 21a, the influence of such perspective can be reduced, and it is possible to more accurately determine whether the overhead wire 30 and the tree 40 are close to each other.

[0048] In the determination system 1 and the determination method, the detection unit 42 cuts out the upper half of the captured image 50 as a partial image 51, and detects a first portion 52 and a second portion 53 from the partial image 51. This makes it possible to further reduce the influence of perspective as described above.

[0049] [Determination method according to the second embodiment] Next, a determination method according to a second embodiment using the determination system 1 will be described. FIG. 10 is a diagram schematically illustrating the positional relationship between the imaging device 2 and an object. In this example, the object is a reinforcement, and the determination system 1 determines whether the height of the reinforcement 60 is low. If the reinforcement 60 is determined to be low, the user adjusts the height of the reinforcement 60. For example, the user adjusts the height of the reinforcement 60 by adding ballast to the reinforcement 60 or by replacing the reinforcement 60. The reinforcement 60 is located on the outside of the rail R1 and is formed by raising the trackbed. The reinforcement 60 is formed, for example, to improve the lateral resistance of the trackbed. In this example, the imaging device 2 uses a downward-facing camera 21c as the imaging unit 21. The downward-facing camera 21c captures images at an angle of 45 degrees diagonally downward relative to the horizontal, for example.

[0050] Fig. 11 is a flowchart showing, as flow S3, an example of a determination method in the determination system 1. Fig. 12 is a diagram showing an example of a method of processing a captured image by the determination system 1.

[0051] In step S31 (imaging step), the imaging unit 21 captures an image of the area around the railway vehicle and outputs an image 71 that captures the excess reinforcement 60 and an image 72 that captures an area other than the excess reinforcement 60 (in this example, concrete C located outside the rail R1). The imaging device 2 transmits the imaged images 71 and 72 to the processing server 4. The imaging unit 21 may transmit only the imaged image 71 to the processing server 4.

[0052] In step S32, the detection unit 42 cuts out a partial image 73 that shows the excess reinforcement 60 from the captured image 71 as part of the captured image 71. The detection unit 42 cuts out a partial image 74 that shows the concrete C from the captured image 72 as part of the captured image 72. The range of the partial images 73, 74 is set based on the angle of the downward-facing camera 21c. The detection unit 42 cuts out, for example, the lower half of the captured image 71 as the partial image 73. The detection unit 42 cuts out the lower half of the captured image 72 as the partial image 74. Alternatively, if the horizontal and vertical lengths of the captured image 71 are set to 100%, the detection unit 42 may generate the partial image 73 by excluding the portion from the top 50% of the captured image 71, the portion from the left 5% of the captured image 71, and the portion from the right 70% of the captured image 71. In this case, the partial image 73 is an image located in the lower half of the captured image 71, in the leftmost region, and is an image focusing on the reinforcement 60 located on the left side of the rail R1. Alternatively, the detection unit 42 may generate the partial image 73 by excluding the portion from the top 50% of the captured image 71, the portion from the left 70% of the captured image 71, and the portion from the right 5% of the captured image 71. In this case, the partial image 73 is an image located in the lower half of the captured image 71, in the rightmost region, and is an image focusing on the reinforcement 60 located on the right side of the rail R1. The partial image 74 may also be cut out from the captured image 72 in the same manner as the partial image 73. The ranges of the partial images 73 and 74 may be changed as appropriate depending on the position of the reinforcement 60 or the concrete C.

[0053] In step S33 (detection step), the detection unit 42 inputs the partial images 73 and 74 into the first trained model and detects a portion 75 in the partial image 73 that corresponds to the excess metal 60. In this example, the first trained model is used to detect the excess metal 60. The first trained model classifies each pixel in the partial images 73 and 74 as either the excess metal 60 or other background. The set of pixels classified as the excess metal 60 corresponds to the portion 75. In this example, two portions 75 are detected corresponding to the two excess metal 60a and 60b contained in the excess metal 60. The detection unit 42 classifies one of the two portions 75 as portion 75a corresponding to the excess metal 60a, and classifies the other of the two portions 75 as portion 75b corresponding to the excess metal 60b. On the other hand, since the excess metal 60 is not detected in the partial image 74, the partial image 74 is not used in steps S34 and thereafter. That is, the detection unit 42 selects an image showing the excess metal 60 to be used in step S34 and thereafter from a group of images including an image showing the excess metal 60 and an image not showing the excess metal 60. The detection unit 42 may input the captured images 71 and 72 to the first trained model without generating the partial images 73 and 74.

[0054] In step S34 (information acquisition step), the information acquisition unit 43 acquires object information regarding the excess reinforcement 60 detected by the detection unit 42. In this example, the object information is height information of the excess reinforcement 60. The height information of the excess reinforcement 60 may be, for example, a quantitative indicator indicating the height to the apex of the excess reinforcement 60 relative to a reference surface (e.g., the top surface of the sleeper R2), or a qualitative indicator such as "high" or "low." The information acquisition unit 43 inputs the partial image 73 in which the excess reinforcement 60 is detected into the second trained model to acquire the height information. The information acquisition unit 43 may acquire depth information for the portion 75 from the second trained model along with the height information. The depth information is information indicating, for each pixel in the image, how far it is from the camera in actual three-dimensional space.

[0055] Alternatively, in step S34, the information acquisition unit 43 may acquire depth information for the portion 75 that captures the excess metal 60, and acquire height information for the excess metal 60 based on this depth information. FIG. 13 is a diagram illustrating an image of a method for acquiring depth information. When the imaging unit 21 is a monocular camera, the information acquisition unit 43 acquires depth information using monocular depth estimation. Monocular depth estimation is performed, for example, by machine learning using a learning model such as CNN. For training this machine learning, for example, captured images with known depth information are used as training data, and when a captured image is input, learning is performed to estimate the depth information. When the imaging unit 21 is a stereo camera or a TOF camera, the information acquisition unit 43 acquires depth information based on information from the imaging unit 21.

[0056] The information acquisition unit 43 then acquires height information of the excess metal deposit 60 based on the acquired depth information. For example, the information acquisition unit 43 acquires height information based on the degree of change in the depth information of the pixels included in the portion 75. The information acquisition unit 43 may determine that the height of the excess metal deposit 60 is low if the change in the depth information within the portion 75 is rapid, and may determine that the height of the excess metal deposit 60 is high if the change is gradual.

[0057] In step S35 (judgment step), the judgment unit 44 judges whether the excess fill 60 is low or not based on the height information of the excess fill 60. For example, the judgment unit 44 judges that the excess fill 60 is low when the height information, which is a quantitative indicator, is below a threshold. The judgment unit 44 judges that the excess fill 60 is low when the height information, which is a qualitative indicator, corresponds to a predetermined indicator. The threshold and the predetermined indicator are input, for example, by the user. In this example, the excess fill 60a is judged to be low, and the excess fill 60b is judged to be high.

[0058] In step S36, the determination unit 44 outputs the determination result. The determination unit 44 may display the processing result on the display device 6, may store the processing result in a predetermined storage device such as a memory or a database, or may transmit the processing result to another computer system.

[0059] [Learning method in the second embodiment] Next, a description will be given of another example of the training method of the first trained model used in the determination method according to the second embodiment. Fig. 14 is a flowchart showing an example of the training method in the determination system 1 as flow S4. Flow S4 corresponds to the training phase.

[0060] In step S41, the detection unit 42 acquires images showing excess reinforcement from the database 5. These images are, for example, images captured by the downward-facing camera 21c. In step S42, the detection unit 42 cuts out partial images from the acquired images. For example, the detection unit 42 cuts out the partial images using a method similar to that used in step S32. In step S43, the detection unit 42 adds annotation information indicating excess reinforcement to the partial images, and stores the partial images and the annotation information as training data in a predetermined storage device. This annotation information may be information stored in the database 5 or information input by a user. The detection unit 42 may increase the training data by adjusting the brightness of the partial images. The detection unit 42 may use images captured by the imaging device 2 as training data.

[0061] In step S44, the learning unit 41 trains the learning model using the teacher data. The learning unit 41 adjusts the parameters of the learning model until a predetermined termination condition is met. In step S45, the learning unit 41 trains the learning model at the time when the training is completed as a first trained model for detecting excess stock. The learning unit 41 stores the first trained model in a predetermined storage device. This first trained model is used in step S33.

[0062] Next, an example of a training method for the second trained model used in the above-described determination method will be described. Fig. 15 is a flowchart showing an example of the training method in the determination system 1 as flow S5. Flow S5 corresponds to the training phase.

[0063] In step S51, the detection unit 42 acquires images of excess reinforcement from the database 5. These images are, for example, images captured by the downward camera 21c. In step S52, the detection unit 42 assigns height information and depth information of the excess reinforcement to the acquired images, and stores the images and the height information and depth information as training data in a predetermined storage device. This height information and depth information may be information stored in the database 5 or may be information input by the user. The detection unit 42 may use images captured by the imaging device 2 as training data.

[0064] In step S53, the learning unit 41 trains the learning model using the teacher data. The learning unit 41 adjusts the parameters of the learning model until a predetermined termination condition is met. In step S54, the learning unit 41 trains the learning model at the time when the training is completed as a second trained model for estimating the height information and depth information of the excess fill. The learning unit 41 stores the second trained model in a predetermined storage device. This second trained model is used in step S34. Note that the learning unit 41 may train the second trained model for estimating the height information without using the depth information as teacher data.

[0065] [Action and effect] In the determination system 1 and determination method described above, the rail R1 is provided on the track bed, the imaging unit 21 is located outside the rail R1 and captures an image of the excess reinforcement 60 formed by the bulging track bed, the detection unit 42 detects a portion 75 in the captured image 71 that captures the excess reinforcement 60, the information acquisition unit 43 acquires height information relating to the height of the excess reinforcement 60, and the determination unit 44 determines whether the excess reinforcement 60 is low or not based on the height information. By the information acquisition unit 43 acquiring the height information relating to the height of the excess reinforcement 60, the determination unit 44 can more accurately determine whether the excess reinforcement 60 is low or not based on the height information.

[0066] In the determination system 1 and the determination method, the information acquisition unit 43 inputs the captured image 71 in which the excess metal 60 is detected into the second trained model to acquire height information. This makes it possible to acquire height information about the excess metal 60 accurately and easily.

[0067] In the determination system 1 and the determination method, the information acquisition unit 43 may acquire depth information for the portion 75 that shows the excess metal 60, and acquire height information based on the depth information. In this case, the height information of the excess metal 60 can be acquired with high accuracy based on the depth information.

[0068] Although various embodiments of the present invention have been described above, the embodiments of the present invention are not limited to the above-described embodiments. [Explanation of symbols]

[0069] 1...determination system, 21...imaging unit, 21a...upward-facing camera, 30, 31, 32...overhead wire, 40...tree, 42...detection unit, 43...information acquisition unit, 44...determination unit, 50...captured image, 52, 52a, 52b...first part, 53...second part, 60, 60a, 60b...excess reinforcement, 71...captured image, 75, 75a, 75b...part, R1...rail.

Claims

1. an imaging unit mounted on a railway vehicle running on rails, capturing an image of the surroundings of the running railway vehicle, and outputting a captured image of an object; a detection unit that inputs the captured image into a first trained model and detects at least the object from the captured image; an information acquisition unit that acquires object information related to the object detected by the detection unit; a determination unit that determines whether or not there is an abnormality in the object based on the object information, the imaging unit images an overhead line as the object located above the rail and trees located around the overhead line; the detection unit detects a first portion corresponding to the overhead wire and a second portion corresponding to the tree in the captured image; the information acquisition unit acquires, as the object information, the number of pixels included in both the first portion and the second portion in the captured image; the determination unit determines whether the overhead line and the tree are close to each other based on the number of pixels. Judging system.

2. the imaging unit has an upward-facing camera that captures an image obliquely upward with respect to the horizontal direction, the detection unit detects the first portion and the second portion from the captured image captured by the upward camera. The determination system according to claim 1 .

3. an imaging unit mounted on a railway vehicle running on rails, capturing an image of the surroundings of the running railway vehicle, and outputting a captured image of an object; a detection unit that inputs the captured image into a first trained model and detects at least the object from the captured image; an information acquisition unit that acquires object information related to the object detected by the detection unit; a determination unit that determines whether or not there is an abnormality in the object based on the object information, The rail is provided on a track bed, The imaging unit is located outside the rail and captures an image of a reinforcement formed by a rise in the ballast as the target object, The detection unit detects a portion of the captured image that shows the excess deposit, The information acquisition unit inputs the captured image in which the excess buildup is detected into a second trained model as the object information to acquire height information regarding the height of the excess buildup, The determination unit determines whether the excess fill is low or not based on the height information. Judging system.

4. The information acquisition unit acquires depth information for the portion that captures the excess buildup, and acquires the height information based on the depth information. The determination system according to claim 3 .

5. A determination method executed by a determination system, an imaging step of capturing an image of the surroundings of a railway vehicle traveling on a rail and outputting a captured image showing an object; a detection step of inputting the captured image into a first trained model and detecting at least the object from the captured image; an information acquisition step of acquiring object information regarding the object detected by the detection step; a determination step of determining whether or not there is an abnormality in the object based on the object information, In the imaging step, an overhead line as the object located above the rail and a tree located around the overhead line are imaged; In the detecting step, a first portion corresponding to the overhead wire and a second portion corresponding to the tree are detected in the captured image; In the information acquiring step, the number of pixels included in both the first portion and the second portion in the captured image is acquired as the object information; In the determining step, it is determined whether or not the overhead line and the tree are close to each other based on the number of pixels. Judgment method.

6. A determination method executed by a determination system, an imaging step of capturing an image of the surroundings of a railway vehicle traveling on a rail and outputting a captured image showing an object; a detection step of inputting the captured image into a first trained model and detecting at least the object from the captured image; an information acquisition step of acquiring object information regarding the object detected by the detection step; a determination step of determining whether or not there is an abnormality in the object based on the object information, The rail is provided on a track bed, In the imaging step, an image of a reinforcement formed by a rise in the ballast and located outside the rail is taken as the object, In the detection step, a portion of the captured image that shows the excess metal is detected, In the information acquisition step, the captured image in which the excess buildup is detected is input into a second trained model as the object information to acquire height information regarding the height of the excess buildup; In the determination step, it is determined whether the excess fill is low or not based on the height information. Judgment method.

Images