Detection system and detection method

Abstract

To detect all obstacles around railway tracks without fail. [Solution] The detection system 1 is mounted on a railway vehicle running on rails R and comprises an imaging unit 21 that outputs an image of rails R, and a detection unit 43 that inputs the image into a trained model to detect the presence or absence of obstacles OB around rails R. The trained model is a machine learning model that has been trained using embedded images as training data, which are generated by embedding images of obstacles randomly selected from a predetermined group of obstacles into the image output by the imaging unit 21 at randomly determined positions around rails R in the image.

Description

Technical Field

[0004] ,

[0006] , , , ,

[0005] , , , , ,

[0001] The present disclosure relates to a detection system and a detection method.

Background Art

[0002] Patent Document 1 describes an abnormality detection device for a railway line, which includes an image reading unit that reads an image of the railway line, a storage unit that stores the read image, a comparison unit that compares the inspection image read by the image reading unit with a reference image stored in advance in the storage unit, and an output unit that identifies and outputs an abnormal location of the railway line based on the comparison result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] The detection method of the present disclosure is [6] "a detection method performed by a detection system, comprising: an imaging step of outputting an image of the rails using an imaging unit mounted on a railway vehicle running on rails; and a detection step of inputting the image to a trained model to detect the presence or absence of obstacles around the rails, wherein the trained model is a machine learning model trained using image data generated by embedding images of obstacles randomly selected from a predetermined group of obstacles into the image of the rails at randomly determined positions around the rails in the image, which has been previously output by the imaging unit, as training data."

[0008] This detection system or method efficiently captures images of the area around the rails using an imaging unit mounted on a railway vehicle traveling on the rails. The captured images are then input into a trained model to detect the presence or absence of obstacles around the rails. The trained model used in this process is trained using images of randomly selected obstacles embedded at random locations around the rails as training data. Therefore, the model is capable of detecting situations where multiple types of obstacles exist at various locations around the rails. This ensures that all obstacles around the rails are detected without fail.

[0009] The detection system of the present disclosure may also be [2] "the detection system according to [1] above, further comprising a learning unit that learns the machine learning model using the training data and constructs the trained model." The detection method of the present disclosure may also be [7] "the detection method according to [6] above, further comprising a learning step that learns the machine learning model using the training data and constructs the trained model." In this case, a trained model can be constructed that can detect situations in which multiple types of obstacles are present at various positions around the rail.

[0010] The detection system of the present disclosure may also be [3] "the detection system according to [2] above, further comprising a storage unit that stores in advance a plurality of combination information including the type of obstacle and information relating to the shape of the obstacle, wherein the learning unit determines the obstacle and the shape of the obstacle by randomly selecting and referring to the plurality of combination information stored in the storage unit, randomly determines the embedding position around the rail in the captured image, and generates the training data by embedding an image corresponding to the determined obstacle and the shape at the embedding position in the captured image." The detection method of the present disclosure may also be [8] "the detection method according to [7] above, wherein in the learning step, the obstacle and the shape of the obstacle are determined by randomly selecting a plurality of combination information including the type of obstacle and information relating to the shape of the obstacle, randomly determines the embedding position around the rail in the captured image, and generates the training data by embedding an image corresponding to the determined obstacle and the shape at the embedding position in the captured image." In this case, by pre-setting combinations of obstacle types and shapes that are realistically conceivable, it is possible to generate images embedded with images of obstacles that could actually exist as training data. In addition, by randomly determining the embedding position of the obstacle images in those images based on the position of the rails, it is possible to generate images with obstacle images embedded in positions that could actually exist as training data. As a result, it is possible to efficiently build a trained model that can detect situations where multiple types of obstacles exist at various positions around the rails.

[0011] The detection system of the present disclosure may also be [4] "the detection system according to [3] above, wherein the learning unit specifies the embedding position to the generating AI and inputs prompts indicating the determined obstacle and shape, thereby generating the training data using the generating AI." The detection method of the present disclosure may also be [9] "the detection method according to [8] above, wherein in the learning step, the learning unit specifies the embedding position to the generating AI and inputs prompts indicating the determined obstacle and shape, thereby generating the training data using the generating AI." In this case, by using the generating AI, it is not necessary to prepare images of multiple obstacles in advance, and the embedding position of the obstacle images can be easily specified. As a result, images in which images of obstacles that may actually exist are embedded in various embedding positions can be easily generated as training data.

[0012] The detection system of the present disclosure may also be [5] "the detection system according to any one of [1] to [4] above, wherein the imaging unit has a downward-facing camera that images diagonally downward with respect to the horizontal direction, and the detection unit detects the presence or absence of the obstacle based on the image captured by the downward-facing camera." For example, when the imaging unit images the area directly in front, the area around the rail in the captured image will be captured at the bottom of the image, making it difficult to accurately determine the situation around the rail. However, by using an image captured by a downward-facing camera, an image that captures the area around the rail in front can be obtained, and the presence or absence of an obstacle can be determined more accurately based on that image. [Effects of the Invention]

[0013] According to this disclosure, it is possible to detect the occurrence of obstacles around railway tracks without fail. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 shows the configuration of the detection system. [Figure 2] Figure 2 is a block diagram showing the configuration of the imaging device. [Figure 3] FIG. 3 is a block diagram showing a general hardware configuration of the processing server. [Figure 4] FIG. 4 is a block diagram showing the configuration of the processing server. [Figure 5] FIG. 5 is a diagram schematically showing the arrangement relationship between the imaging device and the object in the embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of a detection method in the detection system. [Figure 7] FIG. 7 is a diagram showing a data example of the obstacle list stored in the processing server. [Figure 8] FIG. 8 is a diagram showing an example of an imaging image to be processed by the detection system. [Figure 9] FIG. 9 is a diagram showing an example of an embedded image which is the processing result of the detection system. [Figure 10] FIG. 10 is a flowchart showing an example of a learning method of the learned model.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, a preferred embodiment of the detection system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0016] [Configuration of Detection System]

[0017] First, referring to FIG. 1, the overall configuration of the detection system 1 according to the embodiment will be described. FIG.  1 is a diagram showing the configuration of the detection system 1. The detection system 1 is a system for detecting the occurrence of obstacles around the rails of a railway line. The railway line is composed of, for example, a pair of rails R and sleepers T (see FIG. 5) provided on the roadbed. The obstacle OB around the rail R is an object that can, for example, hinder the running of railway vehicles such as cans and garbage. In this example, the detection system 1 performs the above detection from an image capturing the surroundings of the rail R including the rail R.

[0018] The detection system 1 is composed of an imaging device 2, an intermediate server 3, a processing server 4, a database 5, and a display device 6. The imaging device 2 is connected to the intermediate server 3 via a network N. The intermediate server 3 is connected to the processing server 4 via the network N. Images from the imaging device 2 are transmitted to the processing server 4 via the intermediate server 3. The number of intermediate servers 3 may be any number of 1 or more. The processing server 4 processes the images from the imaging device 2. The processing server 4 is connected to the database 5 and the display device 6 via the network N. Note that the images from the imaging device 2 may be transmitted to the processing server 4 without passing through the intermediate server 3. Alternatively, the images from the imaging device 2 may be transmitted to the database 5 without passing through both the intermediate server 3 and the processing server 4. In this case, the processing server 4 acquires images from the database 5 and processes the images. The images processed by the processing server 4 are stored in the database 5. The display device 6 outputs the processing results of the processing server 4. The processing server 4 may be connected to the database 5 and the display device 6 by a local cable. The imaging device 2 is mounted on a railway vehicle. On the other hand, the intermediate server 3, the processing server 4, the database 5, and the display device 6 may not be mounted on the railway vehicle and may be arranged, for example, in facilities (such as stations) along the railway line.

[0019] The imaging device 2 images the surroundings of a rail R including the rail R. The railway vehicle is a vehicle that runs on the rail R and is, for example, a locomotive or a train. The imaging device 2 is mounted, for example, on the leading vehicle of the railway vehicle. As shown in FIG. 2, the imaging device 2 includes an imaging unit 21, a computer 22, a sensor 23, a communication device 24, an input device 25, a display device 26, and a power supply 27. FIG. 2 is a block diagram showing the configuration of the imaging device 2. Operating conditions are input to the above devices by a user to the input device 25. The display device 26 outputs the image captured by the imaging unit 21. The power supply 27 supplies power to the devices constituting the imaging device 2.

[0020] The imaging unit 21 includes an upward-facing camera 21a, a front-facing camera 21b, and a downward-facing camera 21c. Cameras 21a to 21c are, for example, monocular cameras, stereo cameras, or TOF (Time of Flight) cameras. If cameras 21a to 21c are monocular cameras, the portability of the imaging device 2 and the ease of introducing the imaging device 2 into a railway vehicle are improved. Each of cameras 21a to 21c includes an imaging lens 21d, a control unit 21e, and a status display unit 21f.

[0021] The upward-facing camera 21a captures an image diagonally upward relative to the horizontal direction. The upward-facing camera 21a is mounted on the railway vehicle so that its imaging lens 21d is angled diagonally upward relative to the horizontal direction. The front camera 21b captures an image directly in front of the railway vehicle in the direction of travel. The front camera 21b is mounted on the railway vehicle so that its imaging lens 21d is angled horizontally. The downward-facing camera 21c captures an image diagonally downward relative to the horizontal direction. The downward-facing camera 21c is mounted on the railway vehicle so that its imaging lens 21d is angled diagonally downward relative to the horizontal direction. By using cameras 21a to 21c appropriate to the position of the object to be imaged, the object can be imaged effectively.

[0022] Cameras 21a to 21c output multiple captured images to the computer 22, including images of the rail R and its surroundings. The multiple captured images may include images that do not show the rail R, in addition to images that show the rail R and its surroundings. Cameras 21a to 21c may output only images that show the rail R, and not images that do not show the rail R. Each of the multiple captured images may be a video or a still image. The frame rate of each of the cameras 21a to 21c is, for example, 5fps to 60fps, and one example is 10fps. The shutter speed of each of the cameras 21a to 21c is, for example, 1ms to 5ms, and one example is 2ms. The frame rate and shutter speed may be appropriately adjusted by the control unit 21e according to the running speed of the railway vehicle. The imaging unit 21 only needs to include at least one of the cameras 21a to 21c.

[0023] Computer 22 is connected to the imaging unit 21, sensor 23, communication equipment 24, input device 25, and display device 26, and controls these devices. Computer 22 also stores the images captured by the imaging unit 21. Sensor 23 acquires the position information or acceleration information of the railway vehicle and outputs it to computer 22. Sensor 23 is, for example, a GPS sensor or an acceleration sensor. Computer 22 transmits the captured images, as well as the position information or acceleration information of the railway vehicle, to the intermediate server 3 via the communication equipment 24.

[0024] Figure 3 is a block diagram showing the general hardware configuration of the processing server 4. The processing server 4 includes a CPU (processor) 101 that runs the operating system and application programs, a main memory unit 102 consisting of ROM and RAM, an auxiliary memory unit 103 consisting of a hard disk or flash memory, a communication control unit 104 consisting of a network card or wireless communication module, an input device 105 such as a keyboard or mouse, and an output device 106 such as a display or printer.

[0025] Each functional element of the processing server 4, described later, is realized by loading predetermined software onto the CPU 101 or main memory 102, operating the communication control unit 104, input device 105, output device 106, display device 6, etc., under the control of the CPU 101, and reading and writing data to the main memory 102 or auxiliary memory 103. The data and database necessary for processing are stored in the main memory 102 or auxiliary memory 103.

[0026] When video is transmitted from the imaging device 2, the processing server 4 divides the video into frames and generates still images. The processing server 4 adds position information and acceleration information of the railway vehicle to the still images and stores the still images and this information in a predetermined storage device. The processing server 4 performs image processing on the generated still images.

[0027] Figure 4 is a block diagram showing the configuration of the processing server 4. The processing server 4 comprises a learning unit 41, a generation unit 42, a detection unit 43, and a table storage unit 44 as functional components.

[0028] The learning unit 41 trains a learning model using machine learning with training data to learn (build) a pre-trained model. The learning model is a machine learning model such as a convolutional neural network (CNN). The generation unit 42 converts the number of pixels in the captured image to generate a converted image, which is the target of detection by the detection unit 43. The detection unit 43 inputs the converted image into the pre-trained model and detects the presence or absence of obstacles around the rail R based on the output of a weight calculation performed on the converted image in the pre-trained model.

[0029] [Detection method]

[0030] Next, a detection method according to an embodiment of the detection system 1 will be described. Figure 5 is a schematic diagram showing the arrangement of the imaging device 2 and a pair of rails R. In this example, the imaging device 2 uses a downward-facing camera 21c as the imaging unit 21. The downward-facing camera 21c takes images at an angle of 45 degrees diagonally downward with respect to the horizontal direction, for example.

[0031] Figure 6 is a flowchart showing an example of the detection method in detection system 1, designated as flow S1.

[0032] In step S11 (imaging step), the imaging unit 21 outputs an image of the rail R and its surroundings. The imaging device 2 transmits the image to the processing server 4. For example, the image shows the rail R, the sleepers T, and obstacles OB (see Figure 5) located around the rail R. The number of obstacles OB around the rail R can be any number of one or more.

[0033] In step S12, the generation unit 42 generates a converted image by performing conversions such as pixel count conversion on the captured image. This conversion is performed so that the image conditions, such as the pixel count, in the converted image match the image conditions of the captured image and embedded image included in the training data used by the learning unit 41, which will be described later.

[0034] In step S13 (detection step), the detection unit 43 inputs the transformed image into a trained model to detect the presence or absence of obstacles (OB) around the rail R. Specifically, by inputting the transformed image into the trained model, the detection unit 43 performs weighting calculations on the transformed image. The detection unit 43 then detects the presence or absence of obstacles (OB) based on the output data resulting from these weighting calculations. The trained model is designed to detect the presence or absence of obstacles (OB).

[0035] In step S14, the detection unit 43 outputs the detection result. The detection unit 43 may display the processing result on the display device 6, store the processing result in a predetermined storage device such as memory or a database, or transmit the processing result to another computer system.

[0036] [Learning Methods]

[0037] Next, an example of a training method for the trained model used in the detection method according to the above embodiment will be described. Figure 7 is an example of data for the obstacle list stored in the table storage unit 44 used in the training method, Figure 8 is an example of an image captured and processed by the training method, and Figure 9 is an example of an embedded image which is the processing result of the detection method. Figure 10 is a flowchart showing an example of a training method for the trained model as flow S2. Flow S2 corresponds to the training step.

[0038] In step S21, the learning unit 41 acquires captured images of the rail R and its surroundings from the database 5. These captured images are generally images that do not show obstacles OB. The learning unit 41 then augments the captured images by adjusting the brightness (luminance) of the captured images in various ways (step S22). Then, the following processes in steps S23 to S26 are executed on the multiple captured images.

[0039] In step S23, an embedding range A1 (see Figure 8) is set, which includes the rail R and its surroundings in the captured image G1 to be processed. This embedding range A1 is set in response to input from the user via an input device 105 such as a mouse on the processing server 4. Alternatively, the embedding range A1 may be set by the processing server 4 using a different trained model than the trained model that is the target of processing in flow S2 to detect the position of the rail R in the captured image G1 through segmentation, and then automatically setting a range including the surroundings of the rail R according to that position.

[0040] Next, in step S24, the learning unit 41 randomly determines the embedding positions of obstacle images around the rail R within the embedding range A1 of the captured image G1 set in step S23. For example, the embedding positions are determined using two-dimensional coordinates on the image. Then, in step S25, the learning unit 41 randomly selects and refers to obstacle information corresponding to the obstacle image to be embedded at the embedding position of the captured image G1 from among multiple obstacle information (combination information) contained in the obstacle list stored in the table storage unit 44, thereby determining the type, shape, and color of the obstacle to be embedded.

[0041] As shown in Figure 7, the obstacle information included in the obstacle list contains information about the type of obstacle, the shape of the obstacle, and the color of the obstacle. For example, one piece of obstacle information consists of text data listing "garbage" to indicate the type of obstacle, "box-shaped" to indicate the shape of the obstacle, and "white" to indicate the color of the obstacle. When an example obstacle list like the one shown in Figure 7 is referenced, the learning unit 41 selects, for example, the obstacle information "garbage, box-shaped, gray" from among the multiple obstacle information items included in the obstacle list. As a result, a box-shaped gray piece of garbage is determined to be the obstacle to be embedded in the embedding position of the captured image G1.

[0042] Subsequently, in step S26, the learning unit 41 calls a pre-stored generation AI and inputs a prompt to the generation AI instructing it to embed the captured image G1 and an obstacle image corresponding to the type, shape, and color included in the selected obstacle information into the captured image G1. A possible generation AI used at this time is Stable Diffusion. When inputting a prompt to the generation AI, the learning unit 41 also inputs information specifying the embedding position of the obstacle image to the generation AI. At this time, it is preferable for the learning unit 41 to automatically add an English word such as "trash" in addition to a Japanese word such as "garbage" that identifies the obstacle in the input prompt. This can improve the accuracy of the generated image. Based on the input of the captured image and the prompt, the learning unit 41 obtains an embedded image in which the obstacle image is embedded at the embedding position in the captured image G1, based on the output of the generation AI. The learning unit 41 repeats the above process from steps S24 to S26 for one captured image G1, for the number of embedded images required as training data.

[0043] Figure 9 shows an example of an embedded image acquired by the learning unit 41 using the processing shown in steps S21 to S26. In this way, the learning unit 41 acquires embedded images G2 to G7 in which images of various types of obstacles OB are embedded at random positions around the rail R. For example, as shown in embedded images G2 and G3, images of the same type of obstacle OB placed at different positions are acquired; as shown in embedded images G4, G5, and G7, images of obstacles OB with different shapes and colors are embedded are acquired; as shown in embedded image G6, images of multiple obstacles OB are embedded are acquired, and so on.

[0044] Next, in step S27, the learning unit 41 adds annotation information indicating the presence or absence of obstacles to multiple embedded images obtained from the generating AI and multiple captured images obtained in step S22, as training data, and sequentially stores the embedded images and captured images with the added annotation information in a predetermined storage device as training data. This annotation information may be information stored in the database 5 or information entered by the user.

[0045] In step S28, the learning unit 41 trains the learning model using the training data stored in the memory. The learning unit 41 adjusts the parameters of the learning model until a predetermined termination condition is met. The learning unit 41 stores the trained model in a predetermined memory. This trained model is used in step S13. [Mechanism of Action and Effects]

[0046] According to the detection system 1 and detection method described above, the imaging device 2 mounted on a railway vehicle traveling on the rail R can efficiently image the area around the rail R. By inputting the captured images into a trained model, the presence or absence of obstacles (OB) around the rail R can be detected. The trained model used at this time is trained using embedded images, which are randomly selected images of obstacles (OB) from a group of obstacles embedded in the captured images at random positions around the rail R, as training data. Therefore, the model is constructed to detect situations where multiple types of obstacles (OB) exist at various positions around the rail R. As a result, the occurrence of obstacles (OB) around the rail R can be detected without fail.

[0047] The detection system 1 of this disclosure further includes a learning unit 41 that trains a machine learning model using training data to construct a trained model. In this case, a trained model can be constructed that can detect situations in which multiple types of obstacles OB are present at various positions around the rail R.

[0048] The detection system 1 of this disclosure further includes a table storage unit 44 that pre-stores multiple pieces of obstacle information, including information on the type and shape of the obstacle. The learning unit 41 determines the obstacle and its shape by randomly selecting and referring to the multiple pieces of obstacle information stored in the table storage unit 44, randomly determines the embedding position around the rail R in the captured image, and generates training data by embedding images corresponding to the determined obstacle and shape at the embedding position in the captured image. In this case, if the combination of the obstacle type and its shape is pre-set as a combination that can be realistically assumed, it is possible to generate images with embedded images of obstacles that could actually exist as training data. In addition, by randomly determining the embedding position of the obstacle image in that image based on the position of the rail R, it is possible to generate images with obstacle images embedded at positions that could actually exist as training data. As a result, it is possible to efficiently construct a trained model that can detect situations in which multiple types of obstacles OB exist at various positions around the rail R.

[0049] In the detection system 1 of this disclosure, the learning unit 41 generates training data using the generating AI by specifying the embedding position and inputting prompts indicating the determined obstacle and shape to the generating AI. In this case, by using the generating AI, it is not necessary to prepare images of multiple obstacles in advance, and the embedding position of the obstacle images can be easily specified. As a result, images in which images of obstacles that may actually exist are embedded at various embedding positions can be easily generated as training data.

[0050] In the detection system 1 of this disclosure, the imaging device 2 has a downward-facing camera 21c that captures images diagonally downward with respect to the horizontal direction, and the detection unit 43 detects the presence or absence of an obstacle OB based on the image captured by the downward-facing camera 21c. For example, when the imaging device 2 captures images directly in front, the area around the rail R in the captured image is captured at the bottom of the image, which may make it difficult to accurately determine the situation around the rail R. However, by using the image captured by the downward-facing camera 21c, an image that captures the area around the rail R in front can be obtained, and the presence or absence of an obstacle OB can be determined with greater accuracy by making a determination based on that image.

[0051] Although embodiments of the present invention have been described above, the embodiments of the present invention are not limited to those described above.

[0052] For example, in the learning method described above, step S22 may be omitted. Also, in the determination method described above, step S12 may be omitted. Furthermore, steps S23 to S25 may be executed in a different order. [Explanation of Symbols]

[0053] 1...Detection system, 21...Imaging unit, 21c...Downward-facing camera, 41...Learning unit, 43...Detection unit, G1...Imagine captured image, G2~G7...Embedded images, R...Rail, OB...Obstacle.

Claims

1. An imaging unit mounted on a railway vehicle that runs on rails, which outputs an image of the rails, A detection unit inputs the captured image into a trained model to detect the presence or absence of obstacles around the rail, Equipped with, The aforementioned trained model is a machine learning model trained using image data generated by embedding images of obstacles randomly selected from a predetermined group of obstacles into the captured image output by the imaging unit at randomly determined positions around the rail in the captured image. Detection system.

2. The system further includes a learning unit that uses the aforementioned training data to train the machine learning model and construct the trained model. The detection system according to claim 1.

3. The system further includes a storage unit that pre-stores multiple combinations of information, including the type of obstacle and information regarding the shape of the obstacle. The aforementioned learning unit, The obstacle and its shape are determined by randomly selecting and referring to the plurality of combinations of information stored in the storage unit. The embedding positions around the rail in the captured image are randomly determined, The training data is generated by embedding images corresponding to the determined obstacles and their shapes at the embedding positions in the captured images. The detection system according to claim 2.

4. The learning unit generates the training data using the generating AI by specifying the embedding position and inputting prompts indicating the determined obstacle and shape to the generating AI. The detection system according to claim 3.

5. The imaging unit has a downward-facing camera that images diagonally downward with respect to the horizontal direction. The detection unit detects the presence or absence of the obstacle based on the image captured by the downward-facing camera. The detection system according to any one of claims 1 to 4.

6. A detection method performed by a detection system, An imaging step in which an imaging unit mounted on a railway vehicle running on rails is used to output an image of the rails, A detection step involves inputting the captured image into a trained model to detect the presence or absence of obstacles around the rail, Equipped with, The aforementioned trained model is a machine learning model trained using image data generated by embedding images of obstacles randomly selected from a predetermined group of obstacles into the captured image output by the imaging unit at randomly determined positions around the rail in the captured image. Detection method.

7. The system further comprises a learning step of training the machine learning model using the aforementioned training data to construct the trained model. The detection method according to claim 6.

8. In the aforementioned learning step, The obstacle and its shape are determined by randomly selecting from a plurality of combinations of information, including information on the type of obstacle and the shape of the obstacle. The embedding positions around the rail in the captured image are randomly determined, The training data is generated by embedding images corresponding to the determined obstacles and their shapes at the embedding positions in the captured images. The detection method according to claim 7.

9. In the learning step, the generating AI is given prompts indicating the embedding position and the determined obstacle and shape, thereby generating the training data using the generating AI. The detection method according to claim 8.

Images