Status determination device, status determination method, and mobile support system

Abstract

To provide a state determination device that can detect unknown anomalies that may occur.SOLUTION: A state determination device 100 comprises a camera 1 that captures an orbit that is an object to be determined in relative displacement, an orbit detector 2 that detects orbit information that is information to be determined from the images captured by the camera 1 and outputs the detected orbit information and the degree of uncertainty of detection, and a determination unit 3 that determines that the state of the orbit is abnormal when the uncertainty is greater than a predetermined threshold.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a state determination device, a state determination method, and a mobile body support system.

Background Art

[0002] As a conventional state determination device, for example, an abnormal detection device for a linear body as disclosed in Patent Document 1 is known. As a main use of the state determination device, for example, there is a railway operation support system that supports the visual confirmation work of the track state ahead in a running railway vehicle. In the support of the visual confirmation work, in order to replace part or all of the normal or abnormal state determination by the visual inspection of the crew with a machine, a method of mounting a camera on the railway vehicle and performing state determination using an analysis device inside or outside the moving body is useful.

[0003] Patent Document 1 describes that "a learning model 43 that is deeply learned using, as learning data, a learning image related to a linear body and a correct answer value indicating whether or not each pixel in the learning image corresponds to an abnormality, and inputs the image as an input image to the learning model 43, and for each pixel in the input image, infer whether or not the pixel corresponds to an abnormality."

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when the method described in Patent Document 1 is used in the above-described operation support system, only known abnormal states defined in advance as learning data can be detected, so there is room for improvement in the detection of unknown abnormal states.

Means for Solving the Problems

[0006] A state determination device according to an aspect of the present invention is mounted on a railway vehicle and includes an imaging device that images the moving area of ​​the railway vehicle which includes a track that is moving relative to it, and a state determination device based on a normal track image. Orbital parameters, Curvature and other factors indicating the direction of orbital movement The aforementioned The system includes an orbit detection unit that detects orbit parameters from images captured by the imaging device using a model trained to detect orbit parameters, and calculates the variance of the orbit parameters for multiple frames detected from the most recent multiple frames of images, and determines that the orbit state is abnormal if the variance is greater than a predetermined threshold. A railway vehicle support system according to an aspect of the present invention supports a railway vehicle equipped with a state determination device, and comprises: an information control unit provided independently of the railway vehicle; a position information acquisition unit mounted on the railway vehicle that acquires the current time and current position of the railway vehicle; and an information transmission / reception unit mounted on the railway vehicle that transmits reference information linking the current time and current position with the track parameters detected by the track detection unit to the information control unit, wherein the information control unit includes an information storage unit that stores the reference information received from a plurality of railway vehicles supported by the railway vehicle support system, and upon receiving the reference information transmitted from the information transmission / reception unit, it searches for reference information linked to a time within a predetermined time period preceding the time of reception and the current position from the plurality of reference information stored in the information storage unit, and transmits the retrieved reference information to the information transmission / reception unit, and when the information transmission / reception unit receives the retrieved reference information, the track detection unit determines that the state of the track is abnormal based on the detected track parameters and the retrieved reference information. A state determination method according to an aspect of the present invention is a state determination method for determining the state of a track based on an image of the track moving relative to an imaging device mounted on a railway vehicle, wherein a track detection unit mounted on a railway vehicle determines the state of the track based on a normal track image. Orbital parameters, Curvature and other factors indicating the direction of orbital movement The aforementionedThe system detects the orbital parameters from the captured images using a model trained to detect orbital parameters, calculates the variance of the orbital parameters for multiple frames detected from the most recent multiple frames of the captured images, and determines that the orbital state is abnormal if the variance is greater than a predetermined threshold. [Effects of the Invention]

[0007] According to the present invention, it is possible to detect unknown anomalies that may occur. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing the configuration of the state determination device in the first embodiment. [Figure 2] Figure 2 is a flowchart showing an example of the state determination process in the first embodiment. [Figure 3] Figure 3 shows the internal structure of the trajectory detection unit in the second embodiment. [Figure 4] Figure 4 is a schematic diagram illustrating the status of orbital parameters. [Figure 5] Figure 5 is a flowchart showing an example of the internal processing flow for step S203 in Figure 2. [Figure 6] Figure 6 shows the configuration of the state determination device according to the third embodiment. [Figure 7] Figure 7 is a schematic diagram illustrating the calculation of uncertainty in the third embodiment. [Figure 8] Figure 8 shows the trajectory detection unit in the fourth embodiment. [Figure 9] Figure 9 shows the configuration of the state determination device according to the fifth embodiment. [Figure 10] Figure 10 shows an example of an abnormal occurrence pattern. [Figure 11] Figure 11 shows the configuration of the state determination device according to the sixth embodiment. [Figure 12] Figure 12 shows the configuration of the state determination device according to the seventh embodiment. [Figure 13] FIG. 13 is a diagram showing a schematic configuration of a railway operation support system in the eighth embodiment. [Figure 14] FIG. 14 is a diagram showing a configuration of a state determination device in the ninth embodiment. MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments of the state determination device according to the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and for clarity of explanation, omissions and simplifications are made as appropriate. In the following description, the same or similar elements and processes are denoted by the same reference numerals, and duplicate explanations may be omitted. Note that the content described below is merely an example of an embodiment of the present invention, and the present invention is not limited to the following embodiments, and can be implemented in various other forms.

[0010] (First Embodiment) In the present embodiment, a railway operation support system, which is the main application of the state determination device and the state determination method of the present invention, will be described as an example. FIG. 1 is a block diagram showing an example of the state determination device 100. The state determination device 100 forms part of a railway operation support system. The state determination device 100 detects a state abnormality that hinders the running of a railway vehicle using a captured image of the front of the railway vehicle captured by the camera 1. When the state determination device 100 detects a state abnormality, the railway operation support system issues an alert to the crew. In the present embodiment, knowledge about the track obtained through the detection of the track in the captured image is utilized to detect an unknown abnormality that may occur within the track.

[0011] As shown in FIG. 1, the state determination device 100 includes a camera 1, a track detection unit 2, and a determination unit 3. The state determination device 100 is mounted on a railway vehicle T. The camera 1 includes an imaging element such as a CMOS image sensor, and acquires a captured image of the running space mainly in front of the railway vehicle T. The captured data, which is the captured image of the camera 1, is input to the track detection unit 2.

[0012] The state determination device 100 includes a microcomputer, a processor, and arithmetic devices similar thereto, and a ROM, a RAM, a flash memory, a hard disk, an SSD, a memory card, an optical disk, and storage devices similar thereto, and realizes the functions of the track detection unit 2 and the determination unit 3 by executing a program stored in the storage device.

[0013] The imaging data from the camera 1 is input to the track detection unit 2. The track detection unit 2 applies image processing for detecting track information, which is information to be determined, from the imaging data, and outputs the detected track information. Here, the objects to be determined are rails, sleepers, ballasts, etc., which are components of the track. As the track information to be output, for example, semantic segmentation is used for track detection, and for each pixel of the captured image, information indicating which component of the track, such as a rail, a sleeper, or a ballast, it corresponds to or does not correspond to is given. Note that learning for track detection using normal images is performed in advance.

[0014] The track detection unit 2 detects track information from the captured image and outputs the detected track information and the uncertainty of the track information detection. When semantic segmentation is used for track detection, classes (track information) such as rails, sleepers, and ballasts, and prediction scores related to the classes are obtained. Therefore, an amount that has a negative correlation with this prediction score can be introduced and used as the uncertainty. For example, if a prediction score of 0 to 1 is defined, and in a state where the ballast is 0.8, the sleeper is 0.1, and the rail is 0.1, the class of ballast and the prediction score of 0.8 are obtained by semantic segmentation. In this case, an amount such as "1 - prediction score" or "1 - (prediction score)" can be used as the uncertainty. The larger these values are, the higher the uncertainty. 2 should be used as the uncertainty.

[0015] The determination unit 3 determines the condition of the track (rails, sleepers, ballast, etc.) based on the uncertainty input from the track detection unit 2. Specifically, if the uncertainty value is greater than a preset threshold, the determination unit 3 determines that the track condition is abnormal and outputs the determination result. Here, as a method for setting the threshold, for example, track detection is performed on various images of normal tracks to obtain the distribution of uncertainty. Then, a value that exceeds the upper limit of the obtained distribution is set as the threshold.

[0016] [Operation Sequence] Figure 2 is a flowchart showing an example of a state determination process performed by the state determination device 100. When the state determination device 100 is started, the process shown in Figure 2 begins. In step S201, it is determined whether or not there is an operation termination command to end the state determination process. If an operation termination command is received, the state determination process shown in Figure 2 is terminated. On the other hand, if no operation termination command is received, the process proceeds from step S201 to step S202.

[0017] In step S202, the video frame captured by camera 1 is acquired and input to the trajectory detection unit 2. Next, in step S203, the trajectory detection unit 2 detects the trajectory from the acquired video frame and simultaneously calculates the uncertainty of the detection. The trajectory detection unit 2 outputs the trajectory information and uncertainty, which are the trajectory detection results.

[0018] In step S204, the determination unit 3 compares the threshold value mentioned above with the obtained uncertainty value to determine whether the uncertainty value is greater than or equal to the threshold value. If it is determined in step S204 that the uncertainty value is greater than or equal to the threshold value, the process proceeds to step S205, where an abnormality determination result indicating an abnormality in the orbit is output, and then the process returns to step S201. On the other hand, if it is determined in step S204 that the uncertainty value is less than the threshold value, the process returns to step S201.

[0019] As described above, in this embodiment, by utilizing the uncertainty obtained when detecting trajectory information in the captured image, it becomes possible to detect unknown anomalies that may occur in the trajectory even without prior knowledge or training data regarding the state anomalies of the trajectory to be judged. Furthermore, since the aforementioned uncertainty is obtained when detecting trajectory information, the amount of computation can be reduced compared to when detecting trajectory information and detecting state information used for state judgment are performed separately.

[0020] In the embodiment described above, the entire state determination device 100 was mounted on a mobile body. However, it is also possible to mount only the camera 1 on the mobile body, which is the railway vehicle T, and have the processing of the track detection unit 2 and the determination unit 3 performed by a processing unit located elsewhere, such as a cloud server.

[0021] (Second Embodiment) A second embodiment will be described with reference to Figures 3 to 5. In the second embodiment, the system is configured to output the uncertainty of trajectory detection based on the variance of a series of trajectory information detected from consecutive captured images. In this embodiment, the camera 1 and determination unit 3 are the same as those described above, but the internal structure of the trajectory detection unit 2 is as shown in Figure 3. Below, the explanation of the camera 1 and determination unit 3, which have the same configuration, will be omitted, and the trajectory detection unit 2 will be described mainly.

[0022] As shown in FIG. 3, the trajectory detection unit 2 includes a detection unit 21, a trajectory information buffer 22, and an uncertainty calculation unit 23. The detection unit 21 includes, for example, a BNN (Bayesian neural network), applies a BNN for detecting a trajectory region to a captured image from the camera 1, and outputs trajectory information as a detection result. Here, the trajectory information to be output includes, for example, trajectory parameters such as curvature indicating the traveling direction of the trajectory. Also, the trajectory parameters detected for the captured image may be either one or a plurality. In the case of a plurality of trajectory parameters, they may be composed of a plurality of types of trajectory parameters instead of being composed of one type of trajectory parameter (for example, curvature). Hereinafter, the case where one trajectory parameter is output will be described as an example.

[0023] The trajectory information output from the detection unit 21 is output from the trajectory detection unit 2 and stored in the trajectory information buffer 22. The trajectory information buffer 22 has the performance of storing trajectory information for the past M (1 < M) frames including the latest trajectory information. That is, when a new trajectory detection is performed in a state where trajectory information for M frames is stored, the oldest trajectory information is discarded and the latest trajectory information is stored.

[0024] The uncertainty calculation unit 23 extracts the latest, that is, the trajectory information for the most recent N (1 < N ≦ M) frames from the trajectory information buffer 22, and calculates the detection uncertainty based on the variance of the trajectory information for those N frames. As the calculated uncertainty, for example, the variance itself may be used as the uncertainty, or addition or subtraction of bias and multiplication of weighting may be performed on the variance.

[0025] Generally, in a BNN, internal model parameters are defined as a probability distribution, and the model parameters used at the time of inference are sampled from the probability distribution. Therefore, the output trajectory parameters will be different values for each inference. Therefore, for a series of trajectory parameters output by performing N inferences on the same captured image, an expected value and a variance can be defined.

[0026] BNN training is performed using images of normal trajectories without anomalies, so that the expected value of the output trajectory parameters approaches the true value. As a result, the variance of the output trajectory parameters is small for images of normal trajectories. On the other hand, for images of trajectories with unknown anomalies that were not seen during training, the variance of the output trajectory parameters becomes large.

[0027] Incidentally, defining the expected value and variance for the trajectory parameters output by the BNN would normally require performing N inferences on the same captured image, thus reducing the throughput from capture to judgment. However, railway tracks do not change abruptly, and there is a high degree of similarity in image information between consecutively captured images. In this embodiment, this property is utilized to calculate the variance from the trajectory parameters of the most recent N consecutive frames, and the uncertainty of detection is calculated based on that variance, thereby improving the throughput from capture to judgment.

[0028] Figure 4 is a schematic diagram illustrating the status of track parameters detected by the detection unit 21. The captured image F401 shows the rail F402, and the detection unit 21 identifies the space in which the railway vehicle is running by detecting track parameters F404, such as the curvature that indicates the direction of travel of the rail F402. In the example shown in Figure 4, the track parameter F404 is the curvature of the rail F402, and the rail F402 gradually approaches a straight line from a left curve as time progresses. An anomaly F403 (obstacle on the track) occurs during the process of approaching a straight line from a left curve.

[0029] As mentioned above, for the orbital parameter F404, the variance F405 is calculated using the values ​​from the most recent N frames. In Figure 4, the two dashed lines drawn on either side of the solid orbital parameter F404 represent the variance F405. In normal orbits before and after the occurrence of anomaly F403, the value of the orbital parameter F404, which is output in accordance with the change in rail F402 in the captured image F401, also changes, and the variance F405 remains small. On the other hand, when anomaly F403 occurs, the orbital parameter F404 becomes unstable and the variance F405 increases. Therefore, it is possible to calculate the uncertainty of detection based on the variance F405 and determine the state of the orbit based on that uncertainty.

[0030] [Operation Sequence] The basic flow of the state determination process of the state determination device 100 is the same as the flowchart shown in Figure 2, but the processing content of step S203 differs in accordance with the operation of the trajectory detection unit 2 described above. In this embodiment, the internal processing of step S203 in Figure 2 is replaced by the series of processes shown in Figure 5. Figure 5 is a flowchart showing an example of the internal processing flow of step S203 in Figure 2.

[0031] In step S2031 of Figure 5, the detection unit 21 detects the trajectory from the captured image and outputs the trajectory information. In step S2032, the trajectory information is stored in the trajectory information buffer 22. In step S2033, it is determined whether the number of trajectory information entries stored in the trajectory information buffer 22 is N or greater, and if so, the process proceeds to step S2034. In step S2034, the uncertainty calculation unit 23 calculates the uncertainty of trajectory detection from the trajectory information of the latest N frames. On the other hand, if it is determined in step S2033 that the number of trajectory information entries is less than N, the series of processes is terminated. When the series of processes in Figure 5 is completed, that is, when the process in step S203 of Figure 2 is completed, the process proceeds to step S204 of Figure 2.

[0032] As described above, in this embodiment, the uncertainty of trajectory detection is output based on the variance of a series of trajectory information detected from consecutive captured images, so that the state of the trajectory can be determined based on that certainty. Furthermore, while a typical BNN performs N inferences on the same image to obtain N trajectory parameters, this embodiment uses images of N consecutive frames with high similarity to obtain the same N trajectory parameters as in the case of N inferences. And, since the uncertainty is calculated based on the variance of the trajectory information for N frames read from the trajectory information buffer 22 for each frame captured, that is, for each capture cycle time, the time from capture to determination can be shortened compared to when N inferences are performed on one frame image.

[0033] (Third embodiment) A third embodiment will be described with reference to Figures 6 and 7. In the third embodiment, the uncertainty of track detection and the state of the track are determined at the pixel level of the captured images, using a series of captured images and speed information of the railway vehicle.

[0034] As shown in Figure 6, this embodiment is configured by adding a speed information acquisition unit 4 to the state determination device 100 in the configuration shown in Figure 3. The speed information acquisition unit 4 may, for example, acquire the detected value from a speedometer mounted on a railway vehicle, or it may be configured to calculate the speed by differentiating the position information acquired by GPS (Global Positioning System) with respect to time. The configuration other than that shown in Figure 6 is the same as that of the first embodiment described above, and will not be explained below.

[0035] The detailed configuration of the track detection unit 2 will be explained using Figure 6. The detection unit 21 includes, for example, a BNN that performs semantic segmentation. The detection unit 21 applies the BNN for detecting the track region to the image captured from the camera 1 and outputs track information on a pixel-by-pixel basis. Here, the output track information may be, for example, information indicating which of the track components such as rails, sleepers, and ballast each pixel of the captured image corresponds to, or does not correspond to.

[0036] The track information in pixel units detected by the detection unit 21 is output from the track detection unit 2 and stored in the track information buffer 22. The track information buffer 22 has the performance of storing the track information in pixel units for the past M (1 < M) frames including the track information in pixel units of the latest frame. The speed information acquisition unit 4 measures the speed of the current railway vehicle and outputs the speed information to the uncertainty calculation unit 23.

[0037] The uncertainty calculation unit 23 extracts the track information in pixel units for the latest N (1 < N ≤ M) frames from the track information buffer 22 and tracks the pixels indicating the same location between consecutive frames based on the speed information from the speed information acquisition unit 4. Then, the uncertainty calculation unit 23 calculates the variance of the track information of the pixels indicating the same location and calculates the detection uncertainty in pixel units based on the calculated variance.

[0038] FIG. 7 is a schematic diagram showing the calculation of the detection uncertainty in pixel units. FIG. 7 shows a captured image F401(t1) taken at time t1 and a captured image F401(t2) taken at time t2 = t1 + Δt. In the captured image F401, a plurality of pixel sets F701 are set along the region of the rail F402. The pixel sets F701a and F701b shown in the captured images F401(t1) and F401(t2) represent one of the plurality of pixel sets F701 and indicate the same pixel set. That is, during the elapse of a predetermined time Δt, the same pixel set moves from the position of the pixel set F701a to the position of the pixel set F701b within the captured image F401.

[0039] In this way, by tracking pixels representing the same location across consecutive frames based on the speed information of the railway vehicle, it becomes possible to continuously track whether the same location is normal or abnormal. If the track condition at the same location is normal, the variance of track information in pixels representing that location will be small. On the other hand, as shown in Figure 7, if the track condition at the same location is abnormal, the variance of track information in pixels representing that location will be large. Therefore, based on the variance of track information at the pixel level, it becomes possible to calculate the uncertainty of track detection on a pixel-by-pixel basis and determine the track condition on a pixel-by-pixel basis. As described above, in this embodiment, using consecutive captured images and the speed information of the moving object, it is possible to calculate the uncertainty of track detection and determine the track condition at the pixel level of the captured images.

[0040] (Fourth embodiment) A fourth embodiment will be described with reference to Figure 8. In the fourth embodiment, prior knowledge about the trajectory is utilized, and the degree of uncertainty is calculated based on the deviation from the prior knowledge. Figure 8 is a diagram showing the trajectory detection unit 2 in this embodiment. The other configurations are the same as in the first embodiment shown in Figure 1, and will not be described below.

[0041] As shown in Figure 8, the track detection unit 2 comprises a detection unit 21 and an uncertainty calculation unit 23. The detection unit 21 includes, for example, a Deep Neural Network (DNN) that performs semantic segmentation. The detection unit 21 applies the DNN for detecting the track region to the image captured from the camera 1 and outputs track information on a pixel-by-pixel basis. Here, the track information to be output may include, for example, information indicating whether or not each pixel in the captured image corresponds to one of the track components such as rails, sleepers, or ballast.

[0042] The uncertainty calculation unit 23 receives the captured image and the pixel-level track information output from the detection unit 21 as input. The uncertainty calculation unit 23 applies prior knowledge of the track corresponding to the pixel-level track information to the pixel in the captured image or a small region including the vicinity of that pixel. Prior knowledge of the track includes, for example, knowledge such as "rails extend far into the distance" and "sleepers are present at regular intervals."

[0043] The uncertainty calculation unit 23 performs a predetermined calculation regarding prior knowledge and calculates uncertainty based on the discrepancy between the calculation result and the prior knowledge. For example, if the detection unit 21 is able to suitably detect the track, when edge detection is performed on a small area including the vicinity of a pixel that the detection unit 21 has determined to be a rail, an edge extending in the direction of travel should be detected. On the other hand, if there is an abnormality such as damage or an obstruction to the rail, the edge detection in the small area may result in the inability to detect an edge in the direction of travel, or an edge in a different direction from the direction of travel being detected.

[0044] Therefore, by calculating the difference between the direction of the detected edge and the angle of travel, it is possible to evaluate the deviation from the prior knowledge of the track, which is that the rail extends into the distance. Then, by calculating the uncertainty based on the deviation, it is possible to determine the state of the track. In evaluating the deviation, the strength of the edge may also be considered in addition to the direction of the edge. Furthermore, image information other than the edge, such as the texture and frequency components of the image, may also be used. In addition, when calculating the uncertainty based on the deviation, for example, the deviation itself may be used as the uncertainty. Alternatively, bias addition or subtraction, or weighted multiplication may be applied to the deviation.

[0045] (Fifth embodiment) A fifth embodiment will be described with reference to Figures 9 and 10. In the fifth embodiment, abnormal events that are likely to occur are estimated based on the abnormal occurrence pattern. Figure 9 is a diagram showing the configuration of the state determination device 100 in the fifth embodiment. The configuration shown in Figure 9 is the same as the configuration shown in Figure 1, but with the addition of an abnormal event estimation unit 5.

[0046] The track detection unit 2, for example, uses semantic segmentation for track detection and outputs pixel-level track information and pixel-level uncertainty. Here, the output pixel-level track information may include, for example, information indicating which of the track components such as rails, sleepers, and ballast a pixel corresponds to, or does not correspond to, a particular pixel.

[0047] The determination unit 3 outputs a pixel-level determination result based on the pixel-level uncertainty input from the trajectory detection unit 2. The abnormal event estimation unit 5 compares the pixel-level trajectory information from the trajectory detection unit 2 with the pixel-level determination result from the determination unit 3 to identify the component where the abnormality is occurring, i.e., the location of the abnormality. Then, the abnormal event estimation unit 5 estimates the abnormal events that are most likely to have occurred based on the abnormality pattern at the location of the abnormality.

[0048] Figure 10 shows an example of an anomaly occurrence location F1001, an anomaly event F1002, and an anomaly occurrence pattern F1003. In the example shown in Figure 10, the pixel-level track information indicates whether each pixel corresponds to a rail, sleeper, or ballast, or not. The anomaly occurrence location F1001, which is identified by comparing the pixel-level track information with the pixel-level determination result, consists of three types: rail, sleeper, and ballast. The estimated anomaly event F1002 consists of five types: rail damage, sleeper damage, ballast collapse, weed overgrowth, and landslide.

[0049] Anomaly occurrence pattern F1003 associates the location of the anomaly with the anomaly event, and is defined in advance. For example, if no anomalies occur in the rails or sleepers, but only in the ballast, it is possible that the ballast has collapsed. Alternatively, if anomalies occur in the rails, sleepers, and ballast, it is possible that a major anomaly affecting the entire track, such as a landslide, has occurred.

[0050] Thus, in the fifth embodiment, it is possible to estimate an abnormal event that is highly likely to have occurred based on the abnormal occurrence pattern. Note that the abnormal occurrence location F1001, abnormal event F1002, and abnormal occurrence pattern F1003 shown in FIG. 10 are merely examples, and each item may be increased or decreased, or the pattern itself may be changed.

[0051] (Sixth Embodiment) The sixth embodiment will be described with reference to FIG. 11. In the sixth embodiment, by relying on the residence time of the abnormality, it is made possible to determine that the abnormality has occurred not in the orbit but in the camera. FIG. 11 is a diagram showing the configuration of the state determination device 100 in the sixth embodiment. The configuration shown in FIG. 11 is obtained by adding a determination result buffer 6 and a camera state determination unit 7 to the configuration shown in FIG. 1.

[0052] The determination result buffer 6 has the function of storing the determination results for the past L (1 < L) frames including the latest determination result output by the determination unit 3. When a new determination is made while the determination results for L frames are stored, the oldest determination result is discarded and the latest determination result is stored.

[0053] The camera state determination unit 7 checks the series of determination results stored in the determination result buffer 6, and if the determination results determined as abnormal exceed the number of frames for a predefined residence time, it determines that the abnormality has occurred not in the orbit but in the camera 1. Then, it outputs that determination result as the camera state determination result. Note that when the determination result is in pixel units, the residence time may be determined in pixel units. Alternatively, when an abnormality has occurred in a predetermined number of pixels or more, it may be regarded as abnormal in image units and the residence time may be determined in image units.

[0054] Examples of abnormalities that can occur in camera 1 include dirt or snow adhering to the lens. Because the railway vehicle is in motion, if the abnormality occurs on the track, it will eventually move out of camera 1's field of view and will not linger for a long period of time. On the other hand, if the abnormality occurs in camera 1, and it is dirt or snow, as long as these remain attached to the lens, the camera may continue to output an abnormality detection result.

[0055] Therefore, based on the duration of the anomaly, it becomes possible to determine that the anomaly occurred in camera 1 and not in the trajectory. Thus, in this embodiment, it is possible to determine that the anomaly occurred in the camera and not in the trajectory based on the duration of the anomaly.

[0056] (Seventh Embodiment) A seventh embodiment will be described with reference to Figure 12. In the seventh embodiment, it is determined that the abnormality is occurring in the camera rather than the trajectory, based on the motion component of the abnormality. Figure 12 is a diagram showing the configuration of the state determination device 100 in the seventh embodiment. The configuration shown in Figure 12 is the same as the configuration shown in Figure 1, but with the addition of a motion component detection unit 8 and a camera state determination unit 7.

[0057] The motion component detection unit 8 detects motion components in the captured image at the pixel level or small region level by, for example, calculating optical flow from the captured image. The detected motion component information is output to the camera state determination unit 7.

[0058] The camera state determination unit 7, for example, if the determination result output by the determination unit 3 is on a pixel-by-pixel basis, checks the motion component of the pixel position where the anomaly is occurring based on the motion component information. If the motion component is below a predetermined threshold and can be considered stationary, the camera state determination unit 7 determines that the anomaly is occurring in camera 1 and not in the trajectory, and outputs this as the camera state determination result.

[0059] Examples of abnormalities occurring in camera 1 include dirt or snow adhering to the lens. Since the railway vehicle is in motion, if the abnormality occurs on the track, it will move within the captured image along with the movement of the railway vehicle. On the other hand, if the abnormality occurs in camera 1, and it is dirt or snow, it will not move within the captured image as long as it remains attached to the lens. Therefore, based on the motion component of the abnormality, it is possible to determine that the abnormality is occurring in camera 1 and not on the track.

[0060] (Eighth embodiment) The eighth embodiment will be described with reference to Figure 13. In the eighth embodiment, the number of track information samples used to calculate uncertainty is increased by integrating and utilizing track information detected by multiple train sets, thereby improving the accuracy of uncertainty calculation and state determination. Figure 13 is a diagram showing the schematic configuration of the railway operation support system 1000 in the eighth embodiment. The railway operation support system 1000 comprises state determination devices 100 mounted on each of the multiple railway vehicles T1, T2, ... and a server 200 located in a different location from the multiple railway vehicles T1, T2, .... The processing operation will be described below using railway vehicle T1 as an example, but the same processing is performed for the other railway vehicles T2, ....

[0061] The state determination device 100 includes a camera 1, a trajectory detection unit 2, a determination unit 3, and a position information acquisition unit 9. The trajectory detection unit 2 includes a detection unit 21 and an uncertainty calculation unit 23, as shown in Figure 3. The trajectory detection unit 2 further includes a trajectory information transmission / reception unit 24. The server 200 includes a trajectory information transmission / reception unit 10, a trajectory information control unit 11, and a trajectory information buffer 12.

[0062] The location information acquisition unit 9 acquires information regarding the current time and current location of the railway vehicle T1 and outputs the time and location information. As a method for acquiring the current location, for example, a GPS mounted on the railway vehicle T1 may be used. Alternatively, speed information may be acquired from the speedometer equipped on the railway vehicle T1, and the position may be calculated by integrating the speed over time. The track information transmission / reception unit 24 links the time and location information output by the location information acquisition unit 9 with the track information output by the detection unit 21 to form time, location, and track information, and transmits this time, location, and track information to the server 200. As shown in Figure 13, the server 200 receives time, location, and track information not only from the railway vehicle T1 but also from other railway vehicles T2, ... equipped with the state determination device 100.

[0063] When the track information transmission / reception unit 10 of the server 200 receives time, location, and track information from the railway vehicle T1, the track information control unit 11 stores the received time, location, and track information in the track information buffer 12. Furthermore, the track information control unit 11 searches for a group of time, location, and track information received from other railway vehicles T2, ... that have traveled in the past within a predetermined time period from the time of reception, with respect to the location indicated by the location information included in the time, location, and track information. Then, the track information control unit 11 transmits the group of time, location, and track information extracted by the search to the railway vehicle T1.

[0064] The track information transmission / reception unit 24 of the railway vehicle T1, which is traveling at the location in question, receives a group of time, location, and track information from the server 200, and outputs the received group of time, location, and track information, along with the latest track information output by the detection unit 21, to the uncertainty calculation unit 23. The uncertainty calculation unit 23 calculates the uncertainty using the group of time, location, and track information and the latest track information.

[0065] Since railway vehicles only travel on laid tracks, images taken from multiple railway vehicles traveling at the same location tend to have a high degree of similarity. By utilizing this property and using a set of time, location, and track information received from other railway vehicles that have traveled in the past within a predetermined time period from the present, it is possible to increase the number of samples used to calculate the variance of track information, thereby improving the accuracy of uncertainty calculation and state determination.

[0066] In the configuration shown in Figure 13, the track information buffer 12 is provided only on the server 200, and a system is in place to centrally manage time, position, and track information detected by multiple railway vehicles. However, for example, a system could also be used in which track information buffers are installed on the railway vehicles as well, and these can be used in combination.

[0067] (Ninth Embodiment) A ninth embodiment will be described with reference to Figure 14. In the ninth embodiment, track detection and state determination are performed on the captured image, and the speed of the railway vehicle is controlled based on the obtained track information and determination results. Figure 14 is a diagram showing the schematic configuration of the state determination device 100 in the ninth embodiment.

[0068] The configuration of the state determination device 100 shown in Figure 14 is the same as that shown in Figure 1, but with the addition of a mobile body control unit 14. The mobile body control unit 14 controls the speed of the moving object, the railway vehicle T, based on the track information output by the track detection unit 2 and the determination result output by the determination unit 3. The track information output from the track detection unit 2 indicates the shape of the track. Therefore, for example, it is possible to control the speed of the railway vehicle T by reducing it on a curved track compared to a straight track.

[0069] Furthermore, the judgment result of the judgment unit 3 indicates whether or not there is an abnormality in the track. Therefore, for example, if an abnormality occurs in the track, it is possible to immediately apply the brakes. These controls enable semi-automation or complete automation of the operation while maintaining the safety of the operation of the railway vehicle T.

[0070] (Tenth embodiment) In the first to ninth embodiments described above, a railway operation support system 1000 equipped with a condition determination device 100 was explained as an example. On the other hand, in the tenth embodiment, an example of applying the condition determination device 100 to a production line inspection system will be described. In the production line inspection system of this embodiment, abnormal conditions of the production line or goods are detected using images of the production line, such as a belt conveyor, or goods moving on the production line, and alerts are issued to workers.

[0071] The state determination device 100 in the production line inspection system of this embodiment has the same configuration as shown in Figure 1 of the first embodiment, and will be described with reference to Figure 1. However, each processing unit is fixed near the production line and not a moving object such as a railway vehicle T. Camera 1 is mounted near the production line and acquires images of the production line or items moving on the production line, such as a belt conveyor. Track detection unit 2 applies image processing to the captured images to detect the production line area in order to identify the moving space of the production line from the captured images, and outputs track information, which is the information to be determined. Here, the output track information may be, for example, information indicating which of the components of the production line, such as a belt or drive roller, each pixel of the captured image corresponds to, or does not correspond to, by using semantic segmentation for track detection.

[0072] The trajectory detection unit 2 detects the production line area and outputs trajectory information, while simultaneously outputting the uncertainty of the detection. Similar to the first embodiment, if semantic segmentation is used for trajectory detection, for example, a value that has a negative correlation with the predicted score for the output class may be obtained and used as the uncertainty.

[0073] The determination unit 3 determines the state of the production line to be judged based on the uncertainty output from the trajectory detection unit 2. Specifically, if the uncertainty is greater than a predetermined threshold, the determination unit determines the state of the target to be judged as abnormal and outputs the determination result. The method for determining the threshold may be, for example, to perform trajectory detection on images of various normal production lines, obtain the distribution of uncertainty, and set the value that exceeds the upper limit of that distribution as the threshold.

[0074] The railway operation support systems described in the first to ninth embodiments above have a configuration in which each processing unit is mounted on a mobile body, while the production line inspection system of this embodiment has a configuration in which each processing unit is fixed near the production line rather than on a mobile body. However, all configurations have in common that the object to be judged moves relative to the camera 1. Therefore, the present invention can be applied under conditions in which the object to be judged moves within the captured image.

[0075] As described above, in this embodiment, a production line inspection system can be realized by detecting the production line area from the captured image, outputting trajectory information, outputting the degree of uncertainty of the detection, and determining that the condition of the production line or goods is abnormal when the degree of uncertainty is large.

[0076] Furthermore, while the above-described embodiment mainly described the application of the present invention to a railway operation support system for detecting abnormal conditions that may hinder the operation of railway vehicles and issuing alerts to crew members, it may also be applied to other uses. For example, it may be applied to operation support systems for automobiles traveling on highways, aircraft traveling on runways, drones flying along power lines, or maintenance support systems.

[0077] According to the embodiments of the present invention described above, the following effects and advantages are achieved.

[0078] (C1) As shown in Figure 1, the state determination device 100 includes a camera 1 as an imaging device that captures the trajectory of the object to be determined, which is moving relative to it; a trajectory detection unit 2 as an object to be determined detection unit that detects trajectory information, which is the object to be determined, from the image captured by the camera 1 and outputs the detected trajectory information and the uncertainty of the detection; and a determination unit 3 that determines the state of the trajectory to be abnormal when the uncertainty is greater than a predetermined threshold. In this way, by utilizing the uncertainty obtained along with the detection of trajectory information in the captured image, it becomes possible to detect unknown anomalies that may occur in the trajectory even without prior knowledge or training data regarding trajectory state anomalies.

[0079] (C2) In (C1) above, as shown in Figures 3 and 4, the trajectory detection unit 2 reads the latest N frames of trajectory information from the trajectory information buffer 22, for example, from the trajectory information for multiple frames detected from the most recent multiple frames of captured images, and calculates the uncertainty of detection based on the variance of the N frames of trajectory information. In this way, the uncertainty of trajectory detection can be calculated based on the variance of a series of trajectory information detected from consecutive captured images, and the time from capture to determination can be shortened compared to when N inferences are performed on a single frame image.

[0080] (C3) In (C1) above, as shown in Figure 8, the trajectory detection unit 2 detects pixel-level trajectory information from the captured image by semantic segmentation, applies prior knowledge of the trajectory to be determined, corresponding to the pixel-level trajectory information, to the corresponding pixel in the captured image or the neighboring region containing the corresponding pixel, calculates the deviation from the prior knowledge, and calculates the detection uncertainty based on the deviation. The state of the trajectory can be determined by the uncertainty calculated in this way.

[0081] (C4) In (C1) above, as shown in Figures 9 and 10, the condition determination device 100 further includes an abnormal event estimation unit 5 that identifies the location of the abnormality and estimates the abnormal event that is most likely to occur. The track detection unit 2 detects track information at the pixel level from the captured image and outputs the detected track information and the uncertainty of the detection at the pixel level. The determination unit 3 outputs a determination result at the pixel level based on the uncertainty at the pixel level. As shown in Figure 10, the abnormal event estimation unit 5 identifies the location of the abnormality F1001 based on the track information at the pixel level and the determination result at the pixel level and estimates the abnormal event that is most likely to occur. For example, if no abnormality occurs in the rails or sleepers, but only in the ballast, it is possible that the ballast has collapsed.

[0082] (C5) In (C1) above, as shown in Figure 11, the state determination device 100 further includes a camera state determination unit 7 that determines abnormalities in camera 1. The trajectory detection unit 2 detects trajectory information from the most recent multiple frames of captured images and outputs the degree of uncertainty for the most recent multiple frames. The determination unit 3 outputs multiple determination results regarding the trajectory information for the most recent multiple frames. The camera state determination unit 7 determines that camera 1 is abnormal if the number of abnormal determinations included in the multiple determination results exceeds a predetermined number (the number of frames corresponding to a predetermined dwell time). In this way, it is possible to determine whether the abnormality is due to the trajectory or the camera.

[0083] (C6) In (C1) above, as shown in Figure 12, the state determination device 100 further comprises a motion component detection unit 8 that detects motion component information at the pixel level or sub-region level in the captured image, and a camera state determination unit 7 that determines an abnormality in the camera 1. The trajectory detection unit 2 detects trajectory information at the pixel level from the captured image and outputs the detected trajectory information and the uncertainty of the detection at the pixel level, and the determination unit 3 outputs a determination result at the pixel level based on the uncertainty at the pixel level. The camera state determination unit 7 then extracts the motion component of the pixel position in the captured image corresponding to the pixel determined to be abnormal by the determination unit 3 based on the motion component information, and determines that the camera 1 is abnormal if the extracted motion component falls below a predetermined threshold.

[0084] (C7) In (C1) above, as shown in Figure 1, the camera 1 may be mounted on a moving railway vehicle T, and the camera 1 may be configured to image the moving area of ​​the railway vehicle T, which includes the track.

[0085] (C8) In (C7) above, as shown in Figures 6 and 7, the state determination device 100 further includes a speed information acquisition unit 4 that acquires speed information of the railway vehicle T. The track detection unit 2 detects pixel-level track information from each of the most recent multiple frames of captured images by semantic segmentation, tracks pixels indicating the same location across consecutive frames based on the speed information acquired by the speed information acquisition unit 4, calculates the variance of the determination target information of pixels indicating the same location, and calculates the detection uncertainty on a pixel-by-pixel basis based on the variance. In this way, by tracking pixels indicating the same location across consecutive frames based on the speed information of the railway vehicle T, it becomes possible to continuously track whether the same location is normal or abnormal.

[0086] (C9) In (C7) above, as shown in Figure 14, the state determination device 100 further includes a mobile body control unit 14 that controls the speed of the moving railway vehicle T based on the track information detected by the track detection unit and the determination result of the determination unit 3. By including such a mobile body control unit 14, it becomes possible to semi-automate or automate the operation while maintaining the safety of the operation of the railway vehicle T.

[0087] (C10) In (C7) above, as shown in Figure 1, the moving object may be a railway vehicle T.

[0088] (C11) As shown in Figure 13, the railway operation support system 1000 is a mobile support system that supports railway vehicles T1, T2, ... equipped with the state determination device 100 described in (C1) above. The railway operation support system 1000 comprises a track information control unit 11 provided on the server 200 independently of the railway vehicles T1, T2, ..., a position information acquisition unit 9 mounted on the railway vehicles T1, T2, ... that acquires the current position and current time of the railway vehicles T1, T2, ..., and a track information transmission / reception unit 24 mounted on the railway vehicles T1, T2, ... that transmits time, position, and target information (reference information) linked to the current time and current position and track information detected by the track detection unit 2 to the track information control unit 11. The track information control unit 11 is equipped with a track information buffer 12 that stores time, position, and target information received from multiple moving objects T1, T2, ... supported by the railway operation support system 1000. When it receives time, position, and target information transmitted from the track information transmission / reception unit 24, it searches the track information buffer 12 for time, position, and target information linked to the current position within a predetermined time period preceding the time of reception, and transmits the retrieved time, position, and target information to the track information transmission / reception unit 24. When the track information transmission / reception unit 24 receives the retrieved time, position, and target information, the track detection unit 2 calculates the uncertainty based on the detected track information and the retrieved time, position, and target information. In this way, by using the time, position, and target information stored in the track information buffer 12, the number of samples used to calculate the variance of track information can be increased, thereby improving the accuracy of uncertainty calculation and state determination.

[0089] (C12) As shown in Figure 1, this is a state determination method for determining the state of a trajectory based on an image of the trajectory to be determined, which is moving relative to camera 1. The method detects trajectory information from the image, outputs the detected trajectory information and the uncertainty of the detection, and determines that the state of the trajectory is abnormal if the uncertainty is greater than a predetermined threshold. By using the uncertainty of the detection when detecting the trajectory from the image, it is possible to detect unknown anomalies that may occur in the trajectory even without prior knowledge or training data regarding the state anomalies of the trajectory to be determined.

[0090] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0091] Furthermore, each of the above configurations may be implemented either partially or entirely in hardware, or through program execution on a processor. Also, the control lines and information lines shown are those deemed necessary for illustrative purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, almost all configurations can be considered interconnected. [Explanation of symbols]

[0092] 1...Camera, 2...Trajectory detection unit, 3...Determination unit, 4...Speed ​​information acquisition unit, 5...Abnormal event estimation unit, 6...Determination result buffer, 7...Camera state determination unit, 8...Motion component detection unit, 9...Position information acquisition unit, 10,24...Trajectory information transmission / reception unit, 11...Trajectory information control unit, 12,22...Trajectory information buffer, 14...Moving object control unit, 21...Detection unit, 23...Uncertainty calculation unit, 100...State determination device, 1000...Railway operation support system, T,T1,T2...Railway vehicle

Claims

1. A condition determination device installed on a railway vehicle, An imaging device for imaging the moving area of ​​the railway vehicle, which includes a trajectory that moves relative to it, The system includes a trajectory detection unit that detects trajectory parameters from images captured by the imaging device using a model that has learned to detect trajectory parameters such as curvature indicating the direction of trajectory propagation, based on normal trajectory images, and calculates the variance of the trajectory parameters for multiple frames detected from the most recent multiple frames of images, If the aforementioned variance is greater than a predetermined threshold, the state of the orbit is determined to be abnormal. A state determination device characterized by the following features.

2. In the state determination device according to claim 1, The system further includes an imaging device determination unit for determining abnormalities in the imaging device, The trajectory detection unit detects the trajectory parameters from the most recent multiple frames of the captured images, calculates the variance for the most recent multiple frames, The state determination device outputs multiple determination results regarding the trajectory parameters for the most recent multiple frames. The imaging device determination unit is a state determination device that determines that the imaging device is abnormal when the number of abnormal determinations included in the plurality of determination results exceeds a predetermined number.

3. In the state determination device according to claim 1, A state determination device further comprising a railway vehicle control unit that controls the speed of the railway vehicle based on the track parameters detected by the track detection unit and the result of determining the state of the track.

4. A railway vehicle support system for supporting a railway vehicle equipped with the state determination device described in claim 1, An information control unit, which is provided independently of the aforementioned railway vehicle, A location information acquisition unit is installed in the aforementioned railway vehicle and acquires the current time and current location of the railway vehicle, The railway vehicle is equipped with an information transmission / reception unit that transmits reference information linking the current time, the current position, and the track parameters detected by the track detection unit to the information control unit, The information control unit includes an information storage unit that stores the reference information received from multiple railway vehicles supported by the railway vehicle support system, and upon receiving the reference information transmitted from the information transmission / reception unit, it searches the multiple reference information stored in the information storage unit for reference information that links a time within a predetermined time period preceding the time of reception with the current location, and transmits the retrieved reference information to the information transmission / reception unit. When the information transmission / reception unit receives the retrieved reference information, the trajectory detection unit determines that the trajectory state is abnormal based on the detected trajectory parameters and the retrieved reference information. Railway vehicle support system.

5. A method for determining the state of a track based on an image of the track moving relative to an imaging device mounted on a railway vehicle, wherein the state of the track is determined based on the image of the track moving relative to the imaging device mounted on the railway vehicle, A track detection unit mounted on a railway vehicle detects track parameters from captured images using a model that has learned to detect track parameters such as curvature indicating the direction of track travel, based on normal track images, and calculates the variance of the track parameters for multiple frames detected from the most recent multiple frames of captured images. A state determination method for determining the state of the orbit as abnormal when the aforementioned variance is greater than a predetermined threshold.

Images