State determination device and state determination method

Abstract

To provide a state determination device and method capable of detecting abnormality occurring in an object and identifying the abnormality and an erroneous detection factor such as a shadow by using a photographic image of the object.SOLUTION: A state determination device is used, including a camera mounted on a mobile body to acquire a photographic image in a travel space, an object image extraction part for extracting an object image of a small area including an object of state determination in the photographic image, and a determination part for performing image processing on the object image and outputting a state determination result showing the presence / absence of abnormality of the object and an erroneous detection factor determination result showing the presence / absence of an erroneous detection factor.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a state determination device and a state determination method. [Background technology]

[0002] Background art of this technical field is JP 2021-124751 A (Patent Document 1). This publication states that "the imaging unit is provided on the railway vehicle and captures images of railway facilities installed along the railway vehicle's travel path. ~Omitted~ The video captured by the imaging unit is associated with the vehicle position and the video is encoded. ~Omitted~ A motion vector between a first image included in the decoded first video and a second image included in the decoded second video is calculated, and an abnormality in the railway facilities is diagnosed based on the motion vector." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-124751 A Summary of the Invention [Problem to be solved by the invention]

[0004] The main applications of the status determination device and the status determination method include, for example, a wayside facility inspection system that aims to automatically determine the status (normal / abnormal) of wayside facilities in order to realize railway operation support and unmanned operation. For the wayside facility inspection system, a method of equipping a camera on a railway vehicle and determining the status from the captured images using an analysis device inside or outside the vehicle is useful.

[0005] The Patent Document 1 states that "the imaging unit is provided in the railcar and captures railroad equipment installed along the railcar's travel path. ~Omitted~ The video captured by the imaging unit is associated with the vehicle position and the video is encoded. ~Omitted~ A motion vector between a first image included in the decoded first video and a second image included in the decoded second video is calculated, and an abnormality in the railroad equipment is diagnosed based on the motion vector." However, it is difficult to detect an abnormality that does not appear in the motion vector, such as dirt on the surface of the equipment, and there is room for improvement. In particular, how to distinguish between a shadow or the like on the surface of the equipment and an abnormality is one of the important points. For example, if it is not possible to distinguish between a shadow and dirt on the surface of the equipment and the shadow is determined to be an abnormality, unnecessary maintenance work will occur. In addition, if unnecessary alerts are issued frequently, it will cause a decrease in reliability of the alert. An object of the present invention is to provide a state determination device and a state determination method that can detect abnormalities occurring in an object using a captured image of the object and distinguish between abnormalities and causes of erroneous detection such as shadows. [Means for solving the problem]

[0006] In order to achieve the above object, one representative status determination device of the present invention is characterized by comprising: a camera mounted on a moving body and acquiring captured images of the driving space; a target image extraction unit that extracts a target image of a small area including an object for status determination from the captured images; and a determination unit that performs image processing on the target image and outputs a status determination result indicating the presence or absence of an abnormality in the object and a false detection factor determination result indicating the presence or absence of a false detection factor. Furthermore, one representative status determination method of the present invention is characterized in that it includes a video frame acquisition step in which a status determination device acquires a video frame of a traveling space of a moving body, a target image extraction step in which a target image of a small area including an object for status determination in the video frame is extracted, and a determination step in which image processing is performed on the target image, and a status determination result indicating the presence or absence of an abnormality in the object and a false detection factor determination result indicating the presence or absence of a false detection factor. Effect of the Invention

[0007] According to the present invention, it is possible to provide a condition determination device and method that can use a captured image of an object to detect abnormalities occurring in the object and distinguish between abnormalities and causes of erroneous detection such as shadows. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an overall configuration of a lineside facility inspection system according to a first embodiment of the present invention. [Figure 2A] FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle. [Figure 2B] FIG. 2 is a schematic diagram showing an example of a target image. [Figure 2C] FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle. [Figure 2D] FIG. 2 is a schematic diagram showing an example of a target image. [Diagram 3] 2 is a diagram illustrating an internal configuration of a determination unit in the wayside facility inspection system according to the first embodiment of the present invention. FIG. [Figure 4] 5 is a table showing an example of an internal operation of an integrated determination unit in the lineside facility inspection system according to the first embodiment of the present invention. [Diagram 5] 5 is a flowchart illustrating an example of internal processing in the wayside facility inspection system according to the first embodiment of the present invention. [Figure 6] 5 is a flowchart illustrating an example of internal processing of a determination step in the lineside facility inspection system according to the first embodiment of the present invention. [Figure 7] FIG. 4 is a diagram illustrating an overall configuration of a wayside facility inspection system according to a second embodiment of the present invention. [Figure 8] FIG. 11 is a diagram illustrating an internal configuration of a determination unit in a wayside facility inspection system according to a second embodiment of the present invention. [Figure 9A] 13 is a schematic diagram showing an example of an anomaly score history when the state of an object is normal. FIG. [Figure 9B] 13 is a schematic diagram showing an example of an abnormality score history when the state of an object is abnormal. FIG. [Figure 10]FIG. 11 is a diagram illustrating the overall configuration of a lineside facility inspection system according to a third embodiment of the present invention. [Figure 11] FIG. 13 is a diagram illustrating the overall configuration of a lineside facility inspection system according to a fourth embodiment of the present invention. [Figure 12A] FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle. [Figure 12B] FIG. 2 is a schematic diagram showing an example of a captured image of the front of a railway vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. EXAMPLES

[0010] In this embodiment, a wayside facility inspection system, which is a main application of the state determination device and method of the present invention, is described. The wayside facility inspection system uses captured images of the area in front of a railroad vehicle to determine whether or not there is an abnormality in the wayside facility, and uses the information to assist in facility management and repair planning.

[0011] Fig. 1 is a diagram illustrating the overall configuration of a wayside facility inspection system according to an embodiment of the present invention. The purpose of this embodiment is to provide a wayside facility inspection system that uses photographed images of the object to detect abnormalities occurring in the object and is capable of distinguishing between abnormalities and causes of false detection such as shadows.

[0012] [System Configuration] The structure and operation of a wayside facility inspection system according to an embodiment of the present invention will be described below. The wayside facility inspection system according to the present embodiment is composed of a camera 1, a target image extraction unit 2, and a determination unit 3, as shown in FIG. The operation of the wayside facility inspection system will be explained with reference to Fig. 1. Camera 1 is mounted on a moving railway vehicle and captures images of the traveling space, mainly in front of the railway vehicle. Target image extraction unit 2 extracts a target image of a small area including the target from the captured image in order to focus the inspection on the target. Judgment unit 3 performs image processing on the target image and outputs a state judgment result indicating the presence or absence of an abnormality in the target and a false detection factor judgment result indicating the presence or absence of a false detection factor.

[0013] The target objects of the wayside facility inspection system are, for example, signs, traffic signals, ground coils, etc. The abnormalities of the detection target are, for example, dirt, defects, rust, peeling paint, obscuration by vegetation, etc. The mounting position of the camera 1 may be, for example, inside or outside the vehicle near the driver's cab of the railcar, or even on the roof, allowing the camera to capture an image of the entire front of the railcar and target a wide range of equipment. Alternatively, mounting the camera on the side of the vehicle allows for detailed imaging from a position closer to the target. Alternatively, mounting a camera that captures not only the front of the railcar but also the rear allows the camera to target equipment along the opposing track.

[0014] The camera 1 can inspect objects at a long distance by using a telephoto lens or a high-resolution image sensor. In particular, when the system is operated while a railroad vehicle is in operation and it is desired to detect abnormalities that may interfere with operation at an early stage, it is desirable to be able to inspect objects at a long distance.

[0015] For example, when the object has predetermined characters such as a speed sign, the target image extraction unit 2 may extract the target image by recognizing the characters using Optical Character Recognition (OCR). Alternatively, the target image may be extracted by using an image processing technique such as Deep Neural Network (DNN) to output information on the position and size of the object from the captured image, or by detecting pixels in which the object exists using semantic segmentation.

[0016] For example, if the image is taken during the day, the causes of erroneous detection include shadows caused by sunlight and surrounding structures, sunlight reflection, rain, snow, etc., and if the image is taken at night, the causes of erroneous detection include headlights of automobiles traveling along the railway line and shadows caused by surrounding structures, reflection of the headlights, etc. In the following, sunlight and shadows caused by surrounding structures will be described as erroneous detection causes, but the present invention can be applied to other erroneous detection causes using a similar configuration and method.

[0017] Fig. 2A is an example of a captured image of the front of a railway vehicle. A sign F202 is captured in a captured image F201 acquired by a camera 1. A stain F203 has appeared on the sign F202. A target image extraction unit 2 detects the sign F202 from the captured image F201, and outputs a target image F204 shown in Fig. 2B. A determination unit 3 detects the stain F203 from the target image F204, and outputs a state determination result indicating that an abnormality exists.

[0018] Fig. 2C is an example of a captured image of the front of a railway vehicle. A shadow F205 caused by a nearby electric wire is cast on a sign F202. The presence of the shadow F205 is not abnormal for the target object, but it may cause a false detection of an abnormality. Therefore, even if the shadow F205 can be judged as abnormal from the target image F204 shown in Fig. 2D, the judgment unit 3 recognizes the presence of the shadow by a separate process, and makes an integrated judgment from these results to output a state judgment result as normal.

[0019] 3 is a diagram showing an example of the internal configuration of the determination unit 3. The determination unit 3 is made up of an abnormality score calculation unit 301, a shadow score calculation unit 302, and an integrated determination unit 303. The abnormality score calculation unit 301 calculates an abnormality score indicating the degree of deviation from the normal state of the object by image processing. The shadow score calculation unit 302 calculates a shadow score indicating the degree of a shadow caused on the object by image processing. The integrated judgment unit 303 uses the abnormality score and the shadow score, and outputs a state judgment result and a shadow judgment result after taking into account an increase in the abnormality score caused by the shadow.

[0020] If the anomaly score calculation unit 301 is not provided, for example, if the reliability of the extraction of the object is low, it may be assumed that an abnormality has occurred, and the inverse of the reliability of the DNN in the target image extraction unit 2 may be used as the anomaly score. When the anomaly score calculation unit 301 is provided, since it is generally difficult to collect a large amount of abnormal images, a method using a DNN that uses only normal images for learning and can output the degree of deviation from the normal state as an anomaly score can be considered for the image processing used by the anomaly score calculation unit 301. For example, an autoencoder (AE) or a generative adversarial network (GAN) that has learned to reconstruct normal images may be used to output an anomaly score based on a reconstruction error. However, in these methods, there is a possibility that a shadow that appears when conditions such as the installation location of the target object, surrounding structures, and shooting time are met may be erroneously detected as an anomaly. The shadow score calculation unit 302 may use, for example, a DNN that performs binary discrimination of whether an input image has a shadow or not, and may use the confidence of the likelihood of a shadow as the shadow score.

[0021] FIG. 4 is a table showing an example of the internal operation of the integrated judgment unit 303. The abnormality score is classified into two categories based on whether it is larger or smaller than a predetermined abnormality score threshold. Similarly, the shadow score is classified into two categories based on whether it is larger or smaller than a predetermined shadow score threshold. Therefore, the output of the abnormality score and the shadow score is classified into 2×2=4 patterns. Among them, in a pattern in which both the abnormality score and the shadow score are large, it is determined that the increase in the abnormality score is caused by the shadow, and the state of the object is determined to be normal. On the other hand, in a pattern in which the abnormality score is large but the shadow score is small, it is determined that the increase in the abnormality score is caused by the true abnormality factor, not the shadow, and the state of the object is determined to be abnormal. In a pattern in which the abnormality score is small, the state of the object is determined to be normal. The integrated judgment unit 303 outputs these state patterns as state judgment results. In addition, it outputs the classification result of the shadow score as a shadow judgment result.

[0022] The internal operation of the integrated judgment unit 303 shown in FIG. 4 may be adjusted as appropriate depending on whether the system is used for driving support or maintenance support. For example, in driving support, in order to reduce interference with normal driving, some non-detection is tolerated while reducing false positives is prioritized. On the other hand, in maintenance support, since it is undesirable to overlook anomalies that need to be repaired, some false positives are tolerated while reducing non-detection is prioritized. In these cases, the anomaly score threshold is set lower in maintenance support than in driving support. Furthermore, it is possible to adjust the above-mentioned tendency by setting the shadow score threshold higher or disabling it.

[0023] Depending on the conditions of the terrain and surrounding structures, such as urban areas, plains, and mountainous areas, the way shadows are generated may differ. Specifically, it is possible that shadows are generated almost all the time, and conversely, that shadows are hardly generated at all. The shadow score threshold may be adjusted appropriately to suit such conditions. In one example of the internal operation of the integrated judgment unit 303 shown in FIG. 4, the outputs of the anomaly score and the shadow score are classified into 2×2=4 patterns, but the number of patterns is not limited to this, and classification into, for example, 3×3=9 patterns may be used. 4, an example in which the state determination result and the shadow determination result are both output in the form of a binary value in the integrated determination unit 303 has been described, but these may be output in the form of a continuous value. For example, the anomaly score may be corrected by multiplying the anomaly score by the reciprocal of the shadow score so that the anomaly score is higher the greater the deviation from the normal state and the smaller the degree of the shadow, and this may be output as the state determination result. Also, a continuous value of the shadow score may be output as the shadow determination result. For factors other than shadows (such as sunlight reflection, rain, snow, etc.), a DNN may be set up for each factor. In this case, if any of the factors apply, the system will be judged as normal even if the anomaly score is high. A DNN that integrates multiple factors may also be set up.

[0024] [Operation sequence] An example of the process flow of this embodiment is shown in Figure 5. After startup, the system moves to a processing loop, and in system operation continuation confirmation step S1, the system monitors within the processing loop whether or not there is an instruction to end operation of the lineside facility inspection system of this embodiment. If operation is to be continued, in video frame acquisition step S2, a video frame captured by the camera is acquired. Then, in target image extraction step S3, a target image of a small area including the target is extracted from the video frame in order to focus on the target for inspection. Then, in judgment step S4, image processing is performed on the target image, and a status judgment result indicating the presence or absence of an abnormality in the target and a false detection factor judgment result indicating the presence or absence of a false detection factor are output.

[0025] FIG. 6 is an example of a process flow showing the internal process of the determination step S4. In the abnormality score calculation step S401, an abnormality score indicating the degree of deviation from the normal state of the object is calculated by image processing. Then, in the shadow score calculation step S402, a shadow score indicating the degree of a shadow generated on the object is calculated by image processing. Then, in the shadow score comparison step S403, the shadow score is compared with a predetermined shadow score threshold, and if the shadow score exceeds the shadow score threshold, the value of the shadow determination result is set to "shadow is present" in the shadow determination result setting step S404. On the other hand, if the shadow score is equal to or less than the shadow score threshold, the value of the shadow determination result is set to "shadow is absent" in the shadow determination result setting step S405. Then, in the abnormality score comparison step S406, the abnormality score is compared with a predetermined abnormality score threshold, and if the abnormality score exceeds the abnormality score threshold, the shadow determination result is confirmed in the shadow determination result confirmation step S407. If the shadow determination result is "no shadow," it is determined that the increase in the abnormality score is due to a true abnormality factor and not a shadow, and the value of the status determination result is set to abnormal in status determination result (abnormal) setting step S408. If the abnormality score is equal to or less than the abnormality score threshold in abnormality score comparison step S406, the status of the object is deemed normal, and the value of the status determination result is set to normal in status determination result (normal) setting step S409. Alternatively, if the shadow determination result is "there is a shadow" in shadow determination result confirmation step S407, it is determined that the increase in the abnormality score is due to a shadow, the status of the object is determined to be normal, and similarly, the value of the status determination result is set to normal in status determination result (normal) setting step S409.

[0026] As described above, the present invention makes it possible to realize a trackside facility inspection system that uses captured images of an object to detect abnormalities occurring in the object and distinguish between abnormalities and causes of erroneous detection, such as shadows. EXAMPLES

[0027] Fig. 7 is a diagram illustrating the overall configuration of a wayside facility inspection system according to an embodiment of the present invention. The purpose of this embodiment is to provide a wayside facility inspection system that can distinguish between anomalies and causes of false detection, such as shadows, by accumulating anomaly scores calculated from images of an object captured from the same viewpoint and analyzing the accumulated anomaly score history.

[0028] [System Configuration] The configuration and operation of a wayside facility inspection system according to an embodiment of the present invention will now be described. As shown in Fig. 7, the wayside facility inspection system of this embodiment is composed of a location information acquisition unit 4 and an anomaly score recording unit 5 in addition to the configuration shown in Fig. 1. The location information acquisition unit 4 is mounted on each mobile object. The anomaly score recording unit 5 is preferably installed in a server or the like capable of communicating with multiple mobile objects.

[0029] The operation of the lineside facility inspection system will be described with reference to FIG. 7. The description of the operation overlapping with the first embodiment will be omitted. The position information acquisition unit 4 acquires the current position information of the moving object using a global positioning system (GPS) or the like. When the moving object approaches the vicinity of the target object and the camera 1, the target image extraction unit 2, and the determination unit 3 operate to calculate the abnormality score of the target object, the abnormality score recording unit 5 links the abnormality score and the position information and records them, and outputs the abnormality score history of a predetermined first period in the past at the same position going back from the current time. That is, the abnormality score history is a history of abnormality scores obtained from images taken from the same viewpoint of the target object. The determination unit 3 analyzes the abnormality score history and outputs a state determination result and a false detection factor determination result.

[0030] Fig. 8 is a diagram showing an example of the internal configuration of the determination unit 3. The determination unit 3 is composed of an anomaly score calculation unit 301 and an anomaly score history analysis unit 304. Unlike the determination unit 3 in Fig. 3, the determination unit 3 in Fig. 8 does not include the shadow score calculation unit 302 and the integrated determination unit 303.

[0031] The anomaly score history analysis unit 304 analyzes the anomaly score history output by the anomaly score recording unit 5. Specifically, it outputs a state determination result indicating an anomaly when the anomaly score history has an offset component over a predetermined second period, and outputs an erroneous detection factor determination result indicating the presence of an erroneous detection factor when the difference obtained by subtracting the offset component from the current anomaly score exceeds a predetermined threshold.

[0032] FIG. 9A is an example of an anomaly score history when the state of the object is normal. For example, when the anomaly score increases due to a shadow cast on the object, a periodic fluctuation F901 occurs in the anomaly score history due to the periodicity of solar motion. The anomaly score has a small value during a time period when no shadow is cast. FIG. 9B is an example of an anomaly score history when the state of the object is abnormal. In addition to the periodic fluctuation F901 caused by the shadow, an offset component F902 caused by the true anomaly occurs. Therefore, it is possible to determine the state of the object based on the presence or absence of the offset component F902.

[0033] In other words, although the camera 1 used in this embodiment is a moving camera, it is possible to distinguish between false detection causes such as shadows and true abnormalities and determine the state of the object, as if the object were being observed at a fixed point by a fixed camera.

[0034] In the configuration of this embodiment, since the state is judged using the anomaly score history for a predetermined period, it is difficult to detect an anomaly immediately when it occurs. On the other hand, for example, if the anomaly score can be predicted from the periodicity of the solar movement, the anomaly score caused by a false detection factor such as a shadow that may occur at the current time is predicted, and the difference between the anomaly score at the current time and the predicted anomaly score is calculated, so that an anomaly can be detected immediately when it occurs. The prediction can be made with higher accuracy by correcting the predicted value according to the weather, season, etc.

[0035] In the configuration of this embodiment, since the state is judged using the anomaly score history for a predetermined period, it is desirable to accumulate anomaly scores evenly and frequently on the time axis. For this purpose, it is useful to configure the anomaly score recording unit 5 to be shared among multiple mobile objects, and to integrate and use the anomaly scores obtained by multiple mobile objects.

[0036] As described above, the present invention makes it possible to realize a line equipment inspection system that can distinguish between abnormalities and causes of false detection, such as shadows, by accumulating anomaly scores determined from images of an object taken from the same viewpoint and analyzing the accumulated anomaly score history. Here, the case where a shadow is identified as a cause of false positive detection is exemplified, but the present invention can also be applied to the case where rain or snow is identified as a cause of false positive detection. An increase in the anomaly score caused by rain or snow will disappear when the rain or snow stops over time. Therefore, even if a temporary increase in the anomaly score disappears over time, if there is an offset, it is possible to determine that this offset indicates an anomaly. EXAMPLES

[0037] 10 is a diagram illustrating the overall configuration of a wayside facility inspection system according to an embodiment of the present invention. The purpose of this embodiment is to provide a wayside facility inspection system capable of automatically generating high-quality learning data for image processing used by the shadow score calculation unit in the first embodiment.

[0038] [System Configuration] The configuration and operation of a wayside facility inspection system according to an embodiment of the present invention will be described below. As shown in Fig. 10, the wayside facility inspection system according to this embodiment is composed of a learning data generation unit 6 and a learning data recording unit 7 in addition to the configuration shown in Fig. 1.

[0039] The operation of the lineside facility inspection system will be described with reference to FIG. 10. Descriptions of operations that overlap with those in the first and second embodiments will be omitted. The learning data generation unit 6 automatically performs teacher assignment for the target image output by the target image extraction unit 2 based on the false detection cause judgment result output by the judgment unit 3, and the shadow judgment result if the false detection cause is a shadow. Specifically, when the shadow score calculation unit in the first embodiment uses a DNN that performs binary discrimination of whether an input image has a shadow or not, for example, teacher data of "1" is assigned to a target image with a shadow and "0" is assigned to a target image without a shadow. The learning data assigned by the learning data generation unit 6 is recorded in the learning data recording unit 7.

[0040] In training a DNN that performs binary discrimination between shadow presence and absence, it is desirable to make the conditions other than shadows as consistent as possible in both image groups with and without shadows in order to minimize the influence of factors other than shadows on the discrimination results. According to the configuration of this embodiment, it is possible to collect a large number of images of the object taken from the same viewpoint at different times, that is, images that differ only in the presence or absence of false detection factors such as shadows. Therefore, according to the configuration of this embodiment, it is possible to automatically generate high-quality training data in DNN training.

[0041] As described above, the present invention can realize a wayside facility inspection system capable of automatically generating high-quality learning data for image processing used by the shadow score calculation unit in the first embodiment. EXAMPLES

[0042] 11 is a diagram illustrating the overall configuration of a wayside facility inspection system according to an embodiment of the present invention. The object of this embodiment is to provide a wayside facility inspection system capable of determining the installation state of an object.

[0043] [System Configuration] The configuration and operation of a wayside facility inspection system according to an embodiment of the present invention will be described below. As shown in Fig. 11, the wayside facility inspection system of this embodiment is composed of a location information acquisition unit 4 and an installation state determination unit 8 in addition to the components shown in Fig. 1.

[0044] The operation of the lineside facility inspection system will be described with reference to FIG. 11. The description of the operation overlapping with the first to third embodiments will be omitted. When the target image extraction unit 2 extracts a target image of a small area including an object from the photographed image, it outputs extraction information together with the target image. The extraction information includes information on whether the target image was extracted successfully and, if extracted, coordinate information of the extracted image in the photographed image. The installation state determination unit 8 has information on the installation state of the object, specifically, information on the installation position and height of the object in the real world, stored as a database. The installation state determination unit 8 can also refer to data indicating the dimensions of the moving body and the mounting position of the camera 1. The installation state determination unit 8 refers to the database based on the position information output by the position information acquisition unit 4, and calculates the appearance area of ​​the object in the photographed image. When the moving body is approaching the position where the object is supposed to appear, if the object is extracted from the appearance area of ​​the photographed image, the installation state determination result is determined to be normal. Conversely, if the moving body approaches the position where the target object is supposed to appear but the target image cannot be extracted from the captured image, or even if it is extracted, the extracted coordinate information is far from the appearance area, the installation state determination result is output as an abnormality.

[0045] In comparing the extracted coordinate information with the appearance region, for example, the center coordinates of each may be calculated and the Euclidean distance between them may be used. Alternatively, the degree of overlap between the two may be evaluated using Intersection over Union (IoU).

[0046] Examples of the installation state determination result being abnormal include obscuration by vegetation, large deformation, tilt, and loss. 12A is an example of a captured image of the front of a railroad car. In a captured image F201 acquired by the camera 1, a tree F1201 near a sign F202 is overgrown and covers the sign F202, making it impossible to detect the object within the object appearance area F1202. In this case, the installation state determination unit 8 outputs the installation state determination result of the object as abnormal.

[0047] 12B is an example of a captured image of the front of a railcar. In a captured image F201 acquired by the camera 1, a sign F202 is tilted, and an object cannot be detected within an object appearance area F1202. In this case, the installation state determination unit 8 also outputs the installation state determination result of the object as abnormal.

[0048] As described above, the present invention makes it possible to realize a railway line facility inspection system capable of determining the installation state of an object. Specifically, the disclosed condition determination device includes a camera 1 mounted on a moving body and configured to capture an image of the driving space, a target image extraction unit 2 configured to extract a target image of a small area including an object for condition determination from the captured image, and a determination unit 3 configured to perform image processing on the target image and output a condition determination result indicating the presence or absence of an abnormality in the object and an erroneous detection factor determination result indicating the presence or absence of an erroneous detection factor. With this configuration, the state determination device can use captured images of the object to detect abnormalities occurring in the object and distinguish between abnormalities and causes of erroneous detection, such as shadows.

[0049] In addition, the judgment unit 3 judges one or more factors of erroneous detection, and when an erroneous detection factor judgment result indicating the presence of any of the factors of erroneous detection is obtained, the judgment unit 3 suppresses output of a state judgment result indicating that an abnormality exists in the target object. This operation prevents a result indicating an abnormality from being output based on the cause of a false detection, thereby preventing unnecessary maintenance work from occurring and avoiding a decrease in the reliability of alerts.

[0050] Furthermore, when the cause of erroneous detection is a shadow, the judgment unit 3 may be configured to include an abnormality score calculation unit 301 which receives the target image and calculates, by image processing, an abnormality score indicating the degree of deviation from the normal state of the target, a shadow score calculation unit 302 which receives the target image and calculates, by image processing, a shadow score indicating the degree of shadow cast on the target, and an integrated judgment unit 303 which outputs a shadow judgment result indicating the presence of a shadow when the shadow score is greater than a predetermined shadow score threshold, and outputs a state judgment result indicating an abnormality when the abnormality score is greater than the predetermined abnormality score threshold and the shadow judgment result does not indicate the presence of a shadow. With this configuration, highly accurate state determination can be achieved in real time for a single moving object.

[0051] In addition, the device may further include a location information acquisition unit 4 that acquires current location information of the moving object when the cause of the erroneous detection is a shadow, and the judgment unit 3 associates the location information with the abnormality score output by the abnormality score calculation unit and records them in the abnormality score recording unit 5, and acquires an abnormality score history for the object for a predetermined first period in the past from the abnormality score recording unit 5 based on the location information, and the judgment unit 3 may further include an abnormality score history analysis unit 304 that outputs a state judgment result that is abnormal when the abnormality score history has an offset component over a predetermined second period, and outputs a shadow judgment result that a shadow is present when the difference obtained by subtracting the offset component from the current abnormality score exceeds a predetermined threshold value. In this configuration, the abnormality scores from a large number of moving objects are statistically utilized to easily determine the state.

[0052] Furthermore, for the purpose of learning image processing for calculating a shadow score, the configuration may include a learning data generation unit 6 that uses the shadow determination result to add teacher information to the target image, and a learning data recording unit 7 that records the learning data. In this configuration, learning is performed using the judgment results, and the accuracy of state judgment can be improved.

[0053] The device may further include a position information acquisition unit 4 that acquires current position information of the moving body, and an installation state determination unit 8 that determines the installation state of the object. When the target image extraction unit 2 extracts a target image of a small area including the object from the captured image, the target image extraction unit 2 outputs extraction information including information on whether the target image was extracted successfully and, if extracted, the extracted coordinate information in the captured image together with the target image. The installation state determination unit 8 may have a database of information on the installation state including the installation position and / or height of the object, calculate the appearance area of ​​the object in the captured image based on the position information output by the position information acquisition unit 4 and the database, compare the appearance area with the extraction information, and output an installation state determination result indicating normal when the object is extracted from within the appearance area of ​​the captured image, and abnormal when the target image cannot be extracted from the captured image even though the object should appear, or the extracted coordinate information is away from the appearance area. This configuration also allows for the determination of positional anomalies of the object, which can be done independently of the determination of the object's appearance, such as stains, shadows, etc.

[0054] The moving body is, for example, a railroad car, and the object of state determination is, for example, a sign or a traffic light. If the camera 1 is installed on a railway vehicle, traveling on the rails contributes to identifying the position of the camera 1. Therefore, in the second to fourth embodiments, images from the same viewpoint can be easily collected. Also, in the fourth embodiment, the "position where the object is supposed to appear" can be easily and highly accurately obtained. For objects such as signs and traffic lights, where an abnormality in appearance directly impairs their function, it is particularly important to determine their condition.

[0055] In the above-described embodiment, the present invention has been mainly applied to a wayside facility inspection system for automatically determining the state (normal / abnormal) of wayside facilities to realize railway operation support and unmanned operation, but the present invention may be applied to other purposes. For example, the present invention may be applied to a driving support system or a maintenance support system for automobiles running on highways, aircraft running on runways, drones flying along power lines, etc.

[0056] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0057] In addition, each of the above configurations may be partially or entirely configured as hardware, or may be configured to be realized by executing a program on a processor. In addition, the control lines and information lines are shown as those considered necessary for the explanation, and not all of the control lines and information lines in the product are shown. In reality, it may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0058] 1. Camera 2. Target image extraction section 3...Judgment section 4... Location information acquisition section 5. Anomaly score recording section 6. Learning data generation unit 7. Learning data recording section 8. Installation status determination section 301: Anomaly score calculation unit 302 Shadow score calculation unit 303... Integrated judgment section 304: Anomaly score history analysis unit F201...Photo taken F202... Sign F203....dirt F204...Target image F205...Shadow F901... Periodic fluctuation F902: Offset component F1201···Wood F1202: Area where object appears S1: System operation continuation confirmation step S2: Image frame acquisition step S3: Target image extraction step S4: Judgment step S401: Anomaly score calculation step S402: Shadow score calculation step S403: Shadow score comparison step S404: Shadow determination result (shadow exists) setting step S405: Shadow determination result (no shadow) setting step S406: Anomaly score comparison step S407: Shadow detection result confirmation step S408: Status judgment result (abnormal) setting step S409: Status judgment result (normal) setting step

Claims

1. A camera mounted on the vehicle for capturing images of the vehicle's travel space; a target image extraction unit that extracts a target image of a small region including an object to be subjected to state determination from the captured image; a determination unit that performs image processing on the target image and outputs a state determination result indicating the presence or absence of an abnormality in the target object and an erroneous detection factor determination result indicating the presence or absence of an erroneous detection factor; a location information acquisition unit that acquires current location information of the moving object; Equipped with The cause of the false detection is a shadow, The determination unit is an anomaly score calculation unit that receives the target image and calculates an anomaly score indicating a deviation from a normal state of the target object by image processing; The determination unit is causing an abnormality score recording unit to record the position information and the abnormality score output by the abnormality score calculation unit in association with each other, and acquiring an abnormality score history for the object during a predetermined first period in the past from the abnormality score recording unit based on the position information; The determination unit is An abnormality score history analysis unit outputs a state determination result indicating an abnormality when the abnormality score history has an offset component over a predetermined second period, and outputs a shadow determination result indicating an influence when a difference obtained by subtracting the offset component from the current abnormality score exceeds a predetermined threshold. A state determination device comprising:

2. The state determination device according to claim 1 , The determination unit determines one or more factors of erroneous detection, and when a result of the determination of factors of erroneous detection is obtained indicating that any of the factors of erroneous detection is present, suppresses output of a state determination result indicating that an abnormality exists in the object. A state determination device comprising:

3. The state determination device according to claim 1 , The determination unit is a shadow score calculation unit that receives the target image and calculates a shadow score indicating the degree of a shadow generated on the target object by image processing; an integrated judgment unit that outputs a shadow judgment result indicating the presence of a shadow when the shadow score is greater than a predetermined shadow score threshold, and outputs a state judgment result indicating the presence of an abnormality when the abnormality score is greater than the predetermined abnormality score threshold and the shadow judgment result indicates that a shadow is not present. A state determination device comprising:

4. The state determination device according to claim 3, a learning data generation unit that uses the shadow determination result to add teacher information to the target image for the purpose of learning image processing for calculating the shadow score; A learning data recording unit that records the learning data. A state determination device comprising:

5. The state determination device according to claim 1 , Further comprising an installation state determination unit for determining an installation state of the object, the target image extraction unit, when extracting a target image of a small region including an object from the photographed image, outputs extraction information including information on whether the target image was extracted successfully and, if extracted, coordinate information of the extracted target image in the photographed image together with the target image; The installation state determination unit has a database of information on the installation state including the installation position and / or height of the object, calculates an appearance area of ​​the object in the photographed image based on the position information output by the position information acquisition unit and the database, compares the appearance area with the extraction information, and outputs an installation state determination result indicating that the result is normal when the object is extracted from within the appearance area of ​​the photographed image, and that the result is abnormal when the object image cannot be extracted from the photographed image even though the object should appear or when the extracted coordinate information is away from the appearance area. A state determination device comprising:

6. 2. The state determination device according to claim 1, wherein the moving body is a railway vehicle.

7. 2. The state determination device according to claim 1, wherein the object of state determination is a sign or a traffic light.

8. The state determination device is An image frame acquisition step of acquiring an image frame of a traveling space of a moving object; a target image extraction step of extracting a target image of a small area including a target object for state determination from the video frame; a determination step of performing image processing on the target image and outputting a state determination result indicating the presence or absence of an abnormality in the target object and an erroneous detection factor determination result indicating the presence or absence of an erroneous detection factor; a position information acquisition step of acquiring current position information of the moving object; Equipped with The cause of the false detection is a shadow, an abnormality score calculation step in which the target image is input and an abnormality score indicating a deviation degree from a normal state of the target object is calculated by image processing; a step of associating the position information with the abnormality score output by the abnormality score calculation step and recording the abnormality score in an abnormality score recording unit, and acquiring an abnormality score history for the object during a predetermined first period in the past from the abnormality score recording unit based on the position information; an anomaly score history analysis step of outputting a state determination result indicating an abnormality when the anomaly score history has an offset component over a predetermined second period, and outputting a shadow determination result indicating an influence when a difference obtained by subtracting the offset component from the current anomaly score exceeds a predetermined threshold; Further, A state determination method comprising:

Images