State determination device and state determination method

Abstract

To provide a state determination device and method capable of detecting abnormality occurring in an object and identifying the abnormality and an erroneous detection factor such as a shadow by using a photographic image of the object.SOLUTION: A state determination device is used, including a camera mounted on a mobile body to acquire a photographic image in a travel space, an object image extraction part for extracting an object image of a small area including an object of state determination in the photographic image, and a determination part for performing image processing on the object image and outputting a state determination result showing the presence / absence of abnormality of the object and an erroneous detection factor determination result showing the presence / absence of an erroneous detection factor.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a state determination device and a state determination method.

Background Art

[0002] As background art in this technical field, there is JP 2021-124751 (Patent Document 1). In this publication, it is described that "The imaging unit is provided on a railway vehicle and images railway facilities provided along the running route of the railway vehicle. ~ Omitted ~ The moving image captured by the imaging unit is associated with the vehicle position, and the moving image is encoded. ~ Omitted ~ A motion vector between a first image included in the decoded first moving image and a second image included in the decoded second moving image is calculated, and an abnormality of the railway facilities is diagnosed based on the motion vector."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] <� As a main use of the state determination device and the state determination method, for example, there is a roadside facility inspection system for automatically determining the state (normal / abnormal) of roadside facilities for the purpose of railway driving support and realization of unmanned driving. A roadside facility inspection system that mounts a camera on a railway vehicle and uses an analysis device inside or outside the moving body to perform state determination from a captured image is useful.

[0005] Patent Document 1 states that "the imaging unit is installed on a railway vehicle and images railway equipment installed along the railway vehicle's travel path. ~[omitted]~ The imaging unit associates the captured video with the vehicle's position and encodes the video. ~[omitted]~ It calculates a motion vector between the first image included in the decoded first video and the second image included in the decoded second video, and diagnoses abnormalities in the railway equipment based on the motion vector." However, detecting abnormalities that do not appear in the motion vector, such as dirt on the equipment surface, is difficult and there is room for improvement. In particular, how to distinguish between shadows on the equipment surface and abnormalities is one of the important points. For example, if it is not possible to distinguish between a shadow on the equipment surface and dirt, and the shadow is judged as an abnormality, unnecessary maintenance work will be generated. Also, frequent unnecessary alerts will lead to a decrease in confidence in the alerts. The present invention aims to provide a state determination device and a state determination method that can use captured images of an object to detect abnormalities occurring in the object, and to distinguish between abnormalities and false detection factors such as shadows. [Means for solving the problem]

[0006] To solve the above objective, one representative state determination device of the present invention is characterized by comprising: a camera mounted on a moving body that acquires images of the travel space; a target image extraction unit that extracts a target image of a small region containing the object to be determined from the captured image; and a determination unit that performs image processing on the target image and outputs a state determination result indicating whether or not there is an abnormality in the object and a false detection factor determination result indicating whether or not there is a false detection factor. Furthermore, one representative state determination method of the present invention is characterized in that the state determination device comprises: a video frame acquisition step of acquiring video frames of the travel space of a moving object; a target image extraction step of extracting a target image of a small region containing the object to be determined in the video frame; and a determination step of performing image processing on the target image and outputting a state determination result indicating whether or not there is an abnormality in the object and a false detection factor determination result indicating whether or not there is a false detection factor. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a condition determination device and method that can use captured images of an object to detect abnormalities occurring in the object, and to distinguish between abnormalities and factors causing false detection, such as shadows. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates the overall configuration of a railway line equipment inspection system according to the first embodiment of the present invention. [Figure 2A] This is a schematic diagram showing an example of a photograph taken from the front of a railway vehicle. [Figure 2B] This is a schematic diagram showing an example of the target image. [Figure 2C] This is a schematic diagram showing an example of a photograph taken from the front of a railway vehicle. [Figure 2D] This is a schematic diagram showing an example of the target image. [Figure 3] This figure illustrates the internal configuration of the determination unit in the railway line equipment inspection system according to the first embodiment of the present invention. [Figure 4] This table shows an example of the internal operation of the integrated determination unit in the railway line equipment inspection system according to the first embodiment of the present invention. [Figure 5] This is a flowchart illustrating an example of internal processing in the railway line equipment inspection system according to the first embodiment of the present invention. [Figure 6] This flowchart illustrates an example of the internal processing of the determination step in the railway line equipment inspection system according to the first embodiment of the present invention. [Figure 7] This diagram illustrates the overall configuration of a railway line equipment inspection system according to a second embodiment of the present invention. [Figure 8] This figure illustrates the internal configuration of the determination unit in a railway line equipment inspection system according to a second embodiment of the present invention. [Figure 9A] This is a schematic diagram showing an example of an abnormality score history when the object is in a normal state. [Figure 9B] This is a schematic diagram illustrating an example of an abnormality score history when the object's condition is abnormal. [Figure 10]This is a diagram for explaining the overall configuration of the line-side facility inspection system according to the third embodiment of the present invention. [Figure 11] This is a diagram for explaining the overall configuration of the line-side facility inspection system according to the fourth embodiment of the present invention. [Figure 12A] This is a schematic diagram showing an example of a captured image in front of a railway vehicle. [Figure 12B] This is a schematic diagram showing an example of a captured image in front of a railway vehicle.

Embodiments for Carrying out the Invention

[0009] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings.

Examples

[0010] In this example, a line-side facility inspection system, which is the main application of the state determination device and method of the present invention, will be described. The line-side facility inspection system uses a captured video in front of a railway vehicle to determine whether there is an abnormality in the line-side facilities, and aims to be useful for facility management and repair planning.

[0011] FIG. 1 is a diagram for explaining the overall configuration of a line-side facility inspection system according to an embodiment of the present invention. This embodiment aims to provide a line-side facility inspection system that uses a captured image of an object to detect an abnormality occurring in the object and can identify factors causing false detections such as an abnormality and a shadow.

[0012] [System Configuration] The configuration and operation of a line-side facility inspection system according to an embodiment of the present invention will be described. As shown in FIG. 1, the line-side facility inspection system of this embodiment is composed of a camera 1, an object image extraction unit 2, and a determination unit 3. The operation of the railway line equipment inspection system will be explained using Figure 1. Camera 1 is mounted on a moving railway vehicle and acquires images of the travel space, mainly in front of the railway vehicle. Target image extraction unit 2 extracts a small area containing the target object from the captured image in order to focus on the target object for inspection. Judgment unit 3 performs image processing on the target image and outputs a state judgment result indicating whether or not there is an abnormality in the target object and a false detection factor judgment result indicating whether or not there is a false detection factor.

[0013] The objects targeted by the railway line equipment inspection system include, for example, signs, signals, and ground beacons. The abnormalities detected include, for example, dirt, damage, rust, peeling paint, and concealment by vegetation. The camera 1 can be mounted, for example, inside or outside the train car near the driver's cab, or even on the roof, to capture the entire front of the train car and target a wide range of equipment. Alternatively, mounting it on the side of the train car allows for more detailed photography from a closer position. Furthermore, by mounting a camera that captures the rear of the train car, not just the front, it is possible to target equipment along the opposing track.

[0014] The aforementioned camera 1 can inspect objects at a distance by using a telephoto lens or a high-resolution image sensor. In particular, when this system is operated while a railway vehicle is in operation and it is desirable to be able to inspect objects from a distance in order to detect abnormalities that may interfere with operation at an early stage.

[0015] The target image extraction unit 2 may, for example, extract the target image by recognizing the characters using Optical Character Recognition (OCR) if the target object has predetermined characters, such as a speed limit sign. Alternatively, it may extract the target image by using image processing techniques such as Deep Neural Network (DNN) to output information on the position and size of the target object from the captured image, or by detecting pixels in which the target object exists through semantic segmentation.

[0016] False detection factors include, for example, if the shooting time is daytime, shadows caused by sunlight and surrounding structures, reflection of sunlight, rain, snow, etc., and if the shooting time is nighttime, shadows caused by the headlights of cars driving along the railway line and surrounding structures, and reflection of the aforementioned headlights, etc. Hereafter, shadows caused by sunlight and surrounding structures will be described as false detection factors, but the present invention can be applied to other false detection factors with a similar configuration and method.

[0017] Figure 2A is an example of a photograph taken from the front of a railway vehicle. The photograph F201 acquired by camera 1 shows a sign F202. There is dirt F203 on the sign F202. The target image extraction unit 2 detects the sign F202 from the photograph F201 and outputs the target image F204 shown in Figure 2B. The judgment unit 3 detects the dirt F203 from the target image F204 and outputs a state judgment result indicating an abnormality.

[0018] Figure 2C is an example of a photograph taken from the front of a railway vehicle. A shadow F205 is cast on sign F202 due to nearby power lines. The presence of shadow F205 is not abnormal for the object, but it can be a cause of false abnormality detection. Therefore, even if the judgment unit 3 can determine that shadow F205 is abnormal from the target image F204 shown in Figure 2D, it recognizes the presence of the shadow through separate processing, and based on these results, it makes an integrated judgment and outputs the status judgment result as normal.

[0019] Figure 3 shows an example of the internal configuration of the judgment unit 3. The judgment unit 3 consists of an abnormal score calculation unit 301, a shadow score calculation unit 302, and an integrated judgment unit 303. The abnormality score calculation unit 301 calculates an abnormality score indicating the degree of deviation from the normal state of the object through image processing. The shadow score calculation unit 302 calculates a shadow score indicating the degree of shadow cast on the object through image processing. The integrated determination unit 303 uses the abnormality score and the shadow score, and after considering the increase in the abnormality score caused by shadows, outputs a state determination result and a shadow determination result.

[0020] If the abnormality score calculation unit 301 is not provided, for example, if the confidence level of object extraction is low, it may be considered that an abnormality has occurred, and the reciprocal of the confidence level of the DNN in the target image extraction unit 2 may be used as the abnormality score. If an anomaly score calculation unit 301 is provided, the image processing used by the anomaly score calculation unit 301 can be considered to be a method that uses only normal images for training and outputs the degree of deviation from the normal state as an anomaly score, since it is generally difficult to collect a large number of abnormal images. For example, an Autoencoder (AE) or Generative Adversarial Network (GAN) trained to reconstruct normal images can be used, and an anomaly score can be output based on the reconstruction error. However, with these methods, there is a possibility of misdetecting shadows that occur when conditions such as the location of the object, surrounding structures, and time of shooting are met as anomalies. The shadow score calculation unit 302 may, for example, use a DNN that performs binary classification of whether the input image has a shadow or not, and use the confidence level of the likelihood of a shadow being present as the shadow score.

[0021] Figure 4 is a table showing an example of the internal operation of the integrated determination unit 303. The abnormal score is classified into two categories based on its magnitude to a predetermined abnormal score threshold. Similarly, the shadow score is classified into two categories based on its magnitude to a predetermined shadow score threshold. Therefore, the output of the abnormal score and shadow score is classified into 2 x 2 = 4 patterns. Of these, patterns where both the abnormal score and shadow score are large are judged to be due to shadows, and the state of the object is considered normal. On the other hand, patterns where the abnormal score is large but the shadow score is small are judged to be due to a true abnormal factor, not shadows, and the state of the object is considered abnormal. Patterns where the abnormal score is small are considered to be the state of the object. The integrated determination unit 303 outputs these state patterns as state determination results. It also outputs the classification result of the shadow score as the shadow determination result.

[0022] The internal operation of the integrated judgment unit 303 shown in Figure 4 may be adjusted as appropriate depending on whether the system is used for driver assistance or maintenance assistance. For example, in driver assistance, priority is given to reducing false detections, while allowing some false detections, in order to minimize disruption to normal operation. On the other hand, in maintenance assistance, it is undesirable to miss abnormalities that require repair, so priority is given to reducing false detections, while allowing some false detections. In these cases, the abnormality score threshold is set lower in maintenance assistance compared to driver assistance. Furthermore, it is possible to adjust the system to exhibit the aforementioned tendencies by setting the shadow score threshold higher or disabling it.

[0023] Depending on the topography and surrounding structures, such as in urban areas, plains, or mountainous regions, the way shadows are cast can vary. Specifically, there may be cases where shadows are almost always present, or conversely, cases where shadows are almost nonexistent. The shadow score threshold may be adjusted as appropriate to suit these conditions. In the example of the internal operation of the integrated judgment unit 303 shown in Figure 4, the output of the abnormal score and shadow score is classified into 2 x 2 = 4 patterns, but the number of patterns is not limited to this, and for example, 3 x 3 = 9 patterns of classification may be used. Figure 4 illustrates an example where the integrated determination unit 303 outputs both the state determination result and the shadow determination result in binary format. However, these may also be output in continuous format. For example, the abnormality score may be corrected by multiplying it by the reciprocal of the shadow score so that it becomes higher the greater the deviation from the normal state and the smaller the degree of shadowing, and this may be output as the state determination result. Alternatively, the continuous shadow score may be output as the shadow determination result. For factors other than shadow (such as reflection of sunlight, rain, and snow), a separate DNN may be established for each factor. In this case, if any of the factors are present, the result will be judged as normal even if the abnormality score is high. Alternatively, a DNN that integrates multiple factors may be established.

[0024] [Operation Sequence] An example of the processing flow of this embodiment is shown in Figure 5. After startup, the system moves into a processing loop, and in the system operation continuation confirmation step S1, it monitors for the presence or absence of an operation termination command for the railway equipment inspection system of this embodiment within the processing loop. If operation continues, in the video frame acquisition step S2, it acquires video frames captured by the camera. Then, in the target image extraction step S3, in order to focus on the target object and perform inspection, a target image of a small region containing the target object is extracted from the video frame. Then, in the judgment step S4, image processing is performed on the target image, and a state judgment result indicating whether or not there is an abnormality in the target object and a false detection factor judgment result indicating whether or not there is a false detection factor are output.

[0025] Figure 6 shows an example of a processing flow illustrating the internal processing of the judgment step S4. In the abnormal score calculation step S401, an abnormal score indicating the degree of deviation from the normal state of the object is calculated by image processing. Then, in the shadow score calculation step S402, a shadow score indicating the degree of shadow cast on the object is calculated by image processing. Then, in the shadow score comparison step S403, the shadow score is compared with a predetermined shadow score threshold. If the shadow score exceeds the shadow score threshold, the shadow judgment result is set to "shadow present" in the shadow judgment result setting step S404. Conversely, if the shadow score is less than or equal to the shadow score threshold, the shadow judgment result is set to "no shadow" in the shadow judgment result setting step S405. Then, in the abnormal score comparison step S406, the abnormal score is compared with a predetermined abnormal score threshold. If the abnormal score exceeds the abnormal score threshold, the shadow judgment result is confirmed in the shadow judgment result confirmation step S407. If the shadow detection result is "no shadow," it is determined that the increase in the abnormal score is due to a true abnormal factor and not a shadow, and the value of the state determination result is set to abnormal in the state determination result (abnormal) setting step S408. If the abnormal score is below the abnormal score threshold in the abnormal score comparison step S406, the state of the object is considered normal, and the value of the state determination result is set to normal in the state determination result (normal) setting step S409. Alternatively, if the shadow detection result is "shadow present" in the shadow detection result confirmation step S407, it is determined that the increase in the abnormal score is due to a shadow, the state of the object is considered normal, and similarly, the value of the state determination result is set to normal in the state determination result (normal) setting step S409.

[0026] As described above, the present invention makes it possible to realize a railway line equipment inspection system that uses captured images of an object to detect abnormalities occurring in the object, and also distinguishes between abnormalities and false detection factors such as shadows. [Examples]

[0027] Figure 7 is a diagram illustrating the overall configuration of a railway line equipment inspection system, which is one embodiment of the present invention. The purpose of this embodiment is to provide a railway line equipment inspection system that can distinguish between abnormalities and false detection factors such as shadows by accumulating abnormality scores obtained from images taken from the same viewpoint of the target object and analyzing the accumulated abnormality score history.

[0028] [System Configuration] The configuration and operation of a railway-side equipment inspection system, which is one embodiment of the present invention, will now be described. As shown in Figure 7, the railway-side equipment inspection system of this embodiment consists of a location information acquisition unit 4 and an abnormal score recording unit 5, in addition to the configuration shown in Figure 1. The location information acquisition unit 4 is mounted on each mobile unit. The abnormal score recording unit 5 is preferably installed on a server or the like that can communicate with multiple mobile units.

[0029] The operation of the railway line equipment inspection system will be explained using Figure 7. Explanations of operations that overlap with those in Example 1 will be omitted. The location information acquisition unit 4 acquires the current location information of the moving object using a Global Positioning System (GPS) or the like. When the moving object approaches the vicinity of the target object, and the camera 1, target image extraction unit 2, and determination unit 3 operate to calculate the abnormality score of the target object, the abnormality score recording unit 5 records the abnormality score linked to the location information and outputs the abnormality score history for a predetermined first period in the past at the same location, working backward from the current time. That is, the abnormality score history is a history of abnormality scores obtained from images taken from the same viewpoint of the target object. The determination unit 3 analyzes the abnormality score history and outputs the state determination result and the false detection factor determination result.

[0030] Figure 8 shows an example of the internal configuration of the judgment unit 3. The judgment unit 3 consists of an abnormal score calculation unit 301 and an abnormal score history analysis unit 304. Unlike the judgment unit 3 in Figure 3, the judgment unit 3 in Figure 8 does not have a shadow score calculation unit 302 and an integrated judgment unit 303.

[0031] The abnormal score history analysis unit 304 analyzes the abnormal score history output by the abnormal score recording unit 5. Specifically, it outputs a state determination result indicating an abnormality if the abnormal score history has an offset component over a predetermined second period, and outputs a false detection factor determination result indicating a false detection factor if the difference obtained by subtracting the offset component from the current abnormal score exceeds a predetermined threshold.

[0032] Figure 9A shows an example of an anomaly score history when the object is in a normal state. For example, if the anomaly score increases due to a shadow cast on the object, a periodic fluctuation F901 occurs in the anomaly score history due to the periodicity of solar motion. The anomaly score is small during periods when no shadow is cast. Figure 9B shows an example of an anomaly score history when the object is in an abnormal state. In addition to the periodic fluctuation F901 caused by the shadow, an offset component F902 caused by the true anomaly occurs. Therefore, it is possible to determine the state of the object based on the presence or absence of the offset component F902.

[0033] In other words, although camera 1 used in this embodiment is a mobile camera, it is possible to distinguish between false detection factors such as shadows and true anomalies, and to determine the state of the object, as if the object were being observed at a fixed point with a stationary camera.

[0034] In this embodiment, since the state is determined using anomaly score history over a predetermined period, immediate detection of anomalies is difficult. On the other hand, if the anomaly score can be predicted from, for example, the periodicity of solar motion, immediate detection of anomalies is possible by predicting the anomaly score caused by false detection factors such as shadows that may occur at the current time, and calculating the difference between the anomaly score at the current time and the predicted anomaly score. The prediction can be made more accurate by correcting the predicted value according to the weather, season, etc.

[0035] In this embodiment, since the state is determined using the abnormal score history over a predetermined period, it is desirable to accumulate abnormal scores evenly and frequently over time. To achieve this, the abnormal score recording unit 5 is configured to be shared among multiple mobile units, and it is useful to integrate and utilize the abnormal scores acquired by multiple mobile units.

[0036] As described above, the present invention enables the realization of a railway line equipment inspection system that can distinguish between abnormalities and false detection factors such as shadows by accumulating abnormality scores obtained from images taken from the same viewpoint of the object and analyzing the accumulated abnormality score history. Here, we have illustrated the case where shadows are identified as a false positive factor, but this method can also be applied to cases where rain or snow is identified as a false positive factor. The increase in the anomaly score caused by rain or snow will resolve as time passes and the rain or snow stops. Therefore, even if the temporary increase in the anomaly score resolves as time passes, if there is an offset, it is possible to determine that this offset indicates an anomaly. [Examples]

[0037] Figure 10 is a diagram illustrating the overall configuration of a railway line equipment inspection system, which is one embodiment of the present invention. The purpose of this embodiment is to provide a railway line equipment inspection system that can automatically generate high-quality training data for image processing used by the shadow score calculation unit in Embodiment 1.

[0038] [System Configuration] The configuration and operation of a railway line equipment inspection system, which is one embodiment of the present invention, will now be described. As shown in Figure 10, the railway line equipment inspection system of this embodiment consists of a learning data generation unit 6 and a learning data recording unit 7, in addition to the configuration shown in Figure 1.

[0039] The operation of the railway line equipment inspection system will be explained using Figure 10. Explanations of operations that overlap with those in Examples 1 and 2 will be omitted. The learning data generation unit 6 automatically performs training on the target images output by the target image extraction unit 2, based on the false detection factor determination result output by the determination unit 3, and the shadow determination result if the false detection factor is a shadow. Specifically, if the shadow score calculation unit in Example 1 uses a DNN that performs binary discrimination of whether the input image has a shadow or not, it assigns training data of "1" to target images with shadows and "0" to target images without shadows. The training data trained by the learning data generation unit 6 is recorded in the learning data recording unit 7.

[0040] In training a DNN that performs binary classification of whether an image has a shadow or not, it is desirable to minimize the influence of factors other than shadows on the classification result by making the conditions other than shadows as similar as possible in both the shadowed and shadowless image sets. With the configuration of this embodiment, it is possible to collect a large number of images of the object from the same viewpoint but at different times, that is, images that differ only in the presence or absence of false detection factors such as shadows. Therefore, with the configuration of this embodiment, it is possible to automatically generate high-quality training data in DNN training.

[0041] As described above, the present invention makes it possible to realize a railway line equipment inspection system that can automatically generate high-quality training data for image processing used by the shadow score calculation unit in Example 1. [Examples]

[0042] Figure 11 is a diagram illustrating the overall configuration of a railway line equipment inspection system, which is one embodiment of the present invention. The purpose of this embodiment is to provide a railway line equipment inspection system capable of determining the installation status of an object.

[0043] [System Configuration] The configuration and operation of a railway-side equipment inspection system, which is one embodiment of the present invention, will now be described. As shown in Figure 11, the railway-side equipment inspection system of this embodiment consists of a location information acquisition unit 4 and an installation status determination unit 8, in addition to the configuration shown in Figure 1.

[0044] The operation of the railway line equipment inspection system will be explained using Figure 11. Explanations of operations that overlap with Examples 1 to 3 will be omitted. The target image extraction unit 2 outputs extraction information along with the target image when extracting a target image of a small region containing the target object from the captured image. The extraction information includes information on whether the target image was extracted and, if extracted, the coordinate information of the extracted object in the captured image. The installation status determination unit 8 has a database containing information on the installation status of the target object, specifically information on the installation position and height of the target object in the real world. The installation status determination unit 8 can also refer to data indicating the dimensions of the mobile body and the mounting position of the camera 1. Based on the position information output by the position information acquisition unit 4, the installation status determination unit 8 refers to the database and calculates the appearance area of ​​the target object in the captured image. When the mobile body is approaching the position where the target object should appear, if the target object is extracted from within the appearance area of ​​the captured image, the installation status determination result is considered normal. Conversely, if the moving object is approaching the position where the target object should appear, but the target image cannot be extracted from the captured image, or if the extracted coordinate information is far from the appearance area, the installation status determination result will be output as abnormal.

[0045] When comparing the extracted coordinate information with the occurrence area, one could, for example, determine the center coordinates of each and use the Euclidean distance between them. Alternatively, the degree of overlap between the two could be evaluated using Intersection over Union (IoU).

[0046] Examples of abnormal installation conditions include being hidden by vegetation, significant deformation, tilting, or being lost. Figure 12A is an example of a photograph taken from the front of a railway vehicle. In the photograph F201 acquired by camera 1, the tree F1201 near the sign F202 is overgrown and obscures the sign F202, making it impossible to detect the object within the object appearance area F1202. In this case, the installation status determination unit 8 outputs the object installation status determination result as abnormal.

[0047] Figure 12B is an example of an image taken from the front of a railway vehicle. In the image F201 acquired by camera 1, the sign F202 is tilted, and the object cannot be detected within the object appearance area F1202. In this case as well, the installation status determination unit 8 outputs the object installation status determination result as abnormal.

[0048] As described above, the present invention makes it possible to realize a railway line equipment inspection system capable of determining the installation status of an object. Specifically, the disclosed state determination device comprises a camera 1 mounted on a moving body that acquires images of the travel space, a target image extraction unit 2 that extracts a target image of a small region containing the object to be determined from the acquired image, and a determination unit 3 that performs image processing on the target image and outputs a state determination result indicating whether or not there is an abnormality in the object and a false detection factor determination result indicating whether or not there is a false detection factor. With this configuration, the condition determination device can use captured images of the object to detect abnormalities occurring in the object, and also distinguish between abnormalities and factors that cause false detections, such as shadows.

[0049] Furthermore, the determination unit 3 makes a determination regarding one or more false detection factors, and if it obtains a false detection factor determination result indicating the presence of any of the false detection factors, it suppresses the output of a status determination result indicating that there is an abnormality in the object. This operation prevents the system from outputting an abnormality result based on false positive factors, thereby suppressing unnecessary maintenance work and avoiding a decrease in confidence in the alerts.

[0050] Furthermore, if the false detection factor is a shadow, the determination unit 3 may be configured to include: an abnormal score calculation unit 301 that receives the target image and calculates an abnormal score indicating the degree of deviation from the normal state of the object through image processing; a shadow score calculation unit 302 that receives the target image and calculates a shadow score indicating the degree of shadow cast on the object through image processing; and an integrated determination unit 303 that outputs a shadow determination result indicating a shadow when the shadow score is greater than a predetermined shadow score threshold, and outputs a state determination result indicating an abnormality when the abnormal score is greater than a predetermined abnormal score threshold and the shadow determination result indicates no shadow. This configuration allows for real-time, high-precision status determination of the mobile unit alone.

[0051] Furthermore, the system may also be configured to further include a location information acquisition unit 4 that acquires the current location information of the moving object when the false detection factor is a shadow, the determination unit 3 associates the location information with the abnormal score output by the abnormal score calculation unit and records it in the abnormal score recording unit 5, and acquires the abnormal score history for a predetermined first period in the past relating to the object from the abnormal score recording unit 5 based on the location information, the determination unit 3 outputs a state determination result that indicates an abnormality if the abnormal score history has an offset component over a predetermined second period, and an abnormal score history analysis unit 304 that outputs a shadow determination result that indicates a shadow exists if the difference obtained by subtracting the offset component from the current abnormal score exceeds a predetermined threshold. In this configuration, the abnormality scores from numerous moving objects can be statistically utilized to easily determine the state of the system.

[0052] Furthermore, for the purpose of learning image processing to calculate shadow scores, the system may also be configured to include a learning data generation unit 6 that uses the shadow determination results to assign training information to the target image, and a learning data recording unit 7 that records the learning data. In this configuration, the system can learn from the judgment results and improve the accuracy of state determination.

[0053] The system further comprises a location information acquisition unit 4 for acquiring the current location information of the moving object, and an installation status determination unit 8 for determining the installation status of the object. The target image extraction unit 2, when extracting a target image of a small region including the object from a captured image, outputs extraction information including success or failure information for extracting the target image and, if extracted, the extracted coordinate information in the captured image, along with the target image. The installation status determination unit 8 has a database containing information regarding the installation status, including the installation position and / or height of the object. Based on the location information output by the location information acquisition unit 4 and the database, it calculates the appearance area of ​​the object in the captured image. The system then compares the appearance area with the extraction information and outputs an installation status determination result indicating that the object is normal if it is extracted from within the appearance area of ​​the captured image, and abnormal if the object should appear but the target image cannot be extracted from the captured image or the extracted coordinate information is far from the appearance area. This configuration also allows for the detection of abnormalities in the object's position. Furthermore, the detection of positional abnormalities can be performed independently of the detection of external appearances of the object, such as dirt or shadows.

[0054] A moving object is, for example, a railway vehicle. Similarly, an object subject to state determination is, for example, a sign or a signal light. If camera 1 is installed on a railway vehicle, its movement along the rails helps in determining the position of camera 1. Therefore, in Examples 2 to 4, images from the same viewpoint can be easily collected. In addition, the "position where the object should appear" in Example 4 can be determined simply and with high accuracy. For objects such as signs and traffic lights, where an abnormal appearance directly impairs their function, determining their condition is particularly important.

[0055] In the embodiments described above, the present invention was mainly applied to a railway line equipment inspection system for automatically determining the condition (normal / abnormal) of railway line equipment, with the aim of assisting railway operation or realizing unmanned operation. However, it may also be applied to other uses. For example, it may be applied to driving assistance systems or maintenance assistance systems for automobiles traveling on highways, aircraft traveling on runways, drones flying along power lines, etc.

[0056] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0057] Furthermore, each of the above configurations may be implemented either partially or entirely in hardware, or through program execution on a processor. Also, the control lines and information lines shown are those deemed necessary for illustrative purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, almost all configurations can be considered interconnected. [Explanation of Symbols]

[0058] 1········Camera 2········Target image extraction unit 3...Judgment section 4... Location information acquisition section 5. Abnormal Score Recording Section 6·········Training Data Generation Unit 7·········Learning Data Recording Section 8. Installation status determination unit 301..... Abnormal Score Calculation Unit 302....Shadow score calculation unit 303...Integrated Judgment Unit 304..... Abnormal Score History Analysis Department F201.....Photographed image F202... Sign F203.....dirt F204·····Target image F205... Shadow F901...Periodic fluctuations F902...Offset component F1201....wood F1202···Appearance area of ​​the object S1·······System operation continuation confirmation step S2·······Steps to acquire video frame S3·······Target image extraction step S4·······Judgment Step S401... Abnormal Score Calculation Step S402... Shadow score calculation step S403... Shadow Score Comparison Step S404... Shadow detection result (shadow present) setting step S405... Shadow detection result (no shadow) setting step S406... Abnormal Score Comparison Step S407... Shadow detection result confirmation step S408...Status determination result (abnormal) setting step S409·····Status determination result (normal) setting step

Claims

1. A camera mounted on a mobile vehicle that captures images of the space in which it is traveling, A target image extraction unit extracts a target image of a small region containing the object to be determined in the captured image, A determination unit that performs image processing on the target image and outputs a state determination result indicating whether or not there is an abnormality in the target object and a false detection factor determination result indicating whether or not there is a false detection factor. A location information acquisition unit that acquires the current location information of the moving object, It is equipped with, The aforementioned false detection factor is a shadow, The determination unit, The system includes an abnormality score calculation unit that receives the aforementioned target image as input and calculates an abnormality score indicating the degree of deviation from the normal state of the object through image processing. The determination unit, The location information and the abnormal score output by the abnormal score calculation unit are associated and recorded in the abnormal score recording unit, and the abnormal score history for a predetermined first period in the past relating to the object is obtained from the abnormal score recording unit based on the location information. The determination unit, The system further comprises an abnormal score history analysis unit that outputs a state determination result indicating an abnormality if the abnormal score history has an offset component over a predetermined second period, and outputs a shadow determination result indicating a shadow if the difference obtained by subtracting the offset component from the current abnormal score exceeds a predetermined threshold. A state determination device characterized by the following.

2. A state determination device according to claim 1, The determination unit makes a determination regarding one or more false detection factors, and if it obtains a false detection factor determination result indicating the presence of any of the false detection factors, it suppresses the output of a status determination result indicating that there is an abnormality in the object. A state determination device characterized by the following.

3. A state determination device according to claim 1, The determination unit, A shadow score calculation unit receives the aforementioned target image as input and calculates a shadow score indicating the degree of shadow cast on the target object through image processing. The system comprises an integrated determination unit that outputs a shadow determination result indicating the presence of a shadow when the shadow score is greater than a predetermined shadow score threshold, and an integrated determination unit that outputs a status determination result indicating an abnormality when the abnormality score is greater than a predetermined abnormality score threshold and the shadow determination result indicates that there is no shadow. A state determination device characterized by the following.

4. A state determination device according to claim 3, A learning data generation unit generates learning data by using the shadow detection results and adding training information to the target image, for the purpose of learning the image processing that calculates the shadow score. The system comprises a learning data recording unit for recording the aforementioned learning data. A state determination device characterized by the following.

5. A state determination device according to claim 1, The system further comprises an installation status determination unit that determines the installation status of the object, When the target image extraction unit extracts a target image of a small region containing the object from the captured image, it outputs extraction information that includes, along with the target image, information on whether the target image was extracted and, if extracted, the coordinate information of the extracted object in the captured image. The installation status determination unit includes a database containing information about the installation status of the object, including its installation position and / or height. Based on the position information output by the position information acquisition unit and the database, it calculates the object's appearance area in the captured image. By comparing the appearance area with the extracted information, it outputs an installation status determination result indicating that the object is normal if it is extracted from within the appearance area of ​​the captured image, and abnormal if the object cannot be extracted from the captured image despite being expected to appear, or if the extracted coordinate information is far from the appearance area. A state determination device characterized by the following.

6. The state determination device according to claim 1, characterized in that the moving body is a railway vehicle.

7. The state determination device according to claim 1, characterized in that the object to state determination is a sign or a traffic light.

8. The state determination device, A video frame acquisition step to acquire video frames of the space in which a moving object is traveling, A target image extraction step of extracting a target image of a small region containing the object to be determined in the aforementioned video frame, A determination step which involves performing image processing on the target image and outputting a state determination result indicating whether or not there is an abnormality in the target object and a false detection factor determination result indicating whether or not there is a false detection factor, A location information acquisition step to acquire the current location information of the moving object, It is equipped with, The aforementioned false detection factor is a shadow, An abnormal score calculation step in which the aforementioned target image is input and an abnormal score indicating the degree of deviation from the normal state of the object is calculated by image processing, The steps include: associating the location information with the abnormal score output by the abnormal score calculation step and recording it in the abnormal score recording unit; and obtaining the abnormal score history for a predetermined first period in the past relating to the object from the abnormal score recording unit based on the location information; An abnormal score history analysis step that outputs a state determination result indicating abnormality if the abnormal score history has an offset component over a predetermined second period, and outputs a shadow determination result indicating shadow presence if the difference obtained by subtracting the offset component from the current abnormal score exceeds a predetermined threshold. To further possess A state determination method characterized by the following.

Images