Equipment malfunction diagnosis device, equipment malfunction diagnosis method, and overhead line equipment diagnosis system

Abstract

This invention provides an equipment malfunction diagnosis device capable of detecting a wider range of abnormalities. [Solution] The equipment abnormality diagnosis device 1 according to the present invention comprises an equipment detection unit 121 that detects equipment to be inspected from images captured by a camera 2, an abnormality candidate acquisition unit 122, an abnormality detection unit 123, and a storage device 13. The storage device 13 stores abnormality candidate data that indicates the type of abnormality mode, which is the type of abnormality that may occur in the equipment, for each piece of equipment. The abnormality candidate data is text-format data generated by a trained AI model capable of generating text in an abnormality candidate data generation device using the specifications of the equipment. The abnormality candidate acquisition unit 122 acquires abnormality candidate data for the equipment detected by the equipment detection unit 121 from the storage device 13. The abnormality detection unit 123 compares the image captured by the camera 2 with the abnormality candidate data acquired by the abnormality candidate acquisition unit 122 to obtain the abnormality mode of the equipment from the image and outputs the abnormality mode to the display device 3.

Description

[[Technical Field]]

[0001] The present invention relates to an apparatus and method for diagnosing equipment abnormality, and a system for diagnosing overhead contact line equipment. [[Background Art]]

[0002] A main application of the equipment abnormality diagnosis apparatus is, for example, an overhead contact line equipment diagnosis system intended for supporting railway maintenance work. The overhead contact line equipment diagnosis system is a system that inspects the installation state of overhead contact line equipment and the appearance of the overhead contact line equipment body, and notifies a user when an abnormality different from the original normal state is found. In the construction of the overhead contact line equipment diagnosis system, a method is useful in which a camera is installed on the roof of a railway vehicle, an analysis device is installed inside the railway vehicle or at a remote location, the analysis device is used to detect an abnormality of the overhead contact line equipment from an image captured by the camera, and a display device is used to notify the user of the presence or absence of the abnormality. By using the overhead contact line equipment diagnosis system, a user can perform repair or replacement of the equipment at an appropriate timing before a railway accident occurs.

[0003] An example of a conventional method for diagnosing equipment abnormality is described in Patent Document 1. The information processing method described in Patent Document 1 can perform determination corresponding to an abnormality occurring in infrastructure equipment, and includes: inputting an acquired image containing infrastructure equipment into a first learning model trained to output the types and regions of a plurality of types of infrastructure equipment included in the image, and outputting the types and regions of the plurality of types of infrastructure equipment; referring to a table in which a plurality of types of second learning models are registered for each type of infrastructure equipment, and specifying a corresponding single or plurality of second learning models for each of the plurality of types of infrastructure equipment included in the image, wherein each of the plurality of types of second learning models is trained to output information related to an abnormality of an infrastructure equipment when an image of the infrastructure equipment is input; and causing a computer to execute a process of inputting an image of the region output from the first learning model into each of the specified second learning models, and outputting information related to the abnormality of the infrastructure equipment included in the region. [[Prior Art Documents]] [[Patent Documents]]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-39226 [Overview of the project] [Problems that the invention aims to solve]

[0005] In image analysis for detecting anomalies, instead of simply outputting "anomaly" as the analysis result when an anomaly is found, clearly outputting the anomaly mode (type of anomaly), such as "corrosion" or "deformation," i.e., the basis for detecting the anomaly, can improve the reliability of the image analysis itself. Furthermore, by outputting the anomaly mode, users can take appropriate action to address the anomaly, such as formulating repair or replacement plans according to each anomaly mode.

[0006] Conventional technologies, such as the one described in Patent Document 1, detect equipment abnormalities based on abnormal modes (types of abnormalities) defined in advance by the user. Therefore, conventional technologies may not be able to detect all abnormalities, and there is room for improvement in detecting abnormal modes other than those defined in advance.

[0007] The object of the present invention is to provide an equipment malfunction diagnosis device and equipment malfunction diagnosis method capable of detecting a wider range of abnormalities, and an overhead line equipment diagnosis system equipped with this equipment malfunction diagnosis device. [Means for solving the problem]

[0008] The equipment abnormality diagnosis device according to the present invention comprises an equipment detection unit that detects equipment to be inspected from an image captured by a camera, an abnormality candidate acquisition unit, an abnormality detection unit, and a storage device. The storage device stores abnormality candidate data that indicates an abnormality mode, which is the type of abnormality that may occur in the equipment, for each piece of equipment. The abnormality candidate data is text-format data generated by a trained AI model capable of generating text in an abnormality candidate data generation device using the specifications of the equipment. The abnormality candidate acquisition unit acquires the abnormality candidate data for the equipment detected by the equipment detection unit from the storage device. The abnormality detection unit detects an abnormality in the equipment from the image by comparing the image with the abnormality candidate data acquired by the abnormality candidate acquisition unit, acquires the abnormality mode of the equipment, and outputs the acquired abnormality mode to a display device.

[0009] The equipment abnormality diagnosis method according to the present invention includes an abnormality candidate data in a storage device that indicates abnormality modes, which are types of abnormalities that may occur in the equipment to be inspected, for each piece of equipment, wherein the abnormality candidate data is text-format data generated by a trained AI model capable of generating text using the specifications of the equipment, and comprises an equipment detection step of detecting the equipment from an image captured by a camera, an abnormality candidate acquisition step of acquiring the abnormality candidate data for the equipment detected in the equipment detection step from the storage device, and an abnormality detection step of detecting an abnormality of the equipment from the image by comparing the image with the abnormality candidate data acquired in the abnormality candidate acquisition step, acquiring the abnormality mode of the equipment, and outputting the acquired abnormality mode to a display device.

[0010] The overhead line equipment diagnostic system according to the present invention comprises a camera installed on the roof of a railway vehicle, an equipment abnormality diagnostic device according to the present invention, a display device, and an abnormality candidate data generation device, wherein the equipment to be inspected is the overhead line equipment of a railway. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an equipment malfunction diagnosis device and equipment malfunction diagnosis method that can detect a greater number of abnormalities, and an overhead line equipment diagnosis system equipped with this equipment malfunction diagnosis device. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows the overall configuration of the overhead line equipment diagnostic system according to Embodiment 1 of the present invention. [Figure 2] This flowchart shows an example of the processing performed by the equipment malfunction diagnosis device according to Embodiment 1 of the present invention. [Figure 3] This diagram shows an example of the operation of the equipment detection unit. [Figure 4] This is an illustrative diagram showing an example of how the anomaly detection unit detects equipment anomalies and acquires the anomaly mode by comparing equipment images using LVLM with anomaly candidate data. [Figure 5] This figure shows an example flowchart of the process by which an abnormal candidate data generation device generates abnormal candidate data. [Figure 6] This figure shows an example of a GUI screen for software intended for building and correcting anomaly candidate databases. [Figure 7] This figure shows the overall configuration of the overhead line equipment diagnostic system according to Embodiment 2 of the present invention. [Figure 8] This figure shows the overall configuration of the overhead line equipment diagnostic system according to Embodiment 3 of the present invention. [Figure 9A] This figure shows an example of equipment images where the evaluation of shooting conditions is important, and it is an example of equipment images acquired under conditions without backlighting. [Figure 9B] This figure illustrates an example of equipment images where the evaluation of shooting conditions is important, specifically an example of equipment images acquired under backlighting conditions. [Figure 10] This figure shows the overall configuration of the overhead line equipment diagnostic system according to Embodiment 4 of the present invention. [Modes for carrying out the invention]

[0013] Hereinafter, an equipment abnormality diagnosis apparatus, an equipment abnormality diagnosis method, and an overhead contact line equipment diagnosis system according to embodiments of the present invention will be described with reference to the drawings. In the embodiment described below, as an example, an example where the equipment abnormality diagnosis apparatus diagnoses an abnormality in railway overhead contact line equipment will be described. The equipment abnormality diagnosis method is executed by the equipment abnormality diagnosis apparatus. The overhead contact line equipment diagnosis system is a main application of the equipment abnormality diagnosis apparatus, which is a system that inspects the installation state of railway overhead contact line equipment and the appearance of the overhead contact line equipment main body, and notifies a user of an abnormality when an abnormality different from the original normal state is found. The equipment diagnosed by the equipment abnormality diagnosis apparatus is not limited to railway overhead contact line equipment, and may be, for example, a rail, or any equipment other than railway equipment.

[0014] In the drawings referred to in this specification, the same or corresponding constituent elements are denoted by the same reference numerals, and repeated description of these constituent elements may be omitted.

Example

[0015] An equipment abnormality diagnosis apparatus, an equipment abnormality diagnosis method, and an overhead contact line equipment diagnosis system according to a first embodiment of the present invention will be described. The equipment abnormality diagnosis apparatus according to the present embodiment, by using data (abnormality candidate data) that indicates types of abnormalities that may occur in each piece of equipment, which is generated by a trained AI (Artificial Intelligence) model capable of generating text, matches an image containing the equipment with the abnormality candidate data, thereby enabling output of an abnormality mode matching the image containing the equipment, and can detect more abnormalities. In the present embodiment, the abnormality mode refers to a type of abnormality that may occur in equipment.

[0016] [System Configuration] The configuration and operation of the overhead contact line equipment diagnosis system according to the present embodiment will be described. In the overhead contact line equipment diagnosis system according to the present embodiment, the equipment to be inspected is railway overhead contact line equipment.

[0017] Figure 1 shows the overall configuration of the overhead line equipment diagnostic system according to this embodiment. The overhead line equipment diagnostic system according to this embodiment comprises a camera 2, an equipment abnormality diagnosis device 1, a display device 3, and an abnormality candidate data generation device 6.

[0018] Camera 2 is a device that captures images of equipment, and is installed, for example, on the roof of a railway vehicle to acquire images of the overhead lines while the railway vehicle is in motion. Camera 2 may or may not be directly connected to the equipment malfunction diagnosis device 1. If Camera 2 and the equipment malfunction diagnosis device 1 are not directly connected, the images captured by Camera 2 can be transferred to the equipment malfunction diagnosis device 1, for example, via external storage.

[0019] The equipment malfunction diagnosis device 1 detects equipment malfunctions by analyzing images captured by camera 2.

[0020] The display device 3 displays information about equipment abnormalities detected by the equipment abnormality diagnostic device 1. The display device 3 can output whether or not an abnormality exists and the abnormality mode, and notify the user of these.

[0021] The abnormality candidate data generation device 6 generates abnormality candidate data. This abnormality candidate data indicates the type of abnormality (abnormality mode) that may occur in the equipment, and is in text format, with the abnormality mode described in text. The abnormality candidate data generation device 6 may also include an equipment database 61, as will be described later. The abnormality candidate data generation device 6 will be described in detail later.

[0022] The abnormality candidate data generation device 6 may or may not be included in the equipment abnormality diagnosis device 1. Furthermore, the abnormality candidate data generation device 6 may be installed inside or outside the equipment abnormality diagnosis device 1. If the abnormality candidate data generation device 6 is installed outside the equipment abnormality diagnosis device 1, the equipment abnormality diagnosis device 1 can receive the abnormality candidate data generated by the abnormality candidate data generation device 6 via a network or external storage.

[0023] The equipment malfunction diagnosis device 1 may be installed inside a railway vehicle or at a remote location away from the railway vehicle. When installed inside a railway vehicle, the equipment malfunction diagnosis device 1 can acquire images from camera 2 via a wired network and perform image analysis in real time. When the equipment malfunction diagnosis device 1 is installed at a remote location, images from camera 2 may be transmitted to the equipment malfunction diagnosis device 1 via a wireless network with high transmission capacity, or images taken over a predetermined period may be stored in external storage, and the external storage may be transferred to the remote location for the equipment malfunction diagnosis device 1 to perform image analysis.

[0024] The display device 3, like the equipment malfunction diagnosis device 1, may be installed inside the railway vehicle or at a remote location away from the railway vehicle. If the display device 3 is installed inside the railway vehicle, when the equipment malfunction diagnosis device 1 detects an abnormality in the equipment, the user can immediately go to the site to check the equipment. If the display device 3 is installed at a remote location, the user can perform maintenance work remotely.

[0025] For example, if the equipment malfunction diagnosis device 1 is installed inside a railway vehicle and the display device 3 is installed in a remote location, the equipment malfunction diagnosis device 1 can perform real-time image analysis on the images captured by the camera 2 and transmit the detection results to the remote display device 3 via a wireless network only if an abnormality is detected. In this way, even if the transmission capacity of the wireless network is small, the user can perform maintenance work remotely.

[0026] The configuration and operation of the equipment malfunction diagnosis device 1 will now be described. As shown in Figure 1, the equipment malfunction diagnosis device 1 comprises an input interface 11, a processor 12, a storage device 13, and an output interface 14. The processor 12 executes a program that realizes the functions of the equipment detection unit 121, the malfunction candidate acquisition unit 122, and the malfunction detection unit 123, thereby enabling the equipment malfunction diagnosis device 1 to have these processing units. The storage device 13 contains a malfunction candidate database 131. The malfunction candidate database 131 stores multiple malfunction candidate data for each of multiple pieces of equipment.

[0027] Figure 2 is a flowchart showing an example of the processing performed by the equipment malfunction diagnosis device 1 according to this embodiment.

[0028] In step S101, the processor 12 receives the image (image captured by camera 2) acquired by camera 2 when it photographs the equipment to be inspected via the input interface 11.

[0029] In step S102, the equipment detection unit 121 detects the equipment to be inspected from the image captured by camera 2, extracts an image of a small region containing the equipment to be inspected from this image, and determines the name of the detected equipment (equipment name). The image of the small region containing the equipment extracted from the image captured by camera 2 is called the equipment image. The equipment name is associated with the characteristics of the equipment, such as its shape, size, and location. Therefore, the equipment detection unit 121 can determine the equipment name when it detects equipment by utilizing the characteristics of the equipment.

[0030] In step S103, the equipment detection unit 121 determines whether or not equipment is present in the captured image. If an image of equipment is extracted from the image captured by camera 2 in step S102, it means that equipment is present in the captured image; if no image of equipment is extracted, it means that equipment is not present in the captured image.

[0031] Step S104 is the process to be performed when equipment is present in the captured image. In step S104, the abnormality candidate acquisition unit 122 acquires one or more abnormality candidate data from the abnormality candidate database 131 for equipment detected by the equipment detection unit 121, that is, equipment with the equipment name determined by the equipment detection unit 121. As described above, the abnormality candidate data is text-format data in which the abnormality mode for each piece of equipment is described in text.

[0032] In step S105, the anomaly detection unit 123 compares the equipment image extracted in step S102 with the anomaly candidate data acquired in step S104 to detect an anomaly in the equipment and acquire the anomaly mode of the equipment.

[0033] In step S106, the abnormality detection unit 123 determines whether or not it has detected an abnormality in the equipment from the equipment image.

[0034] Step S107 is the process to be performed when the anomaly detection unit 123 detects an anomaly in the equipment. In step S107, the anomaly detection unit 123 outputs the anomaly mode obtained by comparing the equipment image with the anomaly candidate data to the display device 3 in text format via the output interface 14.

[0035] The input interface 11 may use wired LAN, wireless LAN, USB, CameraLink, etc., depending on the connection method with camera 2. If camera 2 and equipment malfunction diagnosis device 1 are not directly connected, and images captured by camera 2 are transferred to the equipment malfunction diagnosis device 1 via external storage, wired LAN, USB, etc. may be used depending on the connection method with the external storage.

[0036] The processor 12 may use one or more CPUs, one or more GPUs, and one or more accelerators, etc., in an appropriate combination.

[0037] The storage device 13 may use internal storage such as an HDD or SSD, external storage connected via wired LAN or USB, or cloud storage.

[0038] The output interface 14 may use wired LAN, wireless LAN, USB, VGA, HDMI (registered trademark), DisplayPort, etc., depending on the connection method with the display device 3.

[0039] The equipment detection unit 121 detects equipment and determines its name, for example, using a Deep Neural Network (DNN). For example, the equipment detection unit 121 can use a DNN that has been trained using training data related to equipment for which abnormalities are to be diagnosed. The task to be solved in detecting equipment may be, for example, a Semantic Segmentation task that labels images at the pixel level. Alternatively, it may be an Object Detection task that outputs information on the location and size of the equipment. Furthermore, the equipment detection unit 121 may use an Open-Vocabulary-based Semantic Segmentation or Object Detection method that does not use training data related to equipment for which abnormalities are to be diagnosed, allowing the detection target to be specified with arbitrary text. Alternatively, the equipment detection unit 121 may use image processing such as edge detection instead of a DNN.

[0040] Figure 3 shows an example of the operation of the equipment detection unit 121. Figure 3 shows image F101 taken by camera 2 and equipment image F103 extracted from image F101 by the equipment detection unit 121. Both image F101 taken by camera 2 and equipment image F103 show equipment F102, which is used to diagnose abnormalities.

[0041] Various backgrounds captured in the image F101 taken by camera 2 can interfere with the detection of abnormalities in equipment F102. Therefore, the equipment detection unit 121 extracts a small area of ​​equipment image F103, including the equipment F102 to be diagnosed as abnormal, from image F101, so that the abnormality detection unit 123 can perform abnormality detection effectively.

[0042] The anomaly detection unit 123 detects anomalies in the equipment from the equipment image and obtains the anomaly mode of the equipment by comparing the equipment image with anomaly candidate data. For example, the anomaly detection unit 123 encodes the equipment image and anomaly candidate data (data in text format) into a common vector space, calculates the similarity between the vector encoded from the equipment image and the vector encoded from the anomaly candidate data, and uses this similarity to detect anomalies and obtain the anomaly mode. For example, the anomaly detection unit 123 may output anomaly candidate data whose similarity exceeds a predetermined threshold as an anomaly mode that matches the equipment image.

[0043] For image and text encoding, a Large Vision Language Model (LVLM), such as the Contrastive Language-Image Pre-training (CLIP) model, may be used. Alternatively, an Open-Vocabulary-based Semantic Segmentation or Object Detection method, which allows detection targets to be indicated with arbitrary text, may be used to directly detect areas matching candidate anomaly data from equipment images using text indications.

[0044] Figure 4 is an illustrative diagram showing an example of how the anomaly detection unit 123 detects equipment anomalies and acquires the anomaly mode by comparing equipment images using LVLM with anomaly candidate data (text). The numerical values ​​shown in Figure 4 are examples of the similarity between encoded vectors. The texts used for comparison are anomaly candidate data "No anomaly" F201A, anomaly candidate data "Corrosion present" F201B, and anomaly candidate data "Deformation present" F201C.

[0045] Suppose the anomaly detection unit 123 receives an image of a normal piece of equipment, F103A. In this case, the vector encoded from the equipment image F103A has a similarity of 0.7 to the vector encoded from the anomaly candidate data "no anomaly" F201A. That is, the equipment image F103A has the highest similarity to the anomaly candidate data "no anomaly" F201A among the three anomaly candidate data. Therefore, the anomaly detection unit 123 acquires "no anomaly" as the anomaly mode for the equipment and does not detect an anomaly in the equipment.

[0046] Suppose the anomaly detection unit 123 receives an image of the deformed equipment, F103B. In this case, the vector encoded from the equipment image F103B has a similarity of 0.6 to the vector encoded from the anomaly candidate data "deformed" F201C. That is, the equipment image F103B has the highest similarity to the anomaly candidate data "deformed" F201C among the three anomaly candidate data. Therefore, the anomaly detection unit 123 acquires "deformed" as the anomaly mode for the equipment and detects an anomaly in the equipment.

[0047] The anomaly detection unit 123 may output the anomaly mode obtained by comparing the equipment image with the anomaly candidate data as text only, or it may output the text representing the anomaly mode together with the equipment image.

[0048] Furthermore, the anomaly detection unit 123 may add markers to the equipment image indicating the location of the anomaly identified in the anomaly mode, and output the equipment image with the markers along with text representing the anomaly mode. The markers added to the equipment image can indicate the location of the anomaly in any format, such as lines or colors applied to the location of the anomaly, and indicate areas within the equipment image that have a high correlation with the anomaly candidate data. The location of the anomaly can be drawn, for example, in LVLM by identifying image areas with high co-attention, i.e., high correlation, with the anomaly candidate data.

[0049] [Generating abnormal candidate data] The anomaly candidate data generation device 6 pre-generates multiple anomaly candidate data using a trained AI model capable of generating text, and stores them in the anomaly candidate database 131.

[0050] The pre-trained AI model used is one that has been trained on a vast amount of data from the internet, such as ChatGPT. Such a model is expected to have broad knowledge of various abnormality modes. However, if the equipment being diagnosed has an unusual name, it cannot be expected that the pre-trained AI model will have sufficient knowledge of such equipment, and there is room for improvement in appropriately generating abnormality candidate data for such equipment.

[0051] For example, if the equipment shown in Figure 3 (the overhead line equipment included in equipment image F103) is uncommon, the trained AI model described above will not be able to appropriately generate candidate anomaly data for this equipment. However, by providing the AI ​​model with specialized information about the equipment, such as that it is made of metal, the AI ​​model can infer anomaly modes such as corrosion, deformation, and wear, and appropriately generate candidate anomaly data for this equipment.

[0052] To achieve this, it is preferable for the abnormality candidate data generation device 6 to create an equipment database 61, which is a database for relating specialized knowledge (specific domain knowledge) about equipment used to diagnose abnormalities such as equipment names with general knowledge. General knowledge is, for example, knowledge that a trained AI model learns from data on the internet. When the abnormality candidate data generation device 6 questions the trained AI model about abnormal modes that may occur in the equipment, it can appropriately generate abnormality candidate data about the equipment by having the AI ​​model refer to the equipment database 61, for example, using Retrieval-Augmented Generation (RAG) technology. The abnormality candidate data generation device 6 may create and maintain multiple types of equipment databases 61.

[0053] The equipment database 61 is provided in the abnormality candidate data generation device 6 and stores equipment data for the equipment. The equipment data is text data that associates the equipment name with the equipment specifications. The equipment specifications include the equipment's materials, shape, and application, and can be described based on general knowledge. If a catalog showing the details of the equipment exists, the abnormality candidate data generation device 6 may read this catalog and create the equipment database 61.

[0054] Furthermore, the anomaly candidate data generation device 6 may generate anomaly candidate data by first providing a trained AI model with specific domain knowledge about the equipment using a knowledge graph that describes the relationships between components constituting the equipment in a graph structure, and then questioning the AI ​​model about possible anomaly modes that may occur in the equipment. Specific domain knowledge about the equipment includes, for example, equipment specifications such as the equipment's materials, shape, and intended use. Using such a knowledge graph makes it possible to improve the accuracy of anomaly candidate data generation by the trained AI model.

[0055] Figure 5 shows an example flowchart of the process by which the abnormality candidate data generation device 6 generates abnormality candidate data. The abnormality candidate data generation device 6 generates abnormality candidate data for each piece of equipment. It is assumed that multiple pieces of equipment to be inspected and diagnosed for abnormalities are predetermined.

[0056] In step S201, the abnormality candidate data generation device 6 selects one piece of equipment from the equipment to be inspected and obtains the equipment data for the selected piece of equipment from the equipment database 61.

[0057] In step S202, the abnormality candidate data generation device 6 generates abnormality candidate data for the selected equipment using a trained AI model. The trained AI model can appropriately generate abnormality candidate data for equipment by referring to equipment data (equipment name and equipment specifications) obtained from the equipment database 61.

[0058] In step S203, the abnormal candidate data generation device 6 stores the generated abnormal candidate data in the abnormal candidate database 131.

[0059] In step S204, the abnormality candidate data generation device 6 determines whether or not it has generated abnormality candidate data for all the equipment to be inspected. The abnormality candidate data generation device 6 executes the processes from step S201 to step S203 until it has generated abnormality candidate data for all the equipment to be inspected.

[0060] The abnormal candidate data stored in the abnormal candidate database 131 may be made available for users to manually modify.

[0061] Figure 6 shows an example of a GUI (Graphical User Interface) screen for software intended to build and modify the anomaly candidate database 131. This GUI screen can be displayed by the anomaly candidate data generation device 6.

[0062] The anomaly candidate data generation device 6 displays a GUI screen that allows the user to perform at least one of the following actions: deleting anomaly modes from anomaly candidate data generated by the trained AI model, or adding anomaly modes to anomaly candidate data generated by the trained AI model. Using this GUI screen, the user can review the anomaly candidate data generated by the trained AI model and perform at least one of the following actions: deleting anomaly modes that are clearly unlikely to occur in reality, or adding anomaly modes that have actually occurred in the past.

[0063] The equipment selection dropdown F301 is used when the user selects the equipment that generates abnormal candidate data.

[0064] The equipment database selection dropdown F302 is used when the user selects the type of equipment database 61 to use.

[0065] The AI ​​model selection dropdown F303 is used when the user selects a pre-trained AI model to use for generating anomalous candidate data.

[0066] The generate button F304 is used when the user generates the list of potential anomaly data F306. After the user has finished setting the equipment for generating the potential anomaly data, the type of equipment database 61 to be used, and the trained AI model to be used for generating the potential anomaly data, pressing the generate button F304 will generate the list of potential anomaly data F306.

[0067] The user can review the generated list of potential abnormal data (F306). If an abnormal mode has been generated that is clearly impossible to occur in reality, for example, the user can select this abnormal mode and then press the delete button (F307) to remove it from the list of potential abnormal data.

[0068] Furthermore, if a user wishes to edit the generated abnormality candidate data, they can select the abnormality candidate data they want to edit and then press the edit button F308 to perform the edit. Since the abnormality candidate data is data in which the abnormality mode is described in text, the user can, for example, edit a part of the text indicating the abnormality mode.

[0069] Furthermore, if, for example, an abnormal mode that has actually occurred in the past is not included in the abnormality candidate data list F306, the user presses the add button F309. After this, the user can add the abnormal mode that has actually occurred in the past to the abnormality candidate data by performing predetermined operations such as keyboard input or selecting a file that describes the abnormal modes that have actually occurred in the past.

[0070] For example, the user can build and modify the abnormal candidate database 131 using the GUI screen shown in Figure 6, as described above. Once the user has finished building and modifying the abnormal candidate database 131, they press the save button F305 to save the changes and finish their work.

[0071] The abnormal candidate data stored in the abnormal candidate database 131 and acquired by the abnormal candidate acquisition unit 122 is data described in text indicating the abnormal mode, but this text may also be associated with an image indicating the abnormal mode. For example, when the abnormal detection unit 123 detects an abnormality using the LVLM method described above, the abnormal candidate data that it matches with the equipment image of the equipment to be diagnosed may be an image indicating the abnormal mode. This image indicating the abnormal mode is an image acquired when an abnormality occurred in the equipment to be diagnosed in the past, and the equipment name and the equipment's abnormal mode are identified.

[0072] In abnormal candidate data, the text may be associated with one or more images.

[0073] For example, instead of encoding text indicating an abnormal mode, the anomaly detection unit 123 can encode an image indicating an abnormal mode, thereby calculating the similarity between vectors in the same way as described above. By matching images from the same modal in this way, it is possible to detect equipment images whose features are more similar to images acquired when an abnormal mode occurred in the past, compared to matching images from different modals with text.

[0074] This image-to-image matching may be selectively used in conjunction with image-to-text matching. That is, the anomaly detection unit 123 can perform image-to-text matching, or a combination of image-to-text matching and image-to-image matching.

[0075] In the abnormality candidate data stored in the abnormality candidate database 131, associating images with text may be done manually by the user or automatically by the equipment abnormality diagnosis device 1. If the user does it manually, for example, an item for adding an image may be added to the editing operation performed by the user by pressing the edit button F308 on the GUI screen shown in Figure 6. Alternatively, if the equipment abnormality diagnosis device 1 does it automatically, for example, when the abnormality detection unit 123 detects an abnormality in the equipment, it may automatically add an image of the equipment in which the abnormality was detected to the abnormality candidate data that indicates the abnormality mode of the detected abnormality among the abnormality candidate data stored in the abnormality candidate database 131.

[0076] As explained above, in this embodiment, a trained AI model capable of generating text is used to generate candidate anomaly data for potential anomalies in the equipment. By comparing the equipment image with the candidate anomaly data, it is possible to output an anomaly mode that matches the equipment image, thereby enabling the detection of a wider range of anomalies. [Examples]

[0077] This section describes an overhead line equipment diagnostic system according to Embodiment 2 of the present invention. The overhead line equipment diagnostic system according to this embodiment can classify whether an abnormality detected in the equipment is a specific abnormality or not, and output the classification result. A specific abnormality is a predefined important abnormality. The following description will mainly explain the differences between the overhead line equipment diagnostic system according to this embodiment and the overhead line equipment diagnostic system according to Embodiment 1. [System Configuration] The configuration and operation of the overhead line equipment diagnostic system according to this embodiment will be described.

[0078] Figure 7 shows the overall configuration of the overhead line equipment diagnostic system according to Embodiment 2 of the present invention. The overhead line equipment diagnostic system according to this embodiment further includes a specific abnormality database 132 and a classification unit 124 in addition to the overhead line equipment diagnostic system according to Embodiment 1 (Figure 1).

[0079] The specific anomaly database 132 is located in the storage device 13 and stores one or more specific anomaly data. The specific anomaly data is data that indicates the type of specific anomaly among the candidate anomaly data, and indicates a predetermined important type of anomaly (i.e., an important anomaly mode). The specific anomaly data is data in which the type of specific anomaly is described in text, but like the candidate anomaly data, it may be data in which an image is associated with the text.

[0080] The classification unit 124 is provided in the equipment abnormality diagnosis device 1 and compares the abnormality mode acquired by the abnormality detection unit 123 with the specific abnormality data stored in the specific abnormality database 132 to determine whether or not there is specific abnormality data that matches the abnormality mode. The classification unit 124 outputs this determination result as the classification result of the abnormality mode. For example, if the classification unit 124 finds specific abnormality data that matches the abnormality mode acquired by the abnormality detection unit 123, it classifies this abnormality mode as a specific abnormality and outputs this specific abnormality as the classification result.

[0081] For example, the classification unit 124 compares the text indicating the abnormal mode acquired by the abnormality detection unit 123 with the text of the specific abnormal data to determine whether or not specific abnormal data exists. An example of a text comparison method is the method used by the abnormality detection unit 123 to detect equipment abnormalities, as described in Example 1. In this method, the classification unit 124 encodes the two texts to be compared into a common vector space, calculates the similarity between the vectors of these encoded texts, and determines that an abnormal mode whose similarity exceeds a predetermined threshold is an abnormal mode that matches the specific abnormal data.

[0082] Alternatively, if the specific anomaly data is data in which an image is associated with text, the classification unit 124 may encode the equipment image in which the anomaly detection unit 123 detected an anomaly and the image of the specific anomaly data into a common vector space, calculate the similarity between the vectors of these encoded images, and determine that any anomaly mode in which this similarity exceeds a predetermined threshold is an anomaly mode that matches the specific anomaly data.

[0083] The classification unit 124 classifies the abnormal mode as not being a specific abnormality if there is no specific abnormality data that matches the abnormal mode. In this case, the classification unit 124 may output the classification result as, for example, "Other abnormality".

[0084] This embodiment is useful, for example, when introducing an overhead line equipment diagnostic system to support diagnostic tasks that were previously performed manually by users. Specifically, by registering the abnormalities detected by the inspection items for each piece of equipment that were previously referenced by the user as specific abnormalities in the specific abnormality database, it is possible to introduce an overhead line equipment diagnostic system without changing the inspection items from before. Furthermore, if the classification unit 124 outputs "Other abnormalities" as the classification result when there is no specific abnormality data that matches the abnormality mode, it may be possible to detect abnormalities that could not be detected with conventional inspection items (inspection items for detecting specific abnormalities).

[0085] As explained above, in this embodiment, when an abnormality is detected in the equipment, it is possible to classify whether the detected abnormality is a specific abnormality or not and output the classification result. [Examples]

[0086] This section describes an overhead line equipment diagnostic system according to Embodiment 3 of the present invention. This embodiment utilizes a pre-trained AI model capable of generating text during inspections, enabling real-time evaluation of the shooting conditions during inspections. The following description will primarily focus on the differences between this embodiment and the overhead line equipment diagnostic system according to Embodiment 1. [System Configuration] The configuration and operation of the overhead line equipment diagnostic system according to this embodiment will be described.

[0087] Figure 8 shows the overall configuration of the overhead line equipment diagnostic system according to Embodiment 3 of the present invention. The overhead line equipment diagnostic system according to this embodiment further comprises a communication device 15, a shooting condition evaluation unit 125, and a trained AI model 4 in addition to the overhead line equipment diagnostic system according to Embodiment 1 (Figure 1).

[0088] The communication device 15 is provided in the equipment malfunction diagnosis device 1 and enables communication between the equipment malfunction diagnosis device 1 and the trained AI model 4.

[0089] The shooting condition evaluation unit 125 is provided in the equipment malfunction diagnosis device 1 and can access the trained AI model 4 via the communication device 15. Details of the shooting condition evaluation unit 125 will be described later.

[0090] The trained AI model 4 is a model capable of receiving questions using images and generating text, and is a model trained on a vast amount of data from the internet. The trained AI model 4 may be the same model as the trained AI model used to generate the anomaly candidate data, or it may be a different model. The trained AI model 4 is installed outside or inside the equipment anomaly diagnosis device 1.

[0091] For example, the trained AI model 4 may be installed outside the equipment malfunction diagnosis device 1 and run on a remote cloud server. In this case, by using, for example, a wireless LAN as the communication device 15, the shooting condition evaluation unit 125 can access the trained AI model 4 via the communication device 15.

[0092] If the computational specifications of the equipment malfunction diagnosis device 1 are sufficiently large, the trained AI model 4 may be installed inside the equipment malfunction diagnosis device 1. In this case, the shooting condition evaluation unit 125 may be configured to access the trained AI model 4 without using the communication device 15. With this configuration, the equipment malfunction diagnosis device 1 can stably access the trained AI model 4 regardless of radio wave conditions, and is particularly suitable when the equipment malfunction diagnosis device 1 is installed inside a railway vehicle.

[0093] The shooting condition evaluation unit 125 obtains information regarding the presence or absence of disturbances in the captured images acquired by the camera 2 by querying the trained AI model 4 and obtaining a response. Disturbances in captured images are external factors that can hinder stable anomaly detection in captured images, and specific examples include sunlight conditions and weather. Sunlight conditions and weather are general knowledge that the trained AI model 4 can learn from data on the internet. Therefore, the shooting condition evaluation unit 125 can obtain a reasonable response from the trained AI model 4 regarding the presence or absence of disturbances in the captured images.

[0094] For example, the shooting condition evaluation unit 125 uses the captured image as a question for the trained AI model 4 (by inputting it into the trained AI model 4) to obtain information from the trained AI model 4 regarding the presence or absence of disturbances in the captured image, such as the presence or absence of backlighting, which is a sunlight condition.

[0095] The shooting condition evaluation unit 125 acquires information regarding the presence or absence of disturbances in the captured image, then adds this acquired information as secondary information to the abnormality detection result of the abnormality detection unit 123, and outputs the abnormality detection result with the secondary information added to it to the display device 3 via the output interface 14.

[0096] The captured images used by the shooting condition evaluation unit 125 to ask questions to the trained AI model 4 may be images (equipment images) extracted by the equipment detection unit 121 from the images captured by the camera 2, or they may be captured images before the equipment detection unit 121 extracted the equipment images from the captured images.

[0097] In this embodiment, the shooting condition evaluation unit 125 obtains information about the presence or absence of disturbances (e.g., sunlight conditions and weather) in the captured image acquired by camera 2 from a trained AI model 4 as the shooting conditions during the inspection, when camera 2 acquires a captured image during the equipment inspection. Therefore, in this embodiment, the shooting conditions during the inspection can be evaluated in real time.

[0098] Figures 9A and 9B show examples of equipment images where the evaluation of shooting conditions is important. Figure 9A shows an example of equipment image F103C acquired under conditions without backlighting. Figure 9B shows an example of equipment image F103D acquired under conditions with backlighting.

[0099] Let's assume that "missing nuts" are one of the potential abnormal data points.

[0100] In the equipment image F103C shown in Figure 9A, the equipment was photographed under conditions without backlighting, so the nut F401 is clearly visible. Therefore, the anomaly detection unit 123 can use the equipment image F103C to determine that the nut F401 is correctly installed.

[0101] On the other hand, in the equipment image F103D shown in Figure 9B, the equipment was photographed under backlighting conditions, and therefore the nut F401 is not clearly visible due to the backlighting. Consequently, the anomaly detection unit 123 may incorrectly determine that the nut is missing based on the equipment image F103D.

[0102] At this time, if the shooting condition evaluation unit 125 has obtained from the trained AI model 4 that there is backlighting as a shooting condition during inspection (presence or absence of disturbances in the captured image), the user can determine that this misjudgment is due to poor sunlight conditions.

[0103] The shooting condition evaluation unit 125 can treat the acquired information regarding the presence or absence of disturbances in the captured image as secondary information to the abnormality detection result of the abnormality detection unit 123, as described above. Furthermore, if disturbances are found in the captured image and poor shooting conditions are detected, the shooting condition evaluation unit 125 may not output the abnormality detection result, but instead output only a message to the display device 3 indicating that the diagnostic stability of the overhead line equipment diagnostic system has decreased due to poor shooting conditions.

[0104] As explained above, in this embodiment, a pre-trained AI model 4 capable of generating text can be used during inspection to evaluate the shooting conditions during inspection in real time. [Examples]

[0105] This section describes an overhead line equipment diagnostic system according to Embodiment 4 of the present invention. This embodiment of the overhead line equipment diagnostic system can detect abnormalities in the equipment's position by comparing the equipment's position at the time of image capture with its original installation position. The following description will primarily focus on the differences between this embodiment of the overhead line equipment diagnostic system and the overhead line equipment diagnostic system according to Embodiment 1. [System Configuration] The configuration and operation of the overhead line equipment diagnostic system according to this embodiment will be described.

[0106] Figure 10 shows the overall configuration of the overhead line equipment diagnostic system according to Embodiment 4 of the present invention. The overhead line equipment diagnostic system according to this embodiment further comprises a communication device 15, a location information acquisition device 5, an equipment location database 133, and an equipment detection location determination unit 126 in addition to the overhead line equipment diagnostic system according to Embodiment 1 (Figure 1).

[0107] The communication device 15 is provided in the equipment malfunction diagnosis device 1 and enables communication between the equipment malfunction diagnosis device 1 and the location information acquisition device 5.

[0108] The location information acquisition device 5 acquires the position where the image taken by the camera 2 is captured. The location information acquisition device 5 may be composed of, for example, a GPS sensor, or a device that calculates the position based on the rotation speed of the wheels of a railway vehicle. The location information acquisition device 5 is installed, for example, on the camera 2 or on the railway vehicle.

[0109] The equipment location database 133 stores data related to the equipment's intended installation location. The intended installation location of the equipment is a predetermined location where the equipment should be. The equipment location database 133 may be located within the storage device 13 of the equipment malfunction diagnosis device 1, as shown in Figure 10, or it may be located in a remote cloud storage.

[0110] The equipment detection position determination unit 126 is provided in the equipment abnormality diagnosis device 1 and acquires the image capture position in real time from the position information acquisition device 5 via the communication device 15. For example, the equipment detection position determination unit 126 acquires the position of a moving railway vehicle from the position information acquisition device 5 as the image capture position.

[0111] Next, the equipment detection position determination unit 126 retrieves one or more pieces of equipment from the equipment position database 133 whose original installation location is within a predetermined range from the current shooting position of the image. Then, the equipment detection position determination unit 126 compares the equipment detected by the equipment detection unit 121 with the retrieved equipment. By comparing these pieces of equipment, the equipment detection position determination unit 126 can check whether the equipment detected by the equipment detection unit 121 is in its original installation location, detect an anomaly in the equipment's location, and determine whether there is an anomaly in the equipment's location.

[0112] The equipment detection position determination unit 126 outputs the determination result regarding the position of the equipment detected by the equipment detection unit 121 to the display device 3 via the output interface 14.

[0113] If the equipment detected by the equipment detection unit 121 is included in the equipment determined by the equipment detection position determination unit 126, the equipment detection position determination unit 126 determines that the position of the equipment detected by the equipment detection unit 121 is normal. On the other hand, if the equipment detected by the equipment detection unit 121 is not included in the equipment determined by the equipment detection position determination unit 126, the equipment detection position determination unit 126 determines that the position of the equipment detected by the equipment detection unit 121 is abnormal.

[0114] For example, if camera 2 is installed on the roof of a railway vehicle, the images taken by camera 2 from the railway vehicle as it travels on the tracks have high reproducibility in terms of the shooting position. Therefore, even if camera 2 has a fixed shooting direction and field of view, it is possible to repeatedly acquire images of the equipment to be diagnosed each time the vehicle travels. Consequently, in this case, it is possible to detect abnormalities in the position of the equipment using only one-dimensional positional information in the direction of travel.

[0115] As explained above, in this embodiment, an anomaly in the position of the equipment can be detected by comparing the position of the equipment at the time of image capture with the original installation position of the equipment stored in the equipment position database 133.

[0116] The image data used in the embodiments described above is generally color images, but is not limited to color images. For example, the present invention may also use grayscale images, near-infrared images, far-infrared images, and depth images.

[0117] In the above-described embodiment, a system for diagnosing overhead line equipment using a camera 2 installed on the roof of a railway vehicle was explained as an example. However, the present invention is not limited to diagnosing overhead line equipment. For example, by installing camera 2 on the underside of a railway vehicle, a track diagnostic system for detecting rail corrosion, deformation, etc., can be constructed. Furthermore, the mobile body on which camera 2 is installed is not limited to a railway vehicle. For example, by mounting camera 2 on a drone, a power line inspection system can be constructed. Alternatively, by mounting camera 2 on an automobile, a maintenance system for various facilities on and around roads, such as signs, traffic lights, guardrails, and mirrors, can be constructed. Alternatively, by installing camera 2 on a factory line, an inspection system can be constructed. Alternatively, by using an electron microscope as camera 2 to acquire images, a semiconductor inspection device can be constructed.

[0118] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are possible. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and the present invention is not necessarily limited to embodiments having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add configurations from other embodiments to the configuration of one embodiment. Furthermore, it is possible to delete parts of the configuration of each embodiment, or to add or replace other configurations.

[0119] Furthermore, each of the above configurations may be implemented either partially or entirely in hardware, or through program execution on a processor. Also, the control lines and information lines shown are those deemed necessary for illustrative purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, almost all configurations can be considered interconnected. [Explanation of Symbols]

[0120] 1... Equipment anomaly diagnosis device, 2... Camera, 3... Display device, 4... Trained AI model, 5... Location information acquisition device, 6... Anomaly candidate data generation device, 11... Input interface, 12... Processor, 13... Storage device, 14... Output interface, 15... Communication device, 61... Equipment database, 121... Equipment detection unit, 122... Anomaly candidate acquisition unit, 123... Anomaly detection unit, 124... Classification unit, 125... Shooting condition evaluation unit, 126... Equipment detection position determination unit, 131... Anomaly candidate database, 132... Specific anomaly database, 133... Equipment position database, F101... Image captured by camera, F102... Equipment for anomaly diagnosis, F103... Equipment image F103A...Image of normal equipment, F103B...Image of deformed equipment, F103C...Image of equipment acquired under conditions without backlighting, F103D...Image of equipment acquired under conditions with backlighting, F201A...Anomaly candidate data "No abnormality", F201B...Anomaly candidate data "Corrosion present", F201C...Anomaly candidate data "Deformation present", F301...Dropdown for equipment selection, F302...Dropdown for equipment database selection, F303...Dropdown for AI model selection, F304...Generate button, F305...Save button, F306...List of anomaly candidate data, F307...Delete button, F308...Edit button, F309...Add button, F401...Nut.

Claims

1. An equipment detection unit that detects the equipment to be inspected from the images captured by the camera, Abnormal candidate acquisition unit, Anomaly detection unit, Memory device and Equipped with, The aforementioned storage device stores abnormality candidate data that indicates the type of abnormality mode, which is the type of abnormality that may occur in the equipment, for each piece of equipment. The aforementioned abnormal candidate data is text-format data generated in an abnormal candidate data generation device by a trained AI model capable of generating text using the specifications of the equipment. The abnormality candidate acquisition unit acquires the abnormality candidate data for the equipment detected by the equipment detection unit from the storage device, The abnormality detection unit detects an abnormality in the equipment from the image by comparing the image with the abnormality candidate data acquired by the abnormality candidate acquisition unit, acquires the abnormality mode of the equipment, and outputs the acquired abnormality mode to a display device. A device for diagnosing equipment malfunctions, characterized by the following features.

2. The anomaly detection unit encodes the image and the anomaly candidate data into a common vector space, calculates the similarity between the vector encoded from the image and the vector encoded from the anomaly candidate data, and determines the anomaly candidate data whose similarity exceeds a predetermined threshold as the anomaly mode of the equipment. The equipment malfunction diagnosis device according to claim 1.

3. The anomaly detection unit adds a marker to the image indicating the location where the anomaly identified in the anomaly mode has occurred, and outputs the image with the marker added along with the anomaly mode. The equipment malfunction diagnosis device according to claim 1.

4. The aforementioned abnormal candidate data is data generated by the abnormal candidate data generation device using the AI ​​model and referring to the specifications, including the material, shape, and application of the equipment. The equipment malfunction diagnosis device according to claim 1.

5. The aforementioned abnormality candidate data is data obtained by the abnormality candidate data generation device by providing the AI ​​model with specific domain knowledge regarding the equipment using a knowledge graph that describes the relationships between the components constituting the equipment in a graph structure, and then questioning the AI ​​model about the abnormal modes that may occur in the equipment. The equipment malfunction diagnosis device according to claim 1.

6. The abnormal candidate data generation device displays a screen that allows the user to perform at least one of the following actions: delete the abnormal mode from the abnormal candidate data generated by the AI ​​model, or add the abnormal mode to the abnormal candidate data generated by the AI ​​model. The equipment malfunction diagnosis device according to claim 1.

7. In the aforementioned anomaly candidate data, the text and image indicating the anomaly mode are associated with each other. When detecting an anomaly in the equipment from the image, the anomaly detection unit performs a comparison between the image captured by the camera and the text of the anomaly candidate data, or performs a comparison between the image captured by the camera and the text of the anomaly candidate data, and a comparison between the image captured by the camera and the image of the anomaly candidate data. The equipment malfunction diagnosis device according to claim 1.

8. When the abnormality detection unit detects an abnormality in the equipment, it adds the image of the equipment in which the abnormality was detected to the abnormality candidate data that indicates the abnormality mode of the detected abnormality, from among the abnormality candidate data stored in the storage device. The equipment malfunction diagnosis device according to claim 1.

9. Equipped with a classification section, The aforementioned storage device stores specific abnormality data indicating a predetermined type of abnormality. The classification unit compares the abnormal mode acquired by the abnormality detection unit with the specific abnormal data, determines whether or not there is specific abnormal data that matches the abnormal mode, and outputs the result of this determination as the classification result of the abnormal mode. The equipment malfunction diagnosis device according to claim 1.

10. Equipped with a shooting condition evaluation unit, The aforementioned shooting condition evaluation unit obtains information regarding the presence or absence of disturbances in the image by querying a trained AI model capable of receiving questions using the image and generating text, and outputs the obtained information to the display device. The equipment malfunction diagnosis device according to claim 1.

11. A location information acquisition device that acquires the position where the image is captured by the camera, Equipment detection position determination unit, Equipped with, The storage device stores data relating to the original installation location of the equipment. The equipment detection position determination unit is, The shooting position is obtained from the aforementioned position information acquisition device, From the aforementioned storage device, find the equipment whose original installation location is within a predetermined range from the aforementioned shooting location. The equipment detection unit compares the detected equipment with the requested equipment to determine whether there is an abnormality in the equipment's position, and outputs the determination result regarding the equipment's position to the display device. The equipment malfunction diagnosis device according to claim 1.

12. The storage device contains abnormality candidate data that indicates the type of abnormality mode, which is the type of abnormality that may occur in the equipment being inspected, for each piece of equipment. The aforementioned abnormal candidate data is text-format data generated by a trained AI model capable of generating text using the specifications of the equipment. A device detection step in which the device is detected from the image captured by the camera, An abnormality candidate acquisition step involves acquiring the abnormality candidate data for the equipment detected in the equipment detection step from the storage device, An anomaly detection step involves comparing the aforementioned image with the anomaly candidate data acquired in the anomaly candidate acquisition step to detect an anomaly in the equipment from the image, acquire the anomaly mode of the equipment, and output the acquired anomaly mode to a display device. A method for diagnosing equipment malfunctions, characterized by having the following features.

13. A camera mounted on the roof of a train car, The equipment malfunction diagnosis device according to claim 1, Display device and An anomaly candidate data generation device, Equipped with, The equipment to be inspected is the overhead line equipment of a railway. A catenary equipment diagnostic system characterized by the following features.

Images