Equipment inspection device, equipment inspection method, and equipment inspection program

Abstract

This helps to prevent errors in specifying the objects to be inspected. [Solution] To solve the above problems, the equipment inspection device according to this embodiment comprises an image acquisition unit, a first position information acquisition unit, a second position information acquisition unit, and a position information determination unit. The image acquisition unit captures an image of the external scenery of the vehicle at a certain vehicle position using a camera installed on the vehicle. The first position information acquisition unit automatically detects the inspection target object included in the image and acquires first position information indicating the position of the inspection target object in the image. The second position information acquisition unit acquires second position information indicating the position in the image of the inspection target object specified by the inspector riding in the vehicle at the vehicle position. The position information determination unit determines third position information indicating the position of the inspection target object to be inspected in the automatic inspection based on the first position information and the second position information.

Description

Technical Field

[0001] Embodiments of the present invention relate to an equipment inspection device, an equipment inspection method, and an equipment inspection program.

Background Art

[0002] Along railway lines, various facilities such as electrical facilities, communication facilities, signal facilities, and civil engineering facilities (rails, tunnels, bridges) are installed. In recent years, the aging of facilities along railway lines and the shortage of inspectors have overlapped, increasing the load of maintenance and inspection work. Therefore, a method of taking images (inspection images) of railway lines with a dedicated inspection vehicle after the end of business operations, such as at night, and examining the presence or absence of abnormalities in the facilities by image analysis has been experimentally conducted. In this method, first, an inspector manually designates the position of the inspection target object on the inspection image, and then, using an abnormality detection algorithm, determines the presence or absence of abnormalities in the range including the designated position.

[0003] However, the number and types of facilities along railway lines to be inspected are enormous, and it is not realistic for an inspector to designate the positions of all inspection target objects while looking at inspection images.

[0004] On the other hand, it is possible to detect inspection target objects by circle detection, line detection, or object detection by AI using image processing and automatically designate their positions. However, automatically detectable inspection target objects are limited to some facilities with easily detectable features. Also, in the case of automatic detection, designation errors of inspection target objects may occur.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

[0006] The problem that the present invention aims to solve is to provide an equipment inspection device, an equipment inspection method, and an equipment inspection program that can suppress errors in specifying the object to be inspected. However, the present invention is not limited to this problem, and the problems corresponding to the effects of the configurations of each embodiment described later may also be considered problems that the present invention aims to solve. [Means for solving the problem]

[0007] The equipment inspection device according to the embodiment includes an image acquisition unit, a first position information acquisition unit, a second position information acquisition unit, and a position information determination unit. The image acquisition unit captures an image of the external scenery of the vehicle at a certain vehicle location using a camera installed on the vehicle. The first position information acquisition unit automatically detects the inspection target object included in the image and acquires first position information indicating the position of the inspection target object in the image. The second position information acquisition unit acquires second position information indicating the position in the image of the inspection target object specified by the inspector riding in the vehicle at the vehicle location. The position information determination unit determines third position information indicating the position of the inspection target object to be inspected in the automatic inspection based on the first position information and the second position information. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of the equipment inspection system according to this embodiment. [Figure 2] This figure shows an example of an equipment inspection system according to this embodiment, mounted on a vehicle. [Figure 3] This is a functional block diagram of the equipment inspection device according to this embodiment. [Figure 4] This figure shows an example of a storage unit for an equipment inspection device according to this embodiment. [Figure 5]This figure shows an example of an inspection target information database stored in the memory unit according to this embodiment. [Figure 6] This is a flowchart illustrating the method for determining the object to be inspected according to this embodiment. [Figure 7] This figure shows an example of a sample registration screen for an object to be inspected according to this embodiment. [Figure 8] This diagram illustrates the designation of an object to be inspected by an inspector based on their line of sight, according to this embodiment. [Figure 9] This is another diagram illustrating the designation of an object to be inspected by the inspector's line of sight according to this embodiment. [Figure 10] This is a flowchart illustrating the automated inspection according to this embodiment. [Figure 11] This figure shows an example of an inspection results database according to this embodiment. [Figure 12] This is an example of an image taken during abnormality detection according to this embodiment. [Figure 13] This is an example of a confirmation screen for inspection results according to this embodiment. [Figure 14] This is a first example of the importance setting screen according to this embodiment. [Figure 15] This is a second example of the importance setting screen according to this embodiment. [Modes for carrying out the invention]

[0009] The equipment inspection device, equipment inspection method, and equipment inspection program of the embodiment will be described below with reference to the drawings. In each figure, components having equivalent functions are denoted by the same reference numeral, and detailed descriptions of components with the same reference numeral will not be repeated.

[0010] <Equipment Inspection System 1> First, the configuration of the equipment inspection system 1 according to this embodiment will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing an example of the configuration of the equipment inspection system 1. Figure 2 is a diagram showing an example of the equipment inspection system 1 mounted on a vehicle.

[0011] As shown in FIGS. 1 and 2, the facility inspection system 1 is provided in the vehicle 1000 and includes a facility inspection device 10, a line-of-sight sensing device 20, an importance reception device 30, a photographing device 40, and a train operation management device 50. As shown in FIG. 1, in the present embodiment, the facility inspection device 10 and the line-of-sight sensing device 20 are wirelessly connected, and the other connections are wired. However, whether each connection is wired or wireless is not limited to this and is arbitrary.

[0012] As shown in FIG. 2, the inspector P rides in the vehicle 1000 together with a driver (not shown), and visually recognizes the facility A along the line that appears ahead during the running of the vehicle 1000 as an inspection object. In the case of electrical equipment, the inspection objects are, for example, trolley wires, suspension wires, insulators, curve draw gears, movable brackets, hanger ears, etc. The same applies to other facilities and is not limited thereto. The inspection objects are not limited to electrical equipment and may be communication equipment, signal equipment, civil engineering equipment, etc.

[0013] The facility inspection device 10 is configured to acquire an image of the external scenery of the vehicle 1000 from the photographing device 40, acquire the position information of the inspection object specified by the inspector P by line of sight from the line-of-sight sensing device 20, and determine the position information of the inspection object to be inspected in the automatic inspection for automatically performing the facility inspection along the line. The details of the facility inspection device 10 will be described later.

[0014] The line-of-sight sensing device 20 is a device for the inspector P to specify an inspection object by line of sight. The line-of-sight sensing device 20 acquires the line-of-sight information of the inspector P and transmits it to the facility inspection device 10. As shown in FIG. 1, in the present embodiment, the line-of-sight sensing device 20 has, for example, a head-up display worn by the inspector P. When the inspector P discovers the facility to be inspected while looking at the forward scenery displayed on the head-up display, the inspector P gazes at the facility. Thereby, the inspection object is specified. The automatically detected facility may be displayed by a bounding box or the like so that the inspector P can easily visually recognize it.

[0015] The gaze-sensing device 20 may also be a head-up display of a type not worn by the inspector P. In this case, a transparent display may be installed in front of the inspector P as the gaze-sensing device 20. Furthermore, the inspector P is not limited to specifying the inspection target by gaze; the inspection target may also be specified using a mouse or pointer on the transparent display.

[0016] The importance level receiving device 30 receives the importance level specified by the inspector P for each object to be inspected and transmits that information (importance level information) to the equipment inspection device 10.

[0017] In this embodiment, the importance receiving device 30 has a high importance button 31, a medium importance button 32, and a low importance button 33. The high importance button 31 is for setting the highest importance level, the medium importance button 32 is for setting an intermediate importance level, and the low importance button 33 is for setting the lowest importance level. Although not shown, the importance receiving device 30 may also have a cancel button for canceling a button press. Furthermore, the high importance button 31, the medium importance button 32, and the low importance button 33 are not limited to physical buttons or switches, but may also be objects displayed on the touch panel of a tablet terminal.

[0018] When inspector P presses the high-priority button 31, medium-priority button 32, or low-priority button 33 while intently observing an object to be inspected, importance information corresponding to the pressed button is transmitted to the equipment inspection device 10.

[0019] For example, the display of the gaze detection device 20 displays candidates for inspection objects that have been automatically detected in the captured image, enclosed in a rectangle. The inspector P directs their gaze towards the object and presses the high-importance button 31, medium-importance button 32, or low-importance button 33 to specify the location and importance level of the inspection object. Alternatively, if the candidate for inspection object displayed on the display is correct, the inspector P does nothing; if it is incorrect, they direct their gaze towards the object and then press the cancel button on the importance level receiving device 30 to cancel the location specification of the inspection object.

[0020] The camera 40 acquires an image of the external scenery (forward image) of the vehicle 1000 and transmits the image data to the equipment inspection device 10. As shown in Figure 2, the camera 40 is mounted at the front of the vehicle 1000, facing forward, and acquires an image of the front of the vehicle 1000 (forward image). The camera 40 may also acquire a side image or a rear image of the vehicle 1000. Furthermore, the camera 40 is not limited to one unit, but may consist of two or more units.

[0021] The train operation management device 50 acquires the location information of the vehicle 1000 and transmits this location information to the equipment inspection device 10. In this embodiment, the vehicle location information is the distance indicated by the markers (kilometer posts) that indicate the distance from the starting point on the railway line. The vehicle location information may also be information about the latitude and longitude of the vehicle, such as GPS information.

[0022] <Equipment inspection device 10> Next, with reference to Figure 3, the equipment inspection device 10 according to this embodiment will be described in detail.

[0023] The equipment inspection device 10 comprises a communication unit 11, a storage unit 12, and a processing unit 13. The processing unit 13 comprises an image acquisition unit 13a, a first position information acquisition unit 13b, a second position information acquisition unit 13c, a position information determination unit 13d, an importance acquisition unit 13e, a position information correction unit 13f, an anomaly determination unit 13g, and a database creation unit 13h. In this embodiment, the processing unit 13 is composed of one or more processors such as a CPU (Central Processing Unit), and functions corresponding to each program are realized by reading and executing programs from the storage unit 12. At least a portion of the functions of the processing unit 13 may be realized by hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0024] The communication unit 11 is an interface that transmits and receives data and other information to and from each device of the equipment inspection system 1 via wireless communication and / or wired communication over a communication network.

[0025] The storage unit 12 is composed of, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, and a hard disk. This storage unit 12 stores image data captured by the imaging device 40, a database described later, and various processing programs executed by the processing unit 13.

[0026] As shown in Figure 4, the storage unit 12 of this embodiment stores the following: an inspection target sample database 12a, an inspection target information database 12b, an inspection target image database 12c, an inspection result information database 12d, and an inspection result image database 12e.

[0027] The Inspection Target Sample Database 12a is a database that stores sample data for each type of inspection target. Sample data refers to image data of the inspection target, for example, image data of an inspection target in a normal state that has been registered in advance.

[0028] The inspection target information database 12b is a database that stores information about the inspection targets to be inspected in automated inspections. Figure 5 shows an example of the inspection target information database 12b according to this embodiment. In this example, for each inspection target, vehicle location information, inspection target ID, inspection target location information (top-left and bottom-right coordinates of the frame) and importance level are stored in association with each other.

[0029] In the example in Figure 5, the "Vehicle Location Information" column stores the location information of vehicle 1000 (here, the kilometer post value) received by the equipment inspection device 10 from the train operation management device 50. In this embodiment, the vehicle location information is also used as identification information (image ID) for the image taken at that vehicle location. The "Inspection Object ID" column stores the identification information (inspection object ID) assigned to the inspection object included in the image at that vehicle location. The "Inspection Object Location Information" column stores the position coordinates of the inspection object on the image (third location information described later). In this embodiment, the coordinates of two vertices (top-left and bottom-right coordinates of the frame) that define the rectangular frame (bounding box, etc.) surrounding the inspection object are stored as the location information of the inspection object. The "Importance" column stores information indicating the importance of each inspection object (high, medium, or low) received by the equipment inspection device 10 from the importance acquisition unit 13e.

[0030] The inspection target image database 12c is a database that stores image data of the inspection targets to be inspected in automated inspections. The inspection target image database 12c stores image data of the inspection targets when they are in normal condition. Image data of the inspection targets when they are abnormal may also be stored in the database.

[0031] The inspection result information database 12d is a database that stores the results of abnormality detection of inspected objects by automated inspection. The information stored in the inspection result information database 12d is information on all inspected objects that have undergone automated inspection. Note that the database may store only information on inspected objects that were found to be normal, or only information on inspected objects that were found to be abnormal.

[0032] The inspection result image database 12e is a database that stores image data of inspected objects taken during automated inspections. The information in the inspection result information database 12d and the information in the inspection result image database 12e are linked by the inspected object ID.

[0033] Furthermore, at least one of the databases stored in the storage unit 12 may be stored on a server (not shown) located outside the vehicle 1000 and wirelessly connected to the equipment inspection device 10.

[0034] Next, we will explain in detail each function of the processing unit 13.

[0035] The image acquisition unit 13a acquires images captured by the imaging device 40 via the communication unit 11. While the vehicle 1000 is in motion, the image acquisition unit 13a continuously acquires images captured by the imaging device 40. In addition, when registering sample data, the image acquisition unit 13a acquires sample data for each type of inspection target.

[0036] The first position information acquisition unit 13b detects the object to be inspected included in the image acquired by the image acquisition unit 13a. The first position information acquisition unit 13b then acquires the position coordinates (first position information) indicating the position of the object to be inspected within the image. In this embodiment, the first position information acquisition unit 13b automatically detects the object to be inspected within the image based on sample data that has been pre-registered and stored in the object to be inspected sample database 12a, and determines the position coordinates indicating the position of the detected object to be inspected within the image.

[0037] The second position information acquisition unit 13c acquires position coordinates (second position information) indicating the position in the image of the object to be inspected, as specified by the inspector P. In this embodiment, the second position information acquisition unit 13c acquires position coordinates based on the gaze information of the inspector P detected by the gaze sensing device 20. These position coordinates indicate, for example, the area or location that the inspector P gazed upon for a predetermined time or longer.

[0038] The position information determination unit 13d determines position information (third position information) indicating the location of the object to be inspected during the automatic inspection, based on the first position information acquired by the first position information acquisition unit 13b and the second position information acquired by the second position information acquisition unit 13c. For example, the position information determination unit 13d determines the third position information by comparing the first position information and the second position information. The method for determining the third position information will be explained in detail later with reference to Figures 8 and 9.

[0039] The importance acquisition unit 13e acquires importance information set by the inspector P for each object to be inspected using the importance receiving device 30, via the communication unit 11 from the importance receiving device 30. In this embodiment, the database creation unit 13h stores the importance information acquired by the importance acquisition unit 13e in the object to be inspected information database 12b.

[0040] The location information correction unit 13f acquires location information corrected by the inspector P. This corrected location information is information that has been manually corrected by the inspector P from the location information determination unit 13d. The location information correction unit 13f may also acquire importance information corrected by the inspector P. The specific methods for correcting location information and importance will be described later.

[0041] The abnormality determination unit 13g determines whether the object to be inspected, as captured by the imaging device 40, is abnormal during automatic inspection. In this embodiment, the abnormality determination unit 13g determines whether or not there is an abnormality in the equipment based on the position coordinates determined by the position information determination unit 13d. Alternatively, the abnormality determination unit 13g may determine whether or not there is an abnormality in the equipment based on the position coordinates and importance.

[0042] The database creation unit 13h stores the acquired information in the respective databases of the storage unit 12. For example, the database creation unit 13h stores sample data in the inspection target sample database 12a. More details will be provided later.

[0043] <Equipment Inspection Methods> Next, the equipment inspection method according to this embodiment will be described. Here, train inspection refers to an inspection conducted by an inspector riding in the train with the driver and visually inspecting the equipment along the railway line. First, referring to Figure 6, the method for determining the objects to be inspected in the automated inspection will be explained, and then referring to Figure 10, the automated inspection method will be explained. Figure 6 shows a flowchart illustrating the method for determining the objects to be inspected according to this embodiment. Step S11 is performed prior to the train inspection, and steps S12 to S18 are performed during the train inspection.

[0044] Step S11: Register samples of the objects to be inspected. For example, an inspector registers samples of the objects to be inspected using image data of the vehicle's exterior scenery that has been photographed in advance. Specifically, the inspector registers sample data for each type of object to be inspected. Note that multiple sample data taken from different directions may be registered for the same type of object to be inspected. In addition, the registration of samples of objects to be inspected may be performed by someone other than the inspector. Through this step, for example, sample data for all types and orientations of all objects to be inspected is obtained. The database creation unit 13h stores the registered samples of objects to be inspected in the object sample database 12a.

[0045] Figure 7 shows an example of the screen for registering a sample of the object to be inspected in step S11. The device displaying this screen is an information processing device that is communicatively connected to the equipment inspection device 10. While the information processing device is assumed to be a personal computer, it may also be a tablet device or a dedicated terminal with a screen. A video of the vehicle's operation is stored in the information processing device beforehand.

[0046] As shown in Figure 7, the insulator 301 and the transmission tower 302 in the image are registered as objects to be inspected. The inspector can use the control button 305 to play and stop the video. In addition, the seek bar 306 can be used to display an image (frame) at a desired vehicle position. The current position of the vehicle corresponding to the displayed image is shown in current position 309.

[0047] This section explains how to register a new object to be inspected. If the object to be inspected is visible in the image, the inspector drags the mouse or touch panel to draw a rectangle around the object. Then, the inspector presses the Add button 307 to display the settings on the Item Settings screen 312. Next, the inspector selects the type of object to be inspected from the list of objects to be inspected on the Item Settings screen 312. If the type does not exist in the list of objects to be inspected, the inspector presses the New Type button 313 to create a new type. After selecting the type of object to be inspected, the inspector selects the importance level of the object to be inspected (in this case, a transmission tower) in the Settings 311, and then presses the OK button (not shown) that appears. This adds the object to the List of Objects to be Inspected 310.

[0048] When an item is selected from the list of items to be inspected 310, the information for that item is displayed in the settings 311 as shown in the figure, and it is possible to modify the location information, importance level, etc. of the item to be inspected. In this embodiment, as shown in Figure 7, the location information is the coordinates of the top-left and bottom-right points of the rectangle. Note that the shape representing the item to be inspected is not limited to a rectangle, but may also be a circle, ellipse, etc. In the case of a circle, the coordinates of its center and radius may be used as the location information of the item to be inspected. To cancel the registration of coordinates, select the item by clicking on the rectangle in the list of items to be inspected 310 or on the image, and then press the delete button 308.

[0049] The database creation unit 13h stores the image data of the object to be inspected, enclosed in a rectangle, into the object sample database 12a.

[0050] Step S12: The image acquisition unit 13a determines whether or not a new image is input from the imaging device 40. If a new image is input (S12: Yes), the process proceeds to step S13; otherwise, the process ends. In other words, as long as the vehicle position changes and a new image is acquired by the image acquisition unit 13a (as long as the train inspection run is not finished), steps S13 to S18 are repeated.

[0051] Step S13: The first position information acquisition unit 13b determines whether or not there are any unprocessed inspection targets in the image input (acquired) in step S12. If there are unprocessed inspection targets (S13: Yes), the process proceeds to step S14; otherwise, it returns to step S12. That is, as long as there are inspection targets in an image of a certain vehicle position whose position coordinates (third position information) have not been determined, steps S14 to S18 are repeated for that image. When the position coordinates (third position information) of all inspection targets in the image have been determined, the process returns to step S12.

[0052] Furthermore, the algorithm used for automatically detecting objects to be inspected may employ techniques that detect multiple objects in an image in a single process, such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector). Alternatively, machine learning models for object detection, such as R-CNN (Region Based Convolutional Neural Network), may be used to automatically detect objects to be inspected.

[0053] Step S14: The first position information acquisition unit 13b acquires the first position information of the inspection target detected by automatic detection. Specifically, using the inspection target sample database 12a created in step S11, the unit automatically detects inspection targets in the image acquired in step S12 that match the type and orientation of the sample data, and acquires the position coordinates (first position information) of the detected inspection target. For example, the first position information is information that defines the rectangle surrounding the detected inspection target (bottom right coordinates and top left coordinates of the frame).

[0054] Step S15: The second position information acquisition unit 13c acquires second position information of the object to be inspected based on the position specified by the inspector. More specifically, the second position information acquisition unit 13c acquires position coordinates (second position information) indicating the position of the object to be inspected in the image, based on the inspector's gaze information transmitted from the gaze sensing device 20. For example, the second position information is information (center coordinates and radius) that defines a circle surrounding the area that the inspector has been looking at. Alternatively, the second position information may be grayscale image information, where the color of the image is displayed more intensely the longer the inspector has been looking at the location. The second position information acquisition unit 13c may also acquire the importance level received from the importance level receiving device 30 for the object to be inspected.

[0055] Step S16: The position information determination unit 13d determines the third position information based on the first and second position information. Specifically, the position information determination unit 13d compares the position coordinates (first position information) acquired in step S14 with the position coordinates (second position information) acquired in step S15 to determine the position coordinates (third position information) of the object to be inspected to be used during automatic inspection. More specifically, this will be explained in detail with reference to Figures 8 and 9.

[0056] Figure 8 is a diagram illustrating a first example of a method for determining the position coordinates (third position information) of an object to be inspected in this embodiment. The rectangle 401 shows a rectangular area surrounding the object to be inspected based on the first position information. The circle 402 shows a circular area surrounding the object to be inspected based on the second position information derived from the inspector's line of sight. The line segment 403 illustrates the information in the up, down, left, and right directions of the inspector's line of sight for determining the circle 402 determined by the second position information.

[0057] As shown in Figure 8, if the circle 402 is contained within the rectangle 401, the position information determination unit 13d determines the first or second position information as the third position information. On the other hand, if the circle 402 is not contained within the rectangle 401 at all, the position information determination unit 13d does not determine the third position information. In this case, the database creation unit 13h may store a flag in the inspection target information database 12b indicating that the third position information cannot be determined. Note that this determination criterion is merely an example. As another example, a criterion of whether the center coordinates of the circle 402 are contained within the rectangle 401 may be adopted. Furthermore, the relative sizes of the rectangle 401 and the circle 402 are not limited to those shown in Figure 8. If the circle 402 is larger than the rectangle 401 and the rectangle 401 is contained within the circle 402, the first position information may be determined as the position coordinates of the inspection target (third position information). More generally, the position information determination unit 13d may determine the first or second position information as the third position information if there is an overlapping portion between the area determined by the first position information and the area determined by the second position information.

[0058] Figure 9 illustrates another example of the method for determining the third position information in this embodiment. In this example, as shown in Figure 9, a grayscale image 502 based on the second position information is displayed. In the grayscale image 502, the darker the color of the image, the longer the inspector has looked at that position.

[0059] As shown in Figure 9, if the entire grayscale image 502 is contained within the rectangle 501, the position information determination unit 13d determines the first or second position information as the position coordinates of the object to be inspected (third position information). If the grayscale image 502 is not contained within the rectangle 501 at all, the third position information is not determined. In this case, the database creation unit 13h may store a flag in the object to be inspected information database 12b indicating that the third position information cannot be determined. Note that this determination criterion is merely an example. As an example, a criterion of whether a part of the grayscale image 502 is contained within the rectangle 501 may be adopted. Furthermore, the relative sizes of the rectangle 501 and the grayscale image 502 are not limited; if the grayscale image 502 is larger than the rectangle 501 and the rectangle 501 is contained within the grayscale image 502, the first position information may be determined as the position coordinates of the object to be inspected (third position information). In the second example, a grayscale image is used, but as a variation, an image could be used in which areas where the inspector spent less time looking are shown in blue (first color), and as the time spent looking at the area increases, the image transitions from blue to red (second color). Alternatively, an image based on the second location information could be a heatmap image.

[0060] Step S17: The database creation unit 13h stores the position coordinates (third position information) determined in step S16 in the inspection object information database 12b, associating them with other information such as the vehicle position and / or the inspection object (identification information, etc.). By referring to the inspection object information database 12b created in this way, it is possible to determine which inspection object in the image associated with the vehicle position information should be inspected for abnormality detection.

[0061] Step S18: The database creation unit 13h stores the image of the object to be inspected in the third location information into the object image database 12c. The image is stored in the object image database 12c in association with the vehicle location information.

[0062] After the above processing flow, the inspection target information database 12b used during automated inspection is created. For inspection targets for which the third position information could not be determined in step S16, the inspector may manually determine the third position information after the above processing flow is completed by reviewing the image containing the inspection target. After the above processing flow is completed, the inspector may also correct the third position information and / or importance as necessary. In this case, the position information correction unit 13f acquires the corrected third position information and / or importance entered by the inspector. The corrected third position information and / or importance is then stored in the inspection target information database 12b.

[0063] The above processing flow is merely an example, and various modifications are possible. For example, step S15 may be executed before step S14, or steps S14 and S15 may be executed simultaneously. Also, step S18 may be executed before step S17, or steps S17 and S18 may be executed simultaneously.

[0064] Furthermore, the above processing flow (steps S12 to S18) is not limited to being performed during train inspections, but may also be performed on previously recorded video footage.

[0065] Furthermore, the importance level of an object to be inspected is not limited to being set by the importance level reception device 30. For example, the importance level may be set based on the number of times an inspector has performed multiple train inspections. For example, an object that has been designated 3 times may be set to "medium" importance, and an object that has been designated 5 or more times may be set to "high" importance. Note that multiple train inspections may be performed by one inspector multiple times, or by multiple inspectors each performing one inspection.

[0066] According to the above equipment inspection method (method for determining inspection targets), inspection targets included in an image of the external scenery of vehicle 1000 at a certain vehicle position are automatically detected, first position information indicating the position of the inspection target in the image is obtained, second position information indicating the position of the inspection target specified by inspector P at that vehicle position is obtained, and third position information indicating the position of the inspection target to be inspected in the automatic inspection is determined based on the first and second position information. This makes it possible to suppress errors in specifying the inspection targets to be inspected in the automatic inspection. In other words, it is possible to improve the accuracy of specifying inspection targets. Furthermore, by using the third position information obtained by the above method, accurate automatic inspection can be performed.

[0067] Furthermore, by using the gaze detection device 20, a large number of inspection targets can be designated without placing an excessive burden on the inspector P.

[0068] <Automatic Inspection> Using the third position information determined by the equipment inspection method described above, equipment inspections along the railway line are performed automatically. This automated inspection makes it possible to inspect equipment (daily inspections) that do not require an inspector to be on board during operating hours. Figure 10 is a flowchart illustrating the automated inspection according to this embodiment. The automated inspection will be explained in detail with reference to Figure 10.

[0069] Step S21: The image acquisition unit 13a determines whether or not a new image is input from the imaging device 40. If a new image is input (S21: Yes), proceed to step S22; otherwise, proceed to step S26. As long as the vehicle position changes and a new image is acquired by the image acquisition unit 13a (until the automatic inspection run is completed), steps S22 to S25 are repeated.

[0070] Step S22: The abnormality determination unit 13g determines whether or not there are any unprocessed inspection targets in the image input (acquired) in step S21. If there are unprocessed inspection targets (S22: Yes), the process proceeds to step S23; otherwise, it returns to step S21. In other words, as long as there are inspection targets that have not been abnormally determined in the image of a certain vehicle position, steps S23 to S25 are repeated for that image. When the abnormality determination and associated processing of all inspection targets in the image are completed, the process returns to step S21.

[0071] Step S23: The abnormality determination unit 13g performs an abnormality determination on the object to be inspected in the image acquired in step S21. This abnormality determination is performed on the object to be inspected at the third position information of the image taken at the vehicle position, by referring to the vehicle position information and the object to be inspected position information stored in the object to be inspected information database 12b. This abnormality determination is made by comparing it with the image of the object to be inspected stored in the object to be inspected image database 12c.

[0072] For example, the abnormality detection unit 13g calculates the difference (difference in brightness, etc.) between the image of the object to be inspected in the image acquired in step S21 and the image of the object to be inspected stored in the object to be inspected image database 12c, and determines that the object to be inspected is abnormal if the difference is greater than or equal to a standard value. Note that the standard value may be changed according to the importance of the object to be inspected. For example, the standard value can be set lower as the importance of the object to be inspected increases. This makes it possible to increase the abnormality detection sensitivity for highly important objects to be inspected.

[0073] Furthermore, the abnormality detection in this step may be performed using a trained model generated by machine learning with images of the object to be inspected as training data. In this case, the trained model may be stored in the storage unit 12, or it may be stored on a server outside the equipment inspection system 1. The images of the object to be inspected used as training data may be images of the object in a normal state, images of an abnormal state, or both normal and abnormal states. Image data from the object to be inspected image database 12c may also be used as training data.

[0074] Figure 11 shows an example of an image used for abnormality detection according to this embodiment. In this example image, as shown in the enlarged image 602, areas with larger differences compared to the normal image are displayed using a grayscale image, with the color becoming darker in those areas.

[0075] In the image in Figure 11, an abnormality in the insulator 601 is detected. The enlarged image 602 shows a magnified view of the insulator 601. In this example, the crack in the porcelain part 603 (lower left of the enlarged image) and the detachment of the device 604 (lower center of the enlarged image) are displayed in darker colors and are judged to be abnormal. Although a grayscale image is used in this example, a heatmap image where areas with larger differences compared to a normal image are displayed in red would also be acceptable.

[0076] Step S24: The database creation unit 13h stores the results of the abnormality determination related to the inspected object performed in step S23 into the inspection result information database 12d. The inspection result information database 12d will be described in detail with reference to Figure 12. Figure 12 is a diagram showing an example of the inspection result information database 12d according to this embodiment.

[0077] As shown in Figure 12, in the inspection result information database 12d, vehicle location information, inspection object ID, inspection object location information, and judgment result are stored in association with each inspection object. The vehicle location information, inspection object ID, and inspection object location information are the same as those in the inspection object information database 12b described in Figure 5. In the "Judgment Result" column, the result of the abnormality judgment performed on the inspection object is stored as OK if it is normal, and NG if it is abnormal.

[0078] Step S25: The database creation unit 13h stores the images used for abnormality detection in the inspection result image database 12e. These images are stored in association with the vehicle location information and inspection target ID in the inspection result information database 12d.

[0079] Step S26: The location information correction unit 13f corrects the third location information and / or importance. Note that this step is not necessarily required and may not be performed unless a particular problem occurs.

[0080] The correction process in step S26 will be described in more detail with reference to Figure 13. Figure 13 is an example of a screen for confirming the inspection results according to this embodiment. The device that displays this screen is a personal computer that is communicatively connected to the equipment inspection device 10. This device may be a tablet terminal or a dedicated terminal with a screen.

[0081] As shown in Figure 13, the image displays the insulator 701, the transmission tower 702, the transmission tower 703, and the tension adjustment mechanism 704 as objects to be inspected. The current location of the vehicle that acquired the image is displayed as current location 709.

[0082] The Inspection Object List 710 is a list of inspection objects present in the image, and the abnormality determination result (○ or ×) is displayed. The Result Details 711 displays an enlarged image of the abnormality determination for the inspection object selected in the Inspection Object List 710. The sign 715 indicates that the inspection object, which has been set to "High" in importance, has been determined to be abnormal.

[0083] Furthermore, by pressing the report button 714, it is possible to generate a summary of the results of the automatic inspection. The control button 705 allows for operations such as playing and stopping the video. Additionally, the seek bar 706 allows for displaying images (frames) at a desired vehicle position.

[0084] Furthermore, within the range in which the seek bar 706 moves, the portion corresponding to the image where an abnormality was detected (the image showing the inspection target that was judged to be abnormal) may be highlighted with a background color or pattern, as shown in displays 707 and 708.

[0085] Furthermore, if it is necessary to correct the positional information (third positional information) of an object to be inspected within an image, the inspector may correct the positional information (third positional information) of the object by dragging the rectangular frame surrounding the object with a mouse or touch panel to deform the rectangular frame. Note that the specific correction method is not limited to this; the inspector may also correct it by re-specifying the coordinates of the centroid of the rectangular frame, or by inputting the coordinates of the points that define the rectangular frame.

[0086] In Figure 13, if the importance level of an object to be inspected in the image needs to be adjusted, the inspector selects the rectangle of the object to be inspected, or selects an item in the object to be inspected list 710, and then presses the importance button 712 to display the importance setting screen. The importance setting screen will be explained in detail with reference to Figures 14 and 15.

[0087] Figure 14 shows a first example of the importance setting screen according to this embodiment. The inspector can set a continuous importance level (sensitivity) by dragging the slider 802 to move it left or right on the slider bar 801.

[0088] Figure 15 shows a second example of the importance setting screen according to this embodiment. The inspector can set the importance level (sensitivity) in stages by pressing the radio buttons for high importance 901, medium importance 902, or low importance 903. In this example, there are three importance levels, but the number of levels can be four or more, or two or less.

[0089] In this embodiment, the higher the importance level of the object to be inspected, the lower the threshold (criterion) at which the abnormality detection unit 13g determines an abnormality. As a result, for objects with a higher importance level, even minor differences are judged as abnormal. This reduces the likelihood of missing abnormalities in objects with high importance level.

[0090] The above processing flow is merely an example, and various modifications are possible. For example, step S25 may be executed before step S24, or steps S24 and S25 may be executed simultaneously.

[0091] According to the automated inspection method described above, inspections of the target objects can be performed automatically and with high accuracy during operation within business hours.

[0092] According to the embodiments described above, by determining the position coordinates of the object to be inspected in the automatic inspection (third position information) based on the position coordinates of the object to be inspected (first position information) set by automatic detection and the position coordinates of the object to be inspected (second position information) set by the inspector, it is possible to provide an equipment inspection device, equipment inspection method, and equipment inspection program that can suppress errors in specifying the object to be inspected.

[0093] In the above embodiment, the vehicle was a railway, but other means of transportation may also be used, such as a bus, monorail, or automobile.

[0094] Furthermore, although the above-described equipment inspection device 10 was capable of both creating a database for train inspections and performing automatic inspections using the said database, it may perform only one of these functions. That is, if only the creation of a database for train inspections is performed, the equipment inspection device 10 does not need to include at least the abnormality determination unit 13g. Also, if only automatic inspections are performed, the equipment inspection device 10 does not need to include at least the second position information acquisition unit 13c, the position information determination unit 13d, and the importance acquisition unit 13e.

[0095] At least a portion of the processing unit 13 of the equipment inspection device described in the above-described embodiment may be configured as hardware or as software. If configured as software, a program that implements at least a portion of the functions of the processing unit 13 of the equipment inspection device may be stored on a recording medium such as a flexible disk or CD-ROM, loaded into a computer, and executed. The recording medium is not limited to removable ones such as magnetic disks or optical disks, but may also be a fixed recording medium such as a hard disk drive or memory.

[0096] Furthermore, a program that implements at least some of the functions of the processing unit 13 of the equipment inspection device may be distributed via communication lines such as the Internet (including wireless communication). In addition, the program may be encrypted, modulated, or compressed and distributed via wired or wireless lines such as the Internet, or stored on a recording medium.

[0097] While embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0098] 1. Equipment Inspection System 10 Equipment Inspection Devices 11 Communications Department 12 Storage section 12a Sample Database of Items to be Inspected 12b Database of Inspection Targets 12c Inspection Target Image Database 12d Inspection Result Information Database 12e Inspection Result Image Database 13 Processing Unit 13a Image acquisition unit 13b 1st location information acquisition unit 13c Second location information acquisition unit 13d Location information determination section 13e Importance acquisition part 13f Location information correction section 13g Abnormality determination section 13h Database Creation Department 20 Gaze detection device 30. Importance Reception Device 31 High Importance Buttons 32 Medium Importance Buttons 33 Low Importance Buttons 40 Imaging device 50 Train operation control device 1000 vehicles A. Facilities along the railway line P Inspector

Claims

1. An image acquisition unit that captures an image of the external scenery of the vehicle at a certain vehicle position using a camera device installed on the vehicle, A first position information acquisition unit automatically detects an object to be inspected included in the image and acquires first position information indicating the position of the object to be inspected within the image. A second position information acquisition unit acquires second position information indicating the position in the image of an object to be inspected, which is designated by an inspector riding in the vehicle at the aforementioned vehicle position. A position information determination unit determines a third position information indicating the position of the object to be inspected in the automatic inspection, based on the first position information and the second position information. Equipment inspection device equipped with the following features.

2. The equipment inspection device according to claim 1, wherein the first position information acquisition unit detects an object to be inspected in the image based on a pre-registered sample of an object to be inspected.

3. The equipment inspection device according to claim 1, wherein the second position information is a two-dimensional position coordinate based on the line of sight of the inspector detected by the line of sight sensing device.

4. The equipment inspection device according to claim 1, wherein the first position information, the second position information, and the third position information are two-dimensional position coordinates indicating a position on an image of the external scenery.

5. The equipment inspection device according to claim 1, further comprising a importance acquisition unit that acquires the importance level specified by the inspector for the object to be inspected in the automatic inspection.

6. The equipment inspection device according to claim 1, further comprising a database creation unit that stores the third position information in a database in association with the vehicle position identification information and / or the identification information of the object to be inspected.

7. The equipment inspection device according to claim 1, further comprising a position information correction unit for correcting the third position information.

8. The equipment inspection device according to claim 1, further comprising an abnormality determination unit that determines, based on the third position information, whether or not an object to be inspected in an image captured by the image acquisition unit at the vehicle position during the automatic inspection is abnormal.

9. A camera installed on the vehicle captures an image of the external scenery of the vehicle at a certain location. The system automatically detects the object to be inspected in the aforementioned image and acquires first position information indicating the position of the object to be inspected within the aforementioned image. At the aforementioned vehicle position, second position information is acquired that indicates the position in the image of the object to be inspected, which was designated by the inspector riding in the vehicle. Based on the first and second position information, a third position information indicating the location of the object to be inspected in the automatic inspection is determined. Equipment inspection methods.

10. On the computer, A camera installed on the vehicle captures an image of the external scenery of the vehicle at a certain location. The system automatically detects the object to be inspected in the aforementioned image and acquires first position information indicating the position of the object to be inspected within the aforementioned image. At the aforementioned vehicle position, second position information is acquired that indicates the position in the image of the object to be inspected, which was designated by the inspector riding in the vehicle. Based on the first and second position information, a third position information indicating the location of the object to be inspected in the automatic inspection is determined. A program for performing equipment inspections.

Images