Monitoring device, monitoring system, monitoring method, and computer program

Abstract

To provide technology that accurately specifies an image photographing location where the image was photographed for maintenance or monitoring of transportation infrastructure including railways.SOLUTION: A monitoring device that monitors a maintenance target on a movement route of a vehicle comprises a control unit that associates a first image in which the maintenance target is photographed, a scenery image in which the surroundings of the photographing location of the first image were photographed at a photographing time of the first image, and object location information in which an object is recognized on the basis of the scenery image, and the control unit displays the first image and the scenery image associated with the first image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a monitoring device, a monitoring system, a monitoring method, and a computer program. [Background technology]

[0002] BACKGROUND ART Conventionally, a camera is attached to a railway maintenance vehicle or the like, and the camera photographs the rails while the maintenance vehicle or the like is running, and the photographed images are used for rail maintenance.

[0003] Patent Document 1 discloses that an infrared camera attached to a maintenance vehicle or the like photographs rails to generate an infrared image, identifies the location where the infrared image was taken based on the distance traveled by the maintenance vehicle or the like, and generates data that associates the identified location with the infrared image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 10-73413 Summary of the Invention [Problem to be solved by the invention]

[0005] However, to identify a location from a traveled distance as in Patent Document 1, it is necessary to register information indicating the departure position and travel route of a maintenance vehicle, etc. Furthermore, since the traveled distance is calculated using the number of wheel rotations in an odometer equipped on the maintenance vehicle, etc., it may contain large errors due to wheel wear and / or slippage.

[0006] The present disclosure aims to provide a technology that can accurately identify the shooting position of an image taken for the maintenance or monitoring of transportation infrastructure, including railways. [Means for solving the problem]

[0007] A monitoring device according to one aspect of the present disclosure is a monitoring device that monitors a maintenance object on a vehicle's travel route, and includes a control unit that associates a first image of the maintenance object, a landscape image that is taken around the shooting location of the first image at the time the first image is taken, and location information of an object that is recognized based on the landscape image, and the control unit displays the first image and the landscape image that is associated with the first image.

[0008] A monitoring device according to one aspect of the present disclosure is a monitoring device that monitors a maintenance object on a vehicle's travel path, and includes a control unit that associates a first image of the maintenance object with a landscape image taken of the area around the location where the first image was taken at the time the first image was taken, and with location information indicated by characters recognized based on a range marker image taken of a range marker located in the area around the location where the first image was taken at the time the first image was taken, and the control unit displays the first image and the landscape image associated with the first image.

[0009] A monitoring system according to one aspect of the present disclosure is a monitoring system for monitoring a maintenance object on a vehicle's travel path, and includes a first camera that photographs the maintenance object to generate a first image, a second camera that photographs the area around the shooting location of the first image at the time the first image was photographed to generate a landscape image, and a monitoring device that associates the first image of the maintenance object with the landscape image photographed around the shooting location of the first image at the time the first image was photographed, and position information of an object recognized based on the landscape image, and displays the first image and the landscape image associated with the first image.

[0010] A monitoring system according to one aspect of the present disclosure is a monitoring system for monitoring a maintenance object on a vehicle's travel path, and includes a first camera that photographs the maintenance object to generate a first image, a third camera that photographs a rangefinder located in the vicinity of the shooting location of the first image at the time the first image was taken to generate a rangefinder image, and a monitoring device that associates the first image of the maintenance object with a landscape image photographed in the vicinity of the shooting location of the first image at the time the first image was taken, and location information indicated by characters recognized based on a rangefinder image photographed of a rangefinder located in the vicinity of the shooting location of the first image at the time the first image was taken, and displays the first image and the landscape image associated with the first image.

[0011] A monitoring method according to one aspect of the present disclosure is a monitoring method for monitoring a maintenance object on a vehicle's travel path, which associates a first image of the maintenance object with a landscape image taken around the location where the first image was taken at the time the first image was taken, and location information of an object recognized based on the landscape image, and displays the first image and the landscape image associated with the first image.

[0012] A monitoring method according to one aspect of the present disclosure is a monitoring method for monitoring a maintenance object on a vehicle's travel path, which associates a first image of the maintenance object with a landscape image of the area around the location where the first image was taken at the time the first image was taken, and location information indicated by characters recognized based on a rangefinder image of a rangefinder located in the area around the location where the first image was taken at the time the first image was taken, and displays the first image and the landscape image associated with the first image.

[0013] A computer program according to one embodiment of the present disclosure is a computer program for monitoring a maintenance object on a vehicle's travel route, which causes a computer to associate a first image of the maintenance object with a landscape image taken around the location where the first image was taken at the time the first image was taken, and location information of an object recognized based on the landscape image, and to display the first image and the landscape image associated with the first image.

[0014] A computer program according to one embodiment of the present disclosure is a computer program for monitoring a maintenance object on a vehicle's travel route, and causes a computer to associate a first image of the maintenance object, a landscape image taken of the area around the location where the first image was taken at the time the first image was taken, and location information indicated by characters recognized based on a rangefinder image taken of a rangefinder located in the area around the location where the first image was taken at the time the first image was taken, and display the first image and the landscape image associated with the first image.

[0015] A monitoring device according to one aspect of the present disclosure is a monitoring device that monitors a maintenance object on a vehicle's travel path, and includes a control unit that acquires a first image of the maintenance object, location information of an object recognized based on a landscape image captured around the location where the first image was taken at the time the first image was taken, location information indicated by characters recognized based on a rangefinder image captured of a rangefinder located around the location where the first image was taken at the time the first image was taken, and positioning information that determines the location where the first image was taken, and the control unit associates the first image of the maintenance object with at least one of the location information of the object, the location information indicated by the characters, and the positioning information.

[0016] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0017] According to the present disclosure, it is possible to accurately identify the shooting position of an image taken for the maintenance or monitoring of transportation infrastructure including railways and the like. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a monitoring system according to an embodiment of the present invention. [Figure 2] A functional block diagram showing an example of the configuration of a monitoring device according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing an example of the configuration of monitoring information according to the present embodiment. [Figure 4] FIG. 10 is a diagram for explaining an example of a method for associating a monitoring image with positioning information. [Figure 5] FIG. 1 is a diagram illustrating an example of a method for estimating positioning information. [Figure 6] A flowchart showing an example of a process for identifying a shooting position of a monitoring image. [Figure 7] FIG. 10 is a diagram showing a display example of a maintenance UI (User Interface) according to the present embodiment. [Figure 8] FIG. 1 is a diagram showing the hardware configuration of a computer that implements the functional blocks of a monitoring device according to an embodiment of the present invention using a computer program. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0020] (Embodiment 1) <Monitoring system configuration> FIG. 1 is a block diagram showing an example of the configuration of a monitoring system according to this embodiment.

[0021] The monitoring system 2 is a system for monitoring railway maintenance objects. Examples of railway maintenance objects include rails, rail fastening devices, sleepers, track beds, roadbeds, overhead wires, overhead poles, insulators, balancers, traffic lights, tunnel inner walls, bridges, and slopes. Note that the "railway" according to the present disclosure is not limited to tracks on which general electric trains or steam locomotives run, but also includes tracks on which streetcars run, monorails run, and bullet trains run.

[0022] The monitoring system 2 is mounted on a railway maintenance vehicle or the like (hereinafter referred to as "vehicle") 1. The vehicle 1 may be a powered vehicle or a vehicle (bogie) towed by a powered vehicle. Note that a part of the monitoring system 2 may be installed in a location different from the vehicle 1.

[0023] The monitoring system 2 includes cameras 10A, 10B, and 10C, a video recorder 20, a global navigation satellite (hereinafter referred to as a GNSS (Global Navigation Satellite System)) receiving device 30, an RTK (Real Time Kinematic) receiving device 40, and a monitoring device 100.

[0024] The cameras 10A, 10B, and 10C are compatible with PoE (Power over Ethernet) and may be connected to the video recorder 20 via an Ethernet cable. However, the cameras 10A, 10B, and 10C may also be connected to the video recorder 20 via a wired cable other than an Ethernet cable (for example, a USB (Universal Serial Bus) cable) or a wireless system (for example, IEEE802.11 or Bluetooth). Furthermore, the connection system between the cameras 10A, 10B, and 10C and the video recorder 20 is not limited to PoE. For example, the cameras 10A, 10B, and 10C may be supplied with auxiliary power via another connection system.

[0025] The video recorder 20 is connected to the monitoring device 100 by, for example, an Ethernet cable. However, the video recorder 20 may also be connected to the monitoring device 100 by a wired cable other than an Ethernet cable (for example, a USB cable) or by a wireless method (for example, IEEE802.11 or Bluetooth).

[0026] The GNSS receiver 30 is connected to the monitoring device 100 by, for example, a USB cable. However, the GNSS receiver 30 may also be connected to the monitoring device 100 by a wired cable other than a USB cable or by a wireless method.

[0027] The RTK receiving device 40 is connected to the monitoring device 100 by, for example, a USB cable. However, the RTK receiving device 40 may also be connected to the monitoring device 100 by a wired cable other than a USB cable or by a wireless method.

[0028] The camera 10A is installed on the vehicle 1 so that it can capture images of the maintenance target. The number of cameras 10A installed is not limited to one, and may be two or more. Video data captured by the camera 10A from the moving vehicle 1 is called surveillance video. Examples of formats for surveillance video include MPEG2, MPEG4, H.264, H.265, JPEG, and uncompressed data.

[0029] Camera 10B is installed on vehicle 1 so as to capture distance markers around the track. The number of cameras 10B installed is not limited to one, and may be two or more. Video data captured by camera 10B from a moving vehicle 1 and generated is called distance marker video. The format of distance marker video may be the same as that of surveillance video. A distance marker is a sign on a railway that indicates the distance from a certain starting point. Distance markers often have letters indicating the distance, but they do not need to have letters. Note that the "letters" on a distance marker may include numbers, symbols, etc. Furthermore, distance markers are not limited to signs indicating the distance from a certain starting point, but may also indicate a distance, a distance interval, or a specific location.

[0030] The camera 10C is installed on the vehicle 1 so as to capture the scenery visible from the vehicle 1. The number of cameras 10C installed is not limited to one, and may be two or more. Video data captured by the camera 10C from the vehicle 1 while it is moving is called a scenery video. The format of the scenery video may be the same as that of the surveillance video. The camera 10C may be installed so as to capture the scenery behind, in front of, or to the side of the vehicle 1. The camera 10C may be installed inside the vehicle 1 or outside the vehicle 1 (for example, on the roof of the vehicle 1).

[0031] Cameras 10A, 10B, and 10C may be installed on vehicle 1 so that they can capture images in different directions. This allows them to capture images of different things, such as maintenance targets, distance markers, and scenery, as described above. Alternatively, if one camera can capture images of both distance markers and scenery, camera 10B may be used to capture video that includes both distance markers and scenery, without installing camera 10C.

[0032] The video recorder 20 is a device that receives and records the monitoring video, the distance marker video, and the scenery video from the cameras 10A, 10B, and 10C.

[0033] The GNSS receiver 30 periodically receives GNSS signals, which are positioning signals emitted from GNSS satellites 3, and generates GNSS information 201 (see FIG. 2) from the received GNSS signals. The GNSS information 201 includes the time of reception of the GNSS signal, the latitude and longitude of the GNSS receiver 30 at that time, and the positioning accuracy of the latitude and longitude, all associated with each other. The GNSS receiver 30 transmits the generated GNSS information 201 to the monitoring device 100. Hereinafter, the determined latitude and longitude will be referred to as positioning coordinates.

[0034] The RTK receiver 40 periodically receives RTK signals, which are positioning signals transmitted from the wireless base station 4, and generates correction information 202 (see FIG. 2) from the received RTK signals. The correction information 202 includes information for correcting the positioning coordinates indicated by the GNSS signals. The positioning coordinates corrected using the RTK signals indicate a more accurate position than the positioning coordinates before correction. The RTK receiver 40 transmits the generated correction information 202 to the monitoring device 100.

[0035] The monitoring device 100 receives GNSS information 201 and correction information 202 from the GNSS receiver 30 and the RTK receiver 40, respectively. Based on the received GNSS information 201 and correction information 202, the monitoring device 100 generates information (hereinafter referred to as "positioning information") 206 indicating the positioning coordinates of the camera 10A at the reception time included in the GNSS information 201 and correction information 202. For example, by installing the GNSS receiver 30 and the camera 10A so that they are aligned in a straight line in the vertical direction, the positioning coordinates of the camera 10A can be obtained from the GNSS information 201. Alternatively, the positional deviation between the GNSS receiver 30 and the camera 10A is registered in advance, and the positioning coordinates indicated by the GNSS information 201 can be corrected by the registered positional deviation to obtain the positioning coordinates of the camera 10A.

[0036] The monitoring device 100 sets its own internal clock based on at least one of the time information included in the GNSS information 201 and the time information received from a time server 5 (for example, an NTP (Network Time Protocol) server). This synchronizes the internal clock of the monitoring device 100 to the correct time. In addition, the monitoring device 100 synchronizes the internal clocks of the video recorder 20 and the cameras 10A, 10B, and 10C with the internal clock of the monitoring device 100. This synchronizes the internal clocks of the cameras 10A, 10B, and 10C, the video recorder 20, and the monitoring device 100 to the correct time.

[0037] The monitoring device 100 acquires and stores from the video recorder 20 each image frame (hereinafter referred to as "monitoring image") 203 (see Figure 2) that constitutes the monitoring video, each image frame (hereinafter referred to as "distance marker image") 204 (see Figure 2) that constitutes the distance marker video, and each image frame (hereinafter referred to as "landscape image") 205 (see Figure 2) that constitutes the scenery video.

[0038] The monitoring device 100 identifies the shooting location of the monitoring image 203 using positioning information 206 measured at the same time as the shooting time of the monitoring image 203. This allows the shooting location of the monitoring image 203 to be identified with high accuracy. However, when the vehicle 1 is traveling in a mountainous area, an urban area, or inside a tunnel, the GNSS signal may not be received or the positioning accuracy of the GNSS signal may be insufficient, making it impossible to use the positioning information 206. In such cases, the monitoring device 100 identifies the shooting location of the monitoring image 203 using at least one of a rangefinder image 204 and a landscape image 205 captured at the same time as the shooting time of the monitoring image 203. This allows the shooting location of the monitoring image 203 to be identified with high accuracy even when the positioning information 206 cannot be used. Note that in the description of the present disclosure, the expression "same time as the shooting time" does not refer to the exact same time, but rather to a time within a predetermined period including the shooting time. Next, the above content will be described in detail.

[0039] <Configuration of monitoring device> FIG. 2 is a functional block diagram showing an example of the configuration of the monitoring device 100 according to this embodiment.

[0040] The monitoring device 100 has an information storage unit 101, a GNSS information receiving unit 102, a correction information receiving unit 103, a clock synchronization unit 104, an image acquisition unit 105, a positioning information generating unit 106, a monitoring information generating unit 107, a shooting position identification unit 108, and a UI processing unit 109.

[0041] The information storage unit 101 may be realized by a memory 1002, a storage 1003, or a combination thereof, as shown in Fig. 8. The information storage unit 101 stores GNSS information 201, correction information 202, monitoring images 203, distance marker images 204, landscape images 205, positioning information 206, and monitoring information 207.

[0042] The GNSS information receiving unit 102, the correction information receiving unit 103, the clock synchronization unit 104, the image acquisition unit 105, the positioning information generating unit 106, the monitoring information generating unit 107, the shooting position identifying unit 108, and the UI processing unit 109 may be realized by the processor 1001 shown in Fig. 8 working in cooperation with the memory 1002, etc. Therefore, the processing described mainly with these functional blocks 102 to 109 can be interpreted as processing mainly with the processor 1001.

[0043] The GNSS information receiving unit 102 periodically receives the GNSS information 201 from the GNSS receiver 30 and stores it in the information storage unit 101 .

[0044] The correction information receiving unit 103 periodically receives correction information 202 from the RTK receiving device 40 and stores it in the information storage unit 101.

[0045] The clock synchronizer 104 sets the internal clock of the monitoring device 100 based on at least one of the time information included in the GNSS information 201 and the correction information 202 and the time information received from the time server 5. For example, when the clock synchronizer 104 obtains time information from at least one of the GNSS information 201 and the correction information 202, the clock synchronizer 104 sets the internal clock of the monitoring device 100 using the time information. When the clock synchronizer 104 does not obtain time information from either the GNSS information 201 or the correction information 202, the clock synchronizer 104 obtains time information from the time server 5 and sets the internal clock of the monitoring device 100 using the time information.

[0046] In addition, the clock synchronizer 104 synchronizes the internal clocks of the video recorder 20 and the cameras 10A, 10B, and 10C with the internal clock of the monitoring device 100. This synchronizes the internal clocks of the cameras 10A, 10B, and 10C, the video recorder 20, and the monitoring device 100 to the correct time. Therefore, the shooting times of the monitoring image 203, the distance marker image 204, and the scenery image 205 all represent the correct time.

[0047] The image acquisition unit 105 acquires a monitoring image 203 , a distance marker image 204 , and a landscape image 205 from the video recorder 20 , and stores them in the information storage unit 101 .

[0048] Based on the GNSS information 201 and the correction information 202, the positioning information generation unit 106 generates positioning information 206 indicating the positioning coordinates (latitude and longitude) of the camera 10A at the reception time contained in the GNSS information 201 and the correction information 202, and stores it in the information storage unit 101.

[0049] The monitoring information generation unit 107 generates monitoring information 207 by associating a monitoring image 203 with a distance marker image 204 and a landscape image 205 captured at the same time as the monitoring image 203, and positioning information 206 measured at the same time as the capturing of the monitoring image 203. Details of the monitoring information 207 will be described later (see FIG. 3).

[0050] The photographing position identifying unit 108 identifies the photographing position of the monitoring image 203 by using at least one of the positioning information 206, the distance marker image 204, and the landscape image 205 associated with the monitoring image 203 in the monitoring information 207. The photographing position of the monitoring image 203 may be expressed as operating kilometers indicating the distance from a predetermined starting point on a railway line.

[0051] When the positioning accuracy of the positioning information 206 associated with the monitoring image 203 in the monitoring information 207 is sufficient (for example, the positioning accuracy is equal to or greater than a predetermined threshold), the photographing position identifying unit 108 may determine the business kilometers corresponding to the positioning coordinates of the positioning information 206 as the photographing position of the monitoring image 203. For example, the photographing position identifying unit 108 converts the positioning coordinates of the positioning information 206 into business kilometers using a predetermined database that converts positioning coordinates on the orbit into business kilometers.

[0052] If the positioning information 206 is not associated with the monitoring image 203 in the monitoring information 207, or if the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is insufficient (for example, the positioning accuracy is less than a predetermined threshold), the photography location identifying unit 108 may perform the following process. That is, the photography location identifying unit 108 may identify the business kilometer of the photography location of the monitoring image 203 by using at least one of the milepost image 204 and the scenic image 205 associated with the monitoring image 203 in the monitoring information 207. The method of this identification will be described in detail later.

[0053] The UI processing unit 109 displays a maintenance UI 400 (see FIG. 7) based on the monitoring information 207 and accepts operations on the maintenance UI 400. For example, the UI processing unit 109 displays the maintenance UI 400 on a display, which is an example of the output device 1005 shown in FIG. 8. The UI processing unit 109 then accepts operations on the maintenance UI 400 via a keyboard and a mouse, which are an example of the input device 1004 shown in FIG. 8. Details of the maintenance UI 400 will be described later (see FIG. 7).

[0054] <Configuration of monitoring information> FIG. 3 is a diagram showing an example of the configuration of the monitoring information 207 according to this embodiment.

[0055] The monitoring information 207 includes items such as a monitoring image, a distance marker image, a landscape image, a shooting time, positioning information, a route name, operating kilometers, a maintenance necessity flag, a maintenance factor, a shooting means, a photographer, and a memo. In other words, the monitoring information 207 can be said to be information that indicates the correspondence between these items.

[0056] The item of the surveillance image registers the ID (Identification) of the surveillance image 203. An example of the ID of the surveillance image 203 is the file name of the surveillance image 203.

[0057] The item of distance marker image registers the ID of the distance marker image 204. An example of the ID of the distance marker image 204 is the file name of the distance marker image 204.

[0058] The scenery image item registers the ID of the scenery image 205. An example of the ID of the scenery image 205 is the file name of the scenery image 205.

[0059] The shooting time field registers the shooting time of the surveillance image 203. The shooting time field may include not only the time but also the date. The shooting time can be displayed in milliseconds or microseconds depending on the fps of the image capture, allowing the timing of each image capture to be checked in detail. This prevents images captured at different times from being mistaken for having been captured at the same time, even if the fps value is high. For example, if the shooting time is displayed only in seconds for images captured every millisecond, the order in which each image was captured cannot be ascertained. However, by displaying the shooting time in milliseconds, the order in which each image was captured can be ascertained.

[0060] The positioning information field stores the positioning coordinates measured at the same time as the monitoring image 203 was captured, and the positioning accuracy of the positioning coordinates. The positioning accuracy may be expressed as a numerical value indicating the level of accuracy. Alternatively, the positioning accuracy may be expressed in three levels: "good" indicating sufficient accuracy, "poor" indicating insufficient accuracy, and "unacceptable" indicating that positioning was not possible.

[0061] In the item of route name, the name of the route along which the monitoring image 203 was taken is registered.

[0062] In the item of business kilometers, the business kilometers of the photographing position of the monitoring image 203 is registered.

[0063] In the item "maintenance necessity flag," a flag indicating whether or not maintenance work is required for the maintenance object shown in the monitoring image 203 is registered. A maintenance worker may view the monitoring image 203 displayed on a maintenance UI 400 (described later) and determine whether or not maintenance work is required for the maintenance object in the monitoring image 203. Alternatively, the monitoring device 100 may perform image analysis of the monitoring image 203 to automatically determine whether or not maintenance work is required for the maintenance object in the monitoring image 203. A flag indicating the result of the determination may be registered in the item "maintenance necessity flag." For example, the maintenance necessity flag may be registered with "needed," which indicates that maintenance work is required, or "not required," which indicates that maintenance work is not required.

[0064] In the maintenance factor field, when the maintenance necessity flag is "required," the reason why maintenance work is necessary is registered. For example, if there is an abnormality in the equipment of the maintenance object, "equipment abnormality" may be registered in the maintenance factor field. For example, if there is a defect in the maintenance object, "defect" may be registered in the maintenance factor field.

[0065] The imaging means field registers the imaging means of the monitoring image 203. For example, if the monitoring image 203 was captured by a camera 10A mounted on a maintenance vehicle or the like, "maintenance vehicle, etc." may be registered in the imaging means field. For example, if the monitoring image 203 was captured by a camera-equipped terminal carried by a maintenance worker, "handheld terminal" may be registered in the imaging means field.

[0066] The photographer field registers the photographer of the surveillance image 203. For example, if the surveillance image 203 was taken by a maintenance worker named "Pana Hanako," "Pana Hanako" is registered in the photographer field. For example, if the surveillance image 203 was taken by a maintenance worker named "Matsushita Taro," "Matsushita Taro" is registered in the photographer field.

[0067] The memo item registers the content of a memo for the monitoring image 203. For example, when a maintenance worker inputs a memo while looking at the monitoring image 203 displayed on the maintenance UI 400 (described later), the content of the input memo is registered in the memo item.

[0068] <Associating surveillance images with positioning information> FIG. 4 is a diagram for explaining an example of a method for associating the monitoring image 203 with the positioning information 206. In FIG.

[0069] 3, the surveillance information generating unit 107 generates surveillance information 207 by associating a surveillance image 203 with positioning information 206 measured at the same time as the shooting time of the surveillance image 203. This makes it possible to identify the shooting location of the surveillance image 203 from the positioning information 206 associated with the surveillance image 203.

[0070] However, the frame rate of the surveillance video and the reception frequency of the GNSS signal may differ. For example, suppose the frame rate of the surveillance video is 30 fps and the reception frequency of the GNSS signal is 10 Hz. In this case, the frame period of the surveillance video is 1 / 30 seconds and the positioning period using the GNSS signal is 1 / 10 seconds, so it is not possible to associate the positioning information 206 with all surveillance images 203 on a one-to-one basis.

[0071] Therefore, if there is positioning information 206 that was determined at the same time as the shooting time of the monitoring image 203, the shooting position identification unit 108 associates the positioning information 206 with the monitoring image 203, and if there is no positioning information 206 that was determined at the same time as the shooting time of the monitoring image 203, the shooting position identification unit 108 estimates the positioning information 206 corresponding to the monitoring image 203 based on the positioning information 206 that was determined at another time.

[0072] 4, when monitoring images 203 are captured at times t1, t2, t3, and t4, and positioning information 206 is determined at times t1 and t4, the shooting position identifying unit 108 performs the following process. That is, the shooting position identifying unit 108 associates the positioning information 206 determined at time t1 with the monitoring image 203 captured at time t1, and associates the positioning information 206 determined at time t4 with the monitoring image 203 captured at time t4. In addition, the shooting position identifying unit 108 estimates the positioning information 206 at times t2 and t3 based on the positioning information 206 at times t1 and t4. Next, this estimation method will be described.

[0073] FIG. 5 is a diagram for explaining an example of a method for estimating positioning information.

[0074] 5, the photographing position identifying unit 108 divides the line segment L connecting the position P1 indicated by the positioning information 206 at time t1 and the position P4 indicated by the positioning information 206 at time t4 into three parts to obtain division points P2 and P3. The number of divisions may be calculated based on the value obtained by dividing the positioning period by the period of the monitoring image 203. The shape of the line segment L may follow the alignment of the rail.

[0075] The shooting position identifying unit 108 estimates the division point P2 as the positioning information 206 at time t2, and estimates the division point P3 as the positioning information 206 at time t3. Then, in the monitoring information 207, the shooting position identifying unit 108 associates the monitoring image 203 at time t2 with the estimated positioning information 206 at time t2, and associates the monitoring image 203 at time t3 with the estimated positioning information 206 at time t3.

[0076] This allows the positioning information 206 to be associated with each monitoring image 203 even if the frame period of the monitoring video and the positioning period using the GNSS signal are different.

[0077] <When positioning accuracy is insufficient> While the vehicle 1 is traveling in a tunnel, underground, etc., it is not possible to receive GNSS signals, and therefore the positioning information generating unit 106 is unable to generate the positioning information 206 for that period. Therefore, in the monitoring information 207, the positioning information 206 may not be associated with the monitoring image 203 captured while the vehicle 1 is traveling in a tunnel, underground, etc. In this case, the capturing position of the monitoring image 203 cannot be identified from the positioning information 206.

[0078] Furthermore, while the vehicle 1 is traveling in an urban area, a mountainous area, or the like, the GNSS signal may be received but the RTK signal may not be received, or the reception accuracy of the GNSS signal may be insufficient, resulting in insufficient positioning accuracy of the generated monitoring information 207. Therefore, in the monitoring information 207, the positioning information 206 associated with the monitoring image 203 captured while the vehicle 1 is traveling in an urban area, a mountainous area, or the like may have insufficient positioning accuracy. In this case, the shooting position of the monitoring image 203 identified from the positioning information 206 may be significantly different from the actual shooting position.

[0079] Therefore, for a monitoring image 203 that is not associated with positioning information 206 in the monitoring information 207 or is associated with positioning information 206 with insufficient positioning accuracy (below a predetermined threshold), the shooting location identification unit 108 identifies the shooting location of the monitoring image 203 using at least one of a milepost image 204 and a landscape image 205 that were taken at the same time as the monitoring image 203 was taken.

[0080] For example, the photographing location identifying unit 108 identifies the photographing location from the distance marker image 204 as follows: That is, the photographing location identifying unit 108 performs character recognition on the distance marker image 204, which was photographed at the same time as the monitoring image 203 was photographed, and reads the characters on the distance marker. The photographing location identifying unit 108 identifies the commercial kilometer indicated by the distance marker from the characters that were read. In the monitoring information 207, the photographing location identifying unit 108 associates the identified commercial kilometer with the monitoring image 203, which was photographed at the same time as the distance marker image 204.

[0081] For example, the shooting position identifying unit 108 identifies the shooting position from the scenery image 205 as follows. That is, the shooting position identifying unit 108 performs object recognition on the scenery image 205 captured at the same time as the monitoring image 203, and recognizes the object. The shooting position identifying unit 108 queries a predetermined database that has a correspondence relationship between recognized objects and the locations where the objects are located, to find the location of the recognized object, and obtains the location of the object. In the monitoring information 207, the shooting position identifying unit 108 associates the business kilometer of the location of the object with the monitoring image 203 captured at the same time as the scenery image 205. Examples of objects that are captured in a scenery image and whose locations can be identified include tunnels, bridges, train stations, stores, signs, road signs, bridges, and landmarks.

[0082] For example, the photographing position identifying unit 108, in cooperation with the UI processing unit 109, identifies the photographing position from the scenic image 205 as follows: That is, the UI processing unit 109 displays the scenic image 205 that was taken at the same time as the photographing time of the monitoring image 203, and accepts input of the position indicated by the scenic image 205 from the maintenance worker. The maintenance worker looks at the displayed scenic image 205 and inputs the position where the scenic image 205 was taken. The photographing position identifying unit 108 associates the business kilometer of the input position with the monitoring image 203 that was taken at the same time as the photographing time of the scenic image 205 in the monitoring information 207.

[0083] <Process for identifying the shooting position of surveillance images> FIG. 6 is a flowchart showing an example of a process for identifying the photographing position of the monitoring image 203.

[0084] The photographing position specifying unit 108 selects one monitoring image 203 from among the multiple monitoring images 203 included in the monitoring information 207 (S101).

[0085] The photographing position specifying unit 108 acquires the positioning information 206 associated with the monitoring image 203 selected in S101 in the monitoring information 207 (that is, measured at the same time as the photographing time of the monitoring image 203) (S102).

[0086] The monitoring information generating unit 107 determines whether the positioning accuracy included in the positioning information 206 acquired in S102 is equal to or greater than a predetermined threshold (S103).

[0087] If the positioning accuracy is equal to or greater than the threshold (S103: YES), the photographing position specifying unit 108 specifies the business kilometers in the positioning coordinates of the positioning information 206 acquired in S102 (S104). Then, the process of S112 is executed.

[0088] If the positioning accuracy is less than the threshold value, or if the positioning information 206 is not associated with the monitoring image 203 selected in S101 (S103: NO), the photographing position identifying unit 108 performs the following process: The photographing position identifying unit 108 acquires the distance marker image 204 that is associated with the monitoring image 203 selected in S101 in the monitoring information 207 (i.e., that was photographed at the same time as the monitoring image 203 was photographed) (S105).

[0089] The photographing position specifying unit 108 performs character recognition of the distance marker on the distance marker image 204 selected in S105 (S106). The photographing position specifying unit 108 determines whether or not the character recognition of the distance marker in S106 was successful (S107).

[0090] If the character recognition of the distance marker is successful (S107: YES), the photographing position specifying unit 108 specifies the business kilometer indicated by the recognized character of the distance marker (S108), and then the process of S112 is executed.

[0091] If character recognition of the distance marker fails (S107: NO), the photographing position identification unit 108 performs the following process in cooperation with the UI processing unit 109. That is, the UI processing unit 109 displays, as the maintenance UI 400, the scenery image 205 that is associated with the monitoring image 203 selected in S101 in the monitoring information 207 (i.e., that was photographed at the same time as the monitoring image 203), and accepts input of the position indicated by the scenery image 205 (S109).

[0092] The maintenance worker inputs the location information indicated by the displayed scenic image 205 (S110). The photography location identification unit 108 identifies the business kilometers of the location input in S110 (S111). Then, the process of S112 is executed.

[0093] The photographing location identification unit 108 associates the commercial kilometers identified in S104, S108, or S111 with the monitoring image 203 selected in S101 in the monitoring information 207 (S112). After executing the above-mentioned process for each monitoring image 203 included in the monitoring information 207, the monitoring device 100 terminates this process.

[0094] According to the above processing, when the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is sufficient, the photographing location identifying unit 108 identifies the business kilometers of the photographing location of the monitoring image 203 using the positioning information 206. Furthermore, when the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is insufficient or when no positioning information 206 is associated with the monitoring image 203, the photographing location identifying unit 108 identifies the business kilometers of the photographing location of the monitoring image 203 using the distance marker image 204 or the scenery image 205 associated with the monitoring image 203. In this way, the business kilometers of the photographing location of the monitoring image 203 can be identified regardless of the location at which the monitoring image 203 was photographed.

[0095] <Maintenance UI display> FIG. 7 is a diagram showing an example of a display of the maintenance UI 400 according to this embodiment.

[0096] 7 on a display, which is an example of the output device 1005. The maintenance UI 400 includes a monitoring image area 401, a distance marker image area 402, a landscape image area 403, an information area 404, a map area 405, a previous button 406, a next button 407, a maintenance required button 408, and a memo button 409.

[0097] In the monitoring image area 401, one of the monitoring images 203 included in the monitoring information 207 is displayed.

[0098] In the distance marker image area 402, a distance marker image 204 that is associated with the monitoring image 203 displayed in the monitoring image area 401 in the monitoring information 207 is displayed.

[0099] In the scenery image area 403, a scenery image 205 that is associated with the monitoring image 203 displayed in the monitoring image area 401 in the monitoring information 207 is displayed.

[0100] The information area 404 displays information about the surveillance image 203 displayed in the surveillance image area 401. For example, the file name of the surveillance image 203, the date it was taken, the time it was taken, the photographer, notes, business kilometers, and the latitude and longitude indicating the location where it was taken may be displayed. The UI processing unit 109 may display this information in the information area 404 by referring to the surveillance information 207.

[0101] The map area 405 displays a map of the area surrounding the photographing position of the monitoring image 203 displayed in the monitoring image area 401, and a pin 410 indicating the photographing position.

[0102] 6, the UI for receiving input of the position indicated by the scenery image 205 may be the same as the maintenance UI 400. In this case, the maintenance worker can input the location information indicated by the scenery image 205 by looking at the scenery image 205 displayed in the scenery image area 403 and dropping a pin 410 at the position on the map indicated by the scenery.

[0103] When the previous button 406 is pressed, the UI processing unit 109 displays the monitoring image 203, the distance marker image 204, and the scenery image 205 associated with the previous shooting time in the monitoring information 207 in the monitoring image area 401, the distance marker image area 402, and the scenery image area 403, respectively. The UI processing unit 109 also displays in the map area 405 a map of the surrounding area of ​​the shooting location of the monitoring image 203 associated with the previous shooting time, and a pin 410 indicating the shooting location.

[0104] When the next button 407 is pressed, the UI processing unit 109 displays the monitoring image 203, the distance marker image 204, and the scenery image 205 associated with the next shooting time in the monitoring information 207 in the monitoring image area 401, the distance marker image area 402, and the scenery image area 403, respectively. The UI processing unit 109 also displays in the map area 405 a map of the surrounding area of ​​the shooting location of the monitoring image 203 associated with the next shooting time, and a pin 410 indicating the shooting location.

[0105] When the maintenance required button 408 is pressed, the UI processing unit 109 changes the maintenance required flag associated with the currently displayed monitoring image 203 in the monitoring information 207 to "required."

[0106] When the memo button 409 is pressed, the UI processing unit 109 displays a memo input UI (not shown). In addition, the UI processing unit 109 registers the content input into the memo input UI in the memo item associated with the currently displayed monitoring image 203 in the monitoring information 207.

[0107] For example, the maintenance worker presses the next button 407 to switch the displayed monitoring image 203 and check whether there is a problem with the maintenance object. If there is a problem with the maintenance object, the maintenance worker presses the maintenance required button 408. If the maintenance worker wants to add a comment to the maintenance object, the maintenance worker presses the memo button 409 and inputs the memo.

[0108] The UI processing unit 109 may provide a search function. For example, the UI processing unit 109 may search the monitoring information 207 for monitoring images 203 that are not associated with operating kilometers, and display them on the maintenance UI 400. This allows the maintenance worker to efficiently input the location indicated by the scenic image 205. Furthermore, for example, the UI processing unit 109 may search the monitoring information 207 for monitoring images 203 that are associated with a maintenance necessity flag of "necessary," and display them on the maintenance UI 400. This allows the maintenance worker to efficiently check the monitoring images 203 of maintenance objects that require maintenance.

[0109] <Modifications and Supplements> The information storage unit 101 may store the monitoring image 203, the rangefinder image 204, and the scenery image 205 taken by the cameras 10A, 10B, and 10C provided on the vehicle 1, as well as the monitoring image 203, the rangefinder image 204, and the scenery image 205 taken by a camera carried by a maintenance worker. In this case, the monitoring information generation unit 107 may also add the monitoring image 203, the rangefinder image 204, and the scenery image 205 taken by the maintenance worker to the monitoring information 207. The camera carried by the maintenance worker may be a smartphone or a tablet terminal equipped with a camera. Alternatively, the camera carried by the maintenance worker may be attached to a helmet worn by the maintenance worker.

[0110] Although the RTK signal is also used for positioning in the above description, positioning may be performed using only the GNSS signal without using the RTK signal. Alternatively, the monitoring device 100 may identify the shooting position of the monitoring image 203 using the rangefinder image 204 or the landscape image 205 without using either the GNSS signal or the RTK signal.

[0111] Although the above description has been given in the case where the maintenance target is a railway, the maintenance target is not limited to railways. The above-described content can be applied to places where vehicles travel, such as general roads or expressways. For example, by applying the above-described content to expressways, it is possible to accurately identify the shooting positions of images taken for maintenance of tunnels, road shoulders, or collapse prevention walls on expressways.

[0112] Although the above describes cases where the position is determined using GNSS or RTK, the position can also be determined based on information about the route name and operating kilometers. With this configuration, the position can be accurately determined from the route name and operating kilometers recognized from distance markers, etc. Note that the route name may be registered in advance, and the operating kilometers may be recognized from the distance markers.

[0113] <Hardware configuration> The functional blocks of the monitoring device 100 described above may be realized by a computer program. Fig. 8 is a diagram showing the hardware configuration of a computer that realizes the functional blocks of the monitoring device 100 according to this embodiment by a computer program. Note that the term "computer" may be read as an information processing device.

[0114] The computer 1000 includes a processor 1001, a memory 1002, a storage 1003, an input device 1004, an output device 1005, a communication device 1006, a GPU (Graphics Processing Unit) 1007, a reading device 1008, and a bus 1009.

[0115] Each of the devices 1001 to 1008 is connected to a bus 1009, and can transmit and receive data bidirectionally via the bus 1009.

[0116] The processor 1001 is a device that executes a computer program stored in the memory 1002 and realizes the above-mentioned functional blocks. The processor 1001 may be interpreted as a control unit. Examples of the processor 1001 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a controller, an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field-Programmable Gate Array).

[0117] The memory 1002 is a device that stores computer programs and data used by the computer 1000. The memory 1002 may be interpreted as a storage unit. The memory 1002 may include a real-only memory (ROM) and a random access memory (RAM). Examples of RAM include a dynamic random access memory (DRAM), a magnetoresistive random access memory (MRAM), and a ferroelectric random access memory (FeRAM).

[0118] The storage 1003 is configured as a nonvolatile storage medium, and is a device that stores computer programs and data handled by the computer 1000. Examples of the storage 1003 include a hard disk drive (HDD) and a solid state drive (SSD).

[0119] The input device 1004 is a device that accepts data to be input to the processor 1001. Examples of the input device 1004 include a keyboard, a mouse, a touchpad, and a microphone.

[0120] The output device 1005 is a device that outputs data generated by the processor 1001. Examples of the output device 1005 include a display and a speaker.

[0121] The communication device 1006 is a device that transmits and receives data to and from other devices, such as a server, via a communication network. The communication device 1006 may include a transmitter that transmits data and a receiver that receives data. The communication device 1006 may support either wired communication or wireless communication. An example of wired communication is Ethernet (registered trademark). Examples of wireless communication are IEEE802.11, Bluetooth, LTE, 4G, and 5G.

[0122] The GPU 1007 is a device that performs high-speed image rendering processing. The GPU 1007 may be used for processing related to AI (Artificial Intelligence) (for example, deep learning) or processing related to character recognition of distance markers, etc.

[0123] The reading device 1008 is a device that reads data from an external recording medium such as a DVD-ROM (Digital Versatile Disk Read Only Memory) or a USB memory.

[0124] The functional blocks of the monitoring device 100 may be realized as an LSI, which is an integrated circuit. These functional blocks may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Here, we refer to LSI, but depending on the level of integration, it may also be called an IC, system LSI, super LSI, or ultra LSI. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or a different derivative technology, it is natural that the functional blocks may be integrated using that technology.

[0125] Although video recorder 20 in this embodiment receives surveillance video, rangefinder video, and landscape video from cameras 10A, 10B, and 10C, video recorder 20 may receive surveillance images, rangefinder images, and landscape images from cameras 10A, 10B, and 10C, respectively. This configuration eliminates the need for processing to extract images from videos.

[0126] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the above-described embodiments may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0127] The techniques disclosed herein are useful for monitoring and maintaining transportation infrastructure in general, including railways. [Explanation of symbols]

[0128] 1 vehicle 2. Surveillance System 3 GNSS satellites 4. Wireless base stations 5 Time Server 10A, 10B, 10C Camera 20 Video Recorder 30 GNSS receiver 40 RTK receiver 100 Monitoring equipment 101 Information storage unit 102 GNSS information receiver 103 Correction information receiving unit 104 Clock Synchronization Unit 105 Image acquisition unit 106 Positioning information generation unit 107 Monitoring information generation unit 108 Shooting position identification unit 109 UI processing section 201 GNSS Information 202 Correction Information 203 Surveillance Images 204 distance marker images 205 landscape images 206 Positioning Information 207 Monitoring Information 400 Maintenance UI 401 Surveillance Image Area 402 Distance marker image area 403 Landscape Image Region 404 Information area 405 Map Area 406 Front button 407 Next button 408 Maintenance Required Button 409 Memo Button 410 pins 1000 computers 1001 processor 1002 memory 1003 Storage 1004 Input Device 1005 Output Device 1006 Communication equipment 1007 GPU 1008 Reading device 1009 Bus

Claims

1. A monitoring device for monitoring a maintenance object on a vehicle's travel route, a control unit that associates a first image of the maintenance object, a scenery image captured around the shooting position of the first image at the shooting time of the first image, and position information of an object recognized based on the scenery image, the control unit displays the first image and the scenery image associated with the first image. monitoring equipment.

2. A monitoring device for monitoring a maintenance object on a vehicle's travel route, a control unit that associates a first image of the maintenance object with a landscape image of the vicinity of the photographed position of the first image at the photographed time of the first image, and with position information indicated by characters recognized based on a distance marker image of a distance marker existing in the vicinity of the photographed position of the first image at the photographed time of the first image, the control unit displays the first image and the scenery image associated with the first image. monitoring equipment.

3. the control unit associates the first image with position information of the object, a distance marker image obtained by photographing a distance marker present around the photographing position of the first image at the photographing time of the first image, and position information indicated by characters recognized based on the distance marker image. The monitoring device of claim 1 .

4. The control unit displaying the first image, the scenery image associated with the first image, and the distance marker image associated with the first image; The monitoring device according to claim 3.

5. A monitoring system for monitoring a maintenance object on a vehicle's travel route, a first camera that captures an image of the maintenance object to generate a first image; a second camera that captures an image of the surroundings of the capture position of the first image at the capture time of the first image to generate a landscape image; a monitoring device that associates the first image of the maintenance object with a scenery image captured around the shooting position of the first image at the shooting time of the first image, and position information of an object recognized based on the scenery image, and displays the first image and the scenery image associated with the first image, Surveillance system.

6. A monitoring system for monitoring a maintenance object on a vehicle's travel route, a first camera that captures an image of the maintenance object to generate a first image; a third camera that captures an image of a rangefinder present in the vicinity of the capture position of the first image at the capture time of the first image, and generates a rangefinder image; a monitoring device that associates the first image of the maintenance object, a scenery image of the vicinity of the photographed position of the first image at the photographed time of the first image, and position information indicated by characters recognized based on a distance marker image of a distance marker existing in the vicinity of the photographed position of the first image at the photographed time of the first image, and displays the first image and the scenery image associated with the first image. Surveillance system.

7. A monitoring method for monitoring a maintenance object on a vehicle's travel route, comprising: a first image of the maintenance object, a scenery image of the periphery of the photographing position of the first image at the photographing time of the first image, and position information of an object recognized based on the scenery image; displaying the first image and the scenery image associated with the first image; Monitoring method.

8. A monitoring method for monitoring a maintenance object on a vehicle's travel route, comprising: a first image of the maintenance object, a scenery image of the vicinity of the photographing position of the first image at the photographing time of the first image, and position information indicated by characters recognized based on a distance marker image of a distance marker existing in the vicinity of the photographing position of the first image at the photographing time of the first image; displaying the first image and the scenery image associated with the first image; Monitoring method.

9. A computer program for monitoring maintenance objects on a vehicle's travel route, comprising: a first image of the maintenance object, a scenery image of the periphery of the photographing position of the first image at the photographing time of the first image, and position information of an object recognized based on the scenery image; displaying the first image and the scenery image associated with the first image; Computer program.

10. A computer program for monitoring maintenance objects on a vehicle's travel route, comprising: a first image of the maintenance object, a scenery image of the vicinity of the photographing position of the first image at the photographing time of the first image, and position information indicated by characters recognized based on a distance marker image of a distance marker existing in the vicinity of the photographing position of the first image at the photographing time of the first image; displaying the first image and the scenery image associated with the first image; Computer program.

11. A monitoring device for monitoring a maintenance object on a vehicle's travel route, a control unit that acquires a first image of the maintenance object, position information of an object recognized based on a landscape image captured around the shooting position of the first image at the shooting time of the first image, position information indicated by characters recognized based on a range marker image captured of a range marker existing around the shooting position of the first image at the shooting time of the first image, and position measurement information that measures the shooting position of the first image, The control unit a first image of the maintenance object is associated with at least one of the position information of the object, the position information indicated by the characters, and the positioning information; monitoring equipment.

12. The control unit A frame cycle for capturing the first image is different from a positioning cycle for measuring the position of the first image, If the positioning information measured at the time the first image was taken exists, the positioning information is associated with the first image, and if the positioning information measured at the time the first image was taken does not exist, the positioning information corresponding to the first image is estimated based on the positioning information measured at another time. The monitoring device of claim 11.

Images