Image processing device, inspection system, method, and program

Abstract

To provide an image processing device with which it is possible to acquire data for inspecting structures without using a dedicated vehicle for inspection.SOLUTION: An image processing device comprises: generation means for generating an inspection object image including a structure to be inspected and having a preliminarily attached marker by using a captured image in which the structure is captured by an imaging device mounted to a business vehicle provided for business; identification means for identifying the marker included in the inspection object image generated by the generation means; superimposition means for superimposing, on the basis of the position of the marker identified by the identification means, a first inspection object image generated using a first captured image and a second inspection object image generated using a second captured image different from the first captured image; and output means for outputting the inspection object image having been superimposed by the superimposition means.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an inspection system, a method, and a program. [Background technology]

[0002] Patent Document 1 describes a monitoring system for inspecting structures such as tunnels. The monitoring system described in Patent Document 1 periodically and continuously photographs a predetermined position on the structure, and when a marker is detected in the photographed image, controls the start or stop of measurement of the structure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-170051 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the monitoring system described in Patent Document 1 is configured with a photographing device and a marker detection device for detecting the markers, and a measuring device for measuring the structure, and a dedicated inspection vehicle for carrying these devices must be prepared in addition to the vehicles used for business.

[0005] One of the objects of the present disclosure is to provide an image processing device, an inspection system, a method, and a program that can acquire data for inspecting structures without using a vehicle dedicated to inspection. [Means for solving the problem]

[0006] The image processing device according to the present disclosure includes a generation means for generating an inspection target image including a structure to be inspected, on which signs have been installed in advance, using an image captured by an imaging device mounted on a commercial vehicle used for business purposes; an identification means for identifying the signs included in the inspection target image generated by the generation means; a superposition means for superimposing a first inspection target image generated using the first image based on the position of the sign identified by the identification means, and a second inspection target image generated using a second image different from the first image; and an output means for outputting the inspection target image superimposed by the superposition means.

[0007] The inspection system according to the present disclosure comprises an imaging device mounted on a commercial vehicle used for business operations, and an image processing device, wherein the imaging device captures an image of a structure to be inspected that has signs installed in advance, and the image processing device comprises a generation means for generating an image of the inspection target including the structure using the image captured by the imaging device, an identification means for identifying signs included in the image of the inspection target generated by the generation means, a superposition means for superimposing a first image of the inspection target generated using the first image captured based on the position of the sign identified by the identification means, and a second image of the inspection target generated using a second image captured different from the first image captured, and an output means for outputting the image of the inspection target superimposed by the superposition means.

[0008] In the image processing method according to the present disclosure, a computer generates an inspection target image including a structure to be inspected, on which signs have been installed in advance, using an image captured by an imaging device mounted on a commercial vehicle used for business operations, identifies the signs included in the generated inspection target image, and based on the position of the identified signs, superimposes a first inspection target image generated using the first image and a second inspection target image generated using a second image different from the first image, and outputs the superimposed inspection target image.

[0009] The image processing program according to the present disclosure causes a computer to execute a generation process for generating an inspection target image including a structure on which signs have been installed in advance, using an image captured by an imaging device mounted on a commercial vehicle used for business operations; an identification process for identifying signs included in the generated inspection target image; a superposition process for superimposing a first inspection target image generated using the first image based on the position of the identified sign with a second inspection target image generated using a second image different from the first image; and an output process for outputting the superimposed inspection target image. [Effects of the Invention]

[0010] According to the present disclosure, data for inspecting a structure can be obtained without using a vehicle dedicated to inspection. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an inspection system. [Figure 2] FIG. 10 is an explanatory diagram illustrating an example of an imaging direction for each train formation. [Figure 3] FIG. 2 is a block diagram illustrating a functional configuration of the inspection system. [Figure 4] FIG. 10 is an explanatory diagram illustrating a method for identifying a sign included in an inspection target image. [Figure 5] 10 is a flowchart illustrating the operation of the inspection system. [Figure 6] 10 is a flowchart illustrating the operation of the inspection system. [Figure 7] 10 is a flowchart illustrating the operation of the inspection system. [Figure 8] FIG. 1 is a block diagram illustrating a configuration of a computer. [Figure 9] FIG. 1 is a block diagram illustrating the main parts of an inspection system. [Figure 10] FIG. 1 is a block diagram illustrating a main part of an image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0013] An overview of the inspection system will be described. FIG. 1 is an explanatory diagram illustrating an example of the overview of the inspection system. Note that FIG. 1 is an explanatory diagram for facilitating understanding of the overview of the inspection system. Therefore, the configuration and operation of the inspection system are not limited to those shown in FIG. 1. Furthermore, the arrows in FIG. 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectionality. This also applies to other figures.

[0014] FIG. 1 shows an example of a train to which the inspection system 100 is applied. The train shown in FIG. 1 is made up of a commercial vehicle 200. The commercial vehicle 200 is a railway vehicle that runs on a railroad track or a track equivalent to a railroad track. The commercial vehicle 200 is a railway vehicle used for the operation of a railway business. Hereinafter, a railway vehicle used for the operation of a railway business will be referred to as a commercial vehicle. A commercial vehicle is, for example, a railway vehicle that transports passengers or cargo. Railway vehicles such as inspection vehicles that are used exclusively for inspecting structures such as railroad tracks and tunnels are not included in the commercial vehicles.

[0015] The commercial vehicle 200 shown in Fig. 1 is equipped with a camera (e.g., corresponding to the imaging device 201 described later), a video storage device (e.g., corresponding to the video data storage unit 202 described later), and a wireless transmitter (e.g., corresponding to the video data transmission unit 203 described later). The configuration of the commercial vehicle 200 is not limited to the example configuration shown in Fig. 1. For example, the commercial vehicle 200 may be configured to be equipped with a single device that includes all of the functions of the camera, the video storage device, and the wireless transmitter.

[0016] The upper part of FIG. 1 illustrates an example of a train traveling through a tunnel. In the example shown in the upper part of FIG. 1, a sign 210 has been installed in advance in the tunnel to be inspected. Specifically, signs 210a, 210b, 210c, 210d, 210e, and 210f have been installed in advance on the ceiling of the tunnel to be inspected. The signs 210a to 210f are signs on which characteristic patterns associated with location information indicating their installation locations are printed. The characteristic patterns of the signs 210a to 210f are different from each other. For example, the sign 210a is a sign on which a characteristic pattern associated with location information indicating the installation location of the sign 210a is printed. Therefore, the inspection system can identify the installation location of the sign 210 by identifying the characteristic pattern of the sign 210. The characteristic pattern is, for example, composed of two colors, white and black. This configuration makes it easier to recognize characteristic patterns included in an image.

[0017] In the example in the upper part of FIG. 1, the camera mounted on commercial vehicle 200 is installed so as to capture an image above the vehicle (i.e., the tunnel ceiling). Therefore, the camera mounted on commercial vehicle 200 captures an image of the tunnel ceiling while the train is traveling through the tunnel. Note that, because commercial vehicle 200 travels on rails, the distance between the camera mounted on commercial vehicle 200 and the tunnel wall (including the tunnel ceiling) is always constant. The video storage device mounted on commercial vehicle 200 stores the video data captured by the camera.

[0018] The lower part of Fig. 1 illustrates an example of a train returning to a railroad depot. In the example shown in the lower part of Fig. 1, a wireless receiver (e.g., corresponding to a video data receiving unit 301 described later), an image capturing position measuring device (e.g., corresponding to a part of the function of an image processing device 300 described later), and an abnormality determination device (e.g., corresponding to an abnormality determination unit 307 described later) are installed in the railroad depot. Note that the configuration of the railroad depot is not limited to the example configuration shown in Fig. 1. For example, the railroad depot may be installed with a single device that includes all of the functions of the wireless receiver, the image capturing position measuring device, and the abnormality determination device.

[0019] When the train returns to the depot, the wireless transmitter mounted on the commercial vehicle 200 transmits the video data stored in the video storage device to the wireless receiver at the depot. The imaging position measurement device at the depot performs image processing to obtain an image of the inside of the tunnel based on the video data received by the wireless receiver. Then, the abnormality determination device at the depot performs abnormality determination processing to detect an abnormality in the tunnel based on the image of the inside of the tunnel.

[0020] In the example shown in FIG. 1, the camera mounted on the commercial vehicle 200 is installed so as to capture an image in the direction above the vehicle (i.e., the tunnel ceiling). However, the camera mounted on the commercial vehicle 200 can be installed so as to capture an image in a direction other than the direction above the vehicle. For example, the camera may be installed so as to capture an image in a different direction for each train formation. An example of camera installation will be described below.

[0021] FIG. 2 is an explanatory diagram illustrating the imaging direction for each train formation. In this embodiment, a train formation is a group of commercial vehicles used to operate a train. Six train formations are shown in FIG. 2. Each train formation is equipped with a set of cameras, video recording devices, and wireless transmitters. Note that FIG. 2 is an explanatory diagram for making it easy to understand the imaging direction. Therefore, the installation mode of the cameras (for example, the imaging direction of the cameras and the number of cameras installed) is not limited to that shown in FIG. 2.

[0022] Of the six train formations, the first and fourth commercial vehicles 200 are equipped with cameras that capture images above the vehicle (e.g., the tunnel ceiling). The second and fifth commercial vehicles 200 are equipped with cameras that capture images to the right of the vehicle (e.g., the tunnel wall located to the right of the vehicle). The third and sixth commercial vehicles 200 are equipped with cameras that capture images to the left of the vehicle (e.g., the tunnel wall located to the left of the vehicle). This configuration makes it possible to capture images in multiple directions without installing multiple cameras on each train formation. As a result, the cost of the inspection system can be reduced.

[0023] 2, when a camera is installed so as to capture images to the right and left of the vehicle in addition to the direction above the vehicle, signs 210 are installed in advance at positions corresponding to the image capturing directions. For example, one or more signs 210 are installed in advance on each of the side walls corresponding to the right and left of the vehicle among the inner walls of a tunnel to be inspected. Furthermore, each of these signs 210 is printed with a characteristic pattern associated with position information indicating the installation position.

[0024] A camera mounted on the commercial vehicle 200 of each train formation captures images in the imaging direction in which it is installed. A wireless transmitter mounted on the commercial vehicle 200 of each train formation transmits the image data stored in the image storage device to a wireless receiver at the depot. An imaging position measurement device at the depot performs image processing to obtain images of the inside of the tunnel to be inspected based on the image data transmitted from each train formation. In particular, the imaging position measurement device can obtain high-precision images by overlaying images captured in the same imaging direction during image processing.

[0025] In the example shown in FIG. 2, one camera is mounted on each train formation. However, multiple cameras may be mounted on one train formation. That is, the multiple cameras mounted on one train formation may be configured to capture images in the same or different imaging directions. In this case, the multiple cameras may be mounted on the same commercial vehicle 200 or on different commercial vehicles 200.

[0026] 1 and 2, the inspection target of the inspection system is the interior walls, including the ceiling and side walls, of the tunnel. However, the inspection target of the inspection system is not limited to the interior walls of the tunnel. The inspection system can inspect various structures that can be photographed by a camera mounted on the commercial vehicle 200.

[0027] Next, the functional configuration of the inspection system will be described with reference to Fig. 3. Fig. 3 is a block diagram illustrating the functional configuration of the inspection system.

[0028] 3, the inspection system 100 includes a business vehicle 200 and an image processing device 300. The business vehicle 200 shown in FIG. 3 corresponds to the business vehicle 200 shown in FIG.

[0029] The commercial vehicle 200 is a railway vehicle used for the operation of a railway business. The commercial vehicle 200 is equipped with an imaging device 201, a video data storage unit 202, and a video data transmission unit 203.

[0030] The imaging device 201 captures an image of a structure to be inspected, on which a sign 210 has been installed in advance. For example, the imaging device 201 is realized by a camera or the like. The imaging device 201 is installed so as to capture an image in one of a plurality of predetermined imaging directions. For example, the imaging device 201 is installed so as to capture an image in a predetermined imaging direction, such as above the vehicle, to the right of the vehicle, or to the left of the vehicle.

[0031] The video data storage unit 202 stores video data (for example, moving images) captured by the imaging device 201.

[0032] The video data transmission unit 203 transmits the video data stored in the video data storage unit 202 to the image processing device 300 via a wired network or a wireless network (information communication network).

[0033] An image processing device 300 shown in FIG. 3 is a device that includes all of the functions of the wireless receiver, the imaging position measuring device, and the abnormality determining device shown in FIG.

[0034] The image processing device 300 includes a video data receiving unit 301, an image generating unit 302, a similarity calculating unit 303, a position identifying unit 304, an imaging direction classifying unit 305, a superimposing unit 306, an abnormality determining unit 307, a data output unit 308, and a sign storage unit 309.

[0035] The video data receiving unit 301 receives video data from the video data transmitting unit 203 of the commercial vehicle 200 via a wired network or a wireless network (information communication network).

[0036] The image generating unit 302 generates an image including the structure to be inspected using the video data received by the video data receiving unit 301 (i.e., the captured images of each frame captured in time series). Hereinafter, the image including the structure to be inspected generated by the image generating unit 302 is also referred to as an inspection target image. The inspection target image may be composed of a single image or multiple images.

[0037] The image generation unit 302 can generate an inspection object image using various methods. An example of a method for generating an inspection object image is shown below. However, the methods for generating an inspection object image that can be applied to the image generation unit 302 are not limited to this.

[0038] For example, the image generation unit 302 acquires captured images of each frame captured in time series as video data. The frame rate (fps) of the video data captured by the imaging device 201 is defined as α. An imaging range value indicating the imaging range of the imaging device 201 is defined as L. Furthermore, the traveling speed (m / s) of the commercial vehicle 200 on which the imaging device 201 is mounted is defined as k. The traveling speed k of the commercial vehicle 200 is a predetermined value used to generate the inspection target image. The traveling speed k of the commercial vehicle 200 is predetermined based on, for example, the average speed of the commercial vehicle 200. Note that the traveling speed k of the commercial vehicle 200 may be a value measured by the commercial vehicle 200 each time it travels and transmitted to the image processing device 300.

[0039] The image generation unit 302 combines the captured image of the xth frame of the video data so that the captured image of the x+(L×α) / kth frame is adjacent to the captured image of the xth frame. Furthermore, the image generation unit 302 combines the captured image of the x+(L×α) / kth frame so that the captured image of the x+(L×α) / k+(L×α) / kth frame is adjacent to the captured image of the xth frame. By repeating this process, the image generation unit 302 can generate an inspection target image in which the structures to be inspected are included without overlapping.

[0040] As described above, in the inspection system 100, one or more signs 210 are installed in advance on a structure to be inspected. The signs 210 also include characteristic patterns. Specifically, the signs 210 are printed with characteristic patterns associated with location information indicating the installation location. The sign storage unit 309 of the image processing device 300 stores, for each sign 210 installed on a structure to be inspected, the characteristic pattern of the sign 210 and the location information indicating the installation location of the sign 210 in association with each other.

[0041] The similarity calculation unit 303 calculates the similarity between the inspection target image and the characteristic pattern of the sign 210 for each predetermined shift width. For example, the similarity calculation unit 303 calculates the similarity by cross-correlation processing.

[0042] The position specifying unit 304 specifies the sign 210 included in the image to be inspected. For example, the position specifying unit 304 specifies that the sign 210 corresponding to the characteristic pattern is present at a position in the image to be inspected that has the highest similarity to the characteristic pattern.

[0043] Furthermore, the position specifying unit 304 specifies the image capturing time of the sign 210 based on the specified position of the sign 210 in the inspection target image. Furthermore, the position specifying unit 304 specifies the installation position of the sign 210 based on the position information stored by the sign storage unit 309 in association with the characteristic pattern of the sign 210. Then, the position specifying unit 304 stores, for example, the image capturing time of the sign 210 and the installation position in association with each other in a storage unit (not shown).

[0044] The similarity calculation unit 303 can calculate the similarity using various methods. Furthermore, the position identification unit 304 can identify the position of the marker 210 using various methods. Below, examples of a method for calculating the similarity and a method for identifying the marker 210 are shown. However, applicable methods for each are not limited to these.

[0045] Fig. 4 is an explanatory diagram illustrating a method for identifying a sign 210 included in an image to be inspected. Fig. 4 shows an image to be inspected that includes signs 210a, 210b, and 210c. Fig. 4 also shows an example of identifying sign 210b included in the image to be inspected.

[0046] The similarity calculation unit 303 calculates the similarity between an area at one end of the inspection target image and the characteristic pattern of the sign 210b by cross-correlation processing. Next, the similarity calculation unit 303 calculates the similarity between an area shifted a predetermined distance from the area where the similarity was calculated toward the other end of the inspection target image and the characteristic pattern of the sign 210b by cross-correlation processing. In this way, the similarity calculation unit 303 repeats the process of calculating the similarity between the area at one end of the inspection target image and the characteristic pattern of the sign 210b for each predetermined shift distance. As a result, the similarity calculation unit 303 calculates the similarity between the area at one end of the inspection target image and the characteristic pattern of the sign 210b.

[0047] The position identifying unit 304 identifies the position where the similarity to the characteristic pattern of the sign 210b is highest as the position where the sign 210b is located. Then, the position identifying unit 304 identifies the shift width from one end of the image to be inspected to the position where the sign 210b is located. For example, if the position identifying unit 304 identifies the position where the sign 210b is located shifted by x pixels from one end of the image to the other end, the shift width is set to x pixels. Note that the position identifying unit 304 may also identify the shift width as the distance from one end of an area of ​​the image to be inspected that corresponds to the captured image of the frame including the sign 210b to the position where the sign 210b is located.

[0048] The position specifying unit 304 obtains the image capturing time when the sign 210b is captured based on the specified shift width x. This image capturing time can also be said to be the time when the image capturing device 201 that captured the image of the sign 210b passed the installation position of the sign 210b (hereinafter also referred to as the passing time).

[0049] For example, the position identification unit 304 compares two consecutive frame images of the video data that forms the basis of the inspection target image. If the frame rate (fps) of the video data is α and the imaging ranges of the two compared images are shifted by y pixels, the shift of y pixels corresponds to the passage of 1 / α seconds. Therefore, the position identification unit 304 determines that the image of the sign 210b was captured z = x / (y × α) seconds after the image of the frame including the sign 210b was captured.

[0050] The similarity calculation unit 303 and the position identification unit 304 perform the above-mentioned processing for all signs 210 installed on the structure to be inspected (or all signs 210 stored in the sign memory unit 309 in association with the structure to be inspected).

[0051] The position specifying unit 304 associates the image capturing time of the sign 210 calculated from the shift width with the image capturing location (i.e., installation position). For example, the position specifying unit 304 outputs and stores a pulse value (=1) at the image capturing time on the time axis.

[0052] As described above, the sign storage unit 309 stores, for each sign 210 installed on the structure to be inspected, the characteristic pattern of the sign 210 in association with position information indicating the installation position of the sign 210. Therefore, if the position identification unit 304 can identify the image capture time (i.e., the passing time) of each sign 210, it can calculate the traveling speed of the commercial vehicle 200 from the distance between the signs 210 and the time required for passing.

[0053] The position identification unit 304 can calculate the traveling speed k' of the commercial vehicle 200 based on the installation position of each sign 210 and the image capture time (i.e., the passing time) of each sign 210. In this embodiment, the image generation unit 302 generates the inspection target image from the received video data using a traveling speed k that is predetermined as the traveling speed of the commercial vehicle. However, the traveling speed k may significantly deviate from the actual traveling speed. Therefore, for example, the position identification unit 304 determines whether the difference between the calculated traveling speed k' and the traveling speed k exceeds a predetermined threshold (e.g., 0.1 m / s). If the difference exceeds the predetermined threshold (i.e., if the deviation is large), the image generation unit 302 regenerates the inspection target image from the received video data using the traveling speed k'. Then, the similarity calculation unit 303 and the position identification unit 304 re-identify the position and image capture time of each sign 210 for the regenerated inspection target image. With this configuration, the inspection object image can be easily generated using a predetermined traveling speed k, and the inspection object image can be suitably corrected using the calculated traveling speed k'.

[0054] The imaging direction classification unit 305 classifies the inspection target images based on the imaging direction. For example, in the example shown in FIG. 2, the imaging devices 201 of the first and fourth commercial vehicles 200 are installed so as to capture images in the direction above the vehicle. Therefore, the imaging direction classification unit 305 classifies the inspection target images generated using the images captured by the imaging device 201 of the first train set and the inspection target images generated using the images captured by the imaging device 201 of the fourth train set into the same category. Furthermore, the imaging direction classification unit 305 classifies the inspection target images generated using the images captured by the imaging device 201 on a first occasion and the inspection target images generated using the images captured by the same imaging device 201 on a second occasion (for example, on a date and time different from the first occasion) in the same imaging direction into the same category.

[0055] The superimposition unit 306 superimposes multiple inspection target images (i.e., multiple inspection target images captured in the same imaging direction) classified into the same category by the imaging direction classification unit 305 based on the position of the sign 210 identified in each inspection target image by the position identification unit 304.

[0056] Before the superimposing unit 306 performs the superimposing process, the position specifying unit 304 determines whether the positions of the signs 210 specified in each of the images to be inspected match. If the positions of the signs 210 specified in each of the images to be inspected do not match, the position specifying unit 304 changes the specified positions of the signs 210.

[0057] For example, consider the following case: The imaging direction classification unit 305 classified the first, second, and third inspection target images into the same category. Each inspection target image includes a sign 210a, a sign 210b, and a sign 210c, in this order. The signs 210a and 210c share the same specific position in each of the inspection target images. However, the specific position of the sign 210b differs between the first inspection target image, the second inspection target image, and the third inspection target image. In this case, the position identification unit 304 changes the specific position of the sign 210b in the first inspection target image to the specific position of the sign 210b in the second inspection target image and the third inspection target image. The image generation unit 302 also regenerates a predetermined area (e.g., the area between the signs 210a and 210c) in the first inspection target image that includes the sign 210b whose specific position has been changed by the position identification unit 304. With this configuration, the accuracy of the overlay can be improved.

[0058] The anomaly determination unit 307 compares the latest inspection target image with previously generated inspection target images for the same inspection target structure. Based on the comparison result, the anomaly determination unit 307 determines whether or not an abnormality has occurred in the inspection target structure.

[0059] The data output unit 308 outputs the inspection target image generated by the image generation unit 302 and the abnormality determination result by the abnormality determination unit 307. For example, the data output unit 308 outputs the inspection target image and the abnormality determination result to a display unit (not shown) such as a display device for display. Also, for example, the data output unit 308 outputs the inspection target image and the abnormality determination result to a storage unit (not shown) for storage.

[0060] Next, a description will be given of the operation of the inspection system 100. Figures 5 to 7 are flowcharts illustrating the operation of the inspection system. Note that the operation examples shown in Figures 5 to 7 do not limit the operation of the inspection system 100 of the present disclosure.

[0061] 5 is a flowchart illustrating the operation of a device mounted on the commercial vehicle 200. The imaging device 201 captures an image of a structure to be inspected, on which a sign 210 has been installed in advance, while a train including the commercial vehicle 200 is traveling (step S201). The video data storage unit 202 stores the video data captured by the imaging device 201 (step S202).

[0062] When the train comprising the commercial vehicle 200 returns to the vehicle depot, the video data transmission unit 203 transmits the video data stored in the video data storage unit 202 to the image processing device 300 via a wireless network (or a wired network) (step S203).

[0063] 6 and 7 are flowcharts illustrating the operation of the image processing device 300. Fig. 6 is a flowchart illustrating the overall processing executed by the image processing device 300.

[0064] The video data receiving unit 301 receives video data from the video data transmitting unit 203 of each commercial vehicle 200 (step S301).

[0065] Next, the image generating unit 302 generates an inspection target image including the structure to be inspected using the received video data (i.e., the captured images of each frame captured in time series) (step S302). When the image generating unit 302 first generates an inspection target image based on one piece of video data, it generates the inspection target image using a predetermined traveling speed k. When the image generating unit 302 regenerates an inspection target image based on one piece of video data (N in step S303g described later), it generates the inspection target image using the traveling speed k' calculated in step S303f described later.

[0066] Next, the image processing device 300 executes a sign identification process to identify the sign 210 included in the inspection target image (step S303). The sign identification process will be described in detail later.

[0067] The image processing device 300 executes the processes of steps S302 and S303 for the train-related components for which video data has been received (that is, for all received video data) (step S304).

[0068] When the processes of steps S302 and S303 for all received video data are completed (Y in step S304), the imaging direction classification unit 305 classifies the inspection target images based on the imaging direction (step S305).

[0069] Next, if the specified position of the sign 210 does not match between the inspection target images captured in the same imaging direction, the position specifying unit 304 changes the specified position of the sign 210 (step S306).

[0070] Next, the superimposing unit 306 superimposes the multiple inspection target images classified into the same category by the imaging direction classification unit 305 based on the position of the sign 210 identified in each inspection target image by the position identification unit 304 (step S307). After that, for example, the data output unit 308 outputs the superimposed inspection target image.

[0071] Next, the anomaly determination unit 307 compares the latest inspection target image with the previously generated inspection target image for the same inspection target structure. Then, the anomaly determination unit 307 determines whether or not an abnormality has occurred in the inspection target structure based on the comparison result (step S308). Thereafter, for example, the data output unit 308 outputs the anomaly determination result.

[0072] Next, the sign identification process of step S303 will be described in detail. Fig. 7 is a flowchart illustrating the sign identification process executed by the image processing device 300.

[0073] The similarity calculation unit 303 calculates the similarity between the inspection target image and the characteristic pattern of the sign 210 for each predetermined shift width by cross-correlation processing (step S303a).

[0074] Next, the position specifying unit 304 specifies the shift width of the position in the inspection target image that has the highest similarity to the characteristic pattern (step S303b). The position specifying unit 304 specifies that the marker 210 corresponding to the characteristic pattern is present at this position.

[0075] Next, based on the identified shift width, the position identifying unit 304 identifies the image capturing time when the sign 210 was captured. Furthermore, the position identifying unit 304 associates the image capturing time of the sign 210 with the image capturing point (i.e., the installation position) (step S303c).

[0076] Next, the position specifying unit 304 outputs a pulse value (=1) at the imaging time on the time axis, and stores it in a storage unit (not shown) (step S303d).

[0077] The image processing device 300 executes the above-mentioned steps S303a to S303d for all the signs 210 installed on the structure to be inspected (or for all the signs 210 stored in the sign memory unit 309 in association with the structure to be inspected) (step S303e).

[0078] When the processing of steps S303a to S303d for all signs 210 is completed (Y in step S303e), the position identification unit 304 calculates the traveling speed k' of the commercial vehicle 200 from the image capture point (i.e., installation position) and image capture time of each sign 210 (step S303f). Then, the position identification unit 304 determines whether the difference between the calculated traveling speed k' and a predetermined traveling speed k is within a predetermined threshold (step S303g). If the difference between the traveling speed k' and the traveling speed k is within the predetermined threshold (Y in step S303g), the sign identification process ends.

[0079] If the difference between the traveling speed k' and the traveling speed k exceeds a predetermined threshold (N in step S303g), the process proceeds to step S302. Then, the image generation unit 302 generates the inspection target image again using the traveling speed k'. Thereafter, the image processing device 300 executes the sign identification process of step S303 again.

[0080] Next, the effects of this embodiment will be described. Generally, in railways, equipment is loaded onto a vehicle dedicated to inspection, and the vehicle travels at about 20 km / h while acquiring predetermined data and carrying out inspections. For example, a camera, laser, or ultrasound is used to acquire data. However, this type of inspection method requires the preparation of a vehicle dedicated to inspection. Furthermore, since the traveling speed of the vehicle dedicated to inspection is slow, it takes time to acquire data and carry out inspections.

[0081] In this embodiment, an imaging device 201 mounted on a business vehicle 200 used for business captures an image of a structure to be inspected, on which a sign 210 has been installed in advance. An image generation unit 302 of an image processing device 300 generates an inspection target image including the structure using the captured images captured by the imaging device 201 (for example, captured images of each frame captured in time series as video data). A position identification unit 304 identifies the sign 210 included in the inspection target image generated by the image generation unit 302. The superimposing unit 306 superimposes a first inspection target image generated using a first captured image (e.g., an image captured by the imaging device 201 mounted on the commercial vehicle 200 constituting the first trainset shown in FIG. 2 ) on a second inspection target image generated using a second captured image different from the first captured image (e.g., an image captured by the imaging device 201 mounted on the commercial vehicle 200 constituting the first trainset shown in FIG. 2 on an occasion different from the first captured image, or an image captured by the imaging device 201 mounted on the commercial vehicle 200 constituting the fourth trainset shown in FIG. 2 ) based on the position of the sign 210 identified by the position identifying unit 304. The data output unit 308 outputs the inspection target image superimposed by the superimposing unit 306. This configuration makes it possible to acquire data for inspecting structures without using a dedicated inspection vehicle. Furthermore, since data can be acquired while the commercial vehicle is in operation, the time required for inspection work can be reduced. For example, video for inspecting tunnel walls can be acquired in conjunction with the operation of a commercial Shinkansen train. Furthermore, in this embodiment, highly accurate data can be obtained by suitably overlapping a plurality of images of the object to be inspected.

[0082] 8 is a block diagram illustrating the configuration of a computer according to the present disclosure. A CPU 1000 executes processing in accordance with an image processing program stored in a storage device 1001, thereby realizing the functions of the image processing device 300 according to the above embodiment.

[0083] That is, the CPU 1000 executes processing in accordance with the image processing program stored in the storage device 1001, thereby realizing the functions of the image generation unit 302, similarity calculation unit 303, position identification unit 304, imaging direction classification unit 305, superposition unit 306, abnormality determination unit 307, and data output unit 308 of the image processing device 300 shown in FIG. 3.

[0084] The storage device 1001 is, for example, a non-transitory computer readable medium. The non-transitory computer readable medium includes various types of tangible storage media. Specific examples of the non-transitory computer readable medium include semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), and flash ROM). The storage device 1001 realizes, for example, the functions of the video data storage unit 202 and the sign storage unit 309 shown in FIG. 3 .

[0085] The memory 1002 is realized by, for example, a RAM (Random Access Memory), and is a storage means for temporarily storing data when the CPU 1000 executes processing.

[0086] Next, an overview of the present disclosure will be described. Fig. 9 is a block diagram illustrating the main components of an inspection system. The inspection system 10 (e.g., corresponding to the inspection system 100) shown in Fig. 9 includes an imaging device 21 (in the embodiment, realized by the imaging device 201) mounted on a business vehicle 20 (in the embodiment, realized by the business vehicle 200) used for business, and an image processing device 30 (in the embodiment, realized by the image processing device 300). The imaging device 21 captures an image of an inspection target structure (e.g., corresponding to the inner wall of a tunnel) on which a sign (e.g., corresponding to the sign 210) has been installed in advance. The image processing device 30 includes a generation means 31 (in the embodiment, realized by an image generation unit 302) that generates an inspection target image including the structure using the image captured by the imaging device 21, and an identification means 32 (in the embodiment, realized by a position identification unit 304) that identifies the sign included in the inspection target image generated by the generation means 31. and a superimposition means 33 (realized by the superimposition unit 306 in the embodiment) that superimposes a first inspection target image generated using a first captured image (for example, an image captured by the imaging device 201 mounted on the commercial vehicle 200 that constitutes the first formation shown in FIG. 2 ) on a second inspection target image generated using a second captured image different from the first captured image (for example, an image captured by the imaging device 201 mounted on the commercial vehicle 200 that constitutes the first formation shown in FIG. 2 on an occasion different from the first captured image, or an image captured by the imaging device 201 mounted on the commercial vehicle 200 that constitutes the fourth formation shown in FIG. 2 ), based on the position of the sign identified by the identification means 32; and an output means 34 (realized by the data output unit 308 in the embodiment) that outputs the inspection target image superimposed by the superimposition means 33. With this configuration, the inspection system 10 can acquire data for inspecting a structure without using a vehicle dedicated to inspection. In addition, the inspection system 10 can acquire data while the commercial vehicle is in operation, thereby reducing the time required for inspection work. Furthermore, the inspection system 10 can acquire highly accurate data by appropriately overlaying multiple images of the inspection target.

[0087] 10 is a block diagram illustrating a main part of an image processing device. The image processing device 30 (e.g., image processing device 300) shown in FIG. 10 includes: a generating means 31 (realized by an image generating unit 302 in the embodiment) that generates an inspection target image including a structure (e.g., corresponding to an inner wall of a tunnel, etc.) on which a sign (e.g., corresponding to sign 210) is previously installed, using a captured image of the structure (e.g., corresponding to sign 210) captured by an imaging device (e.g., realized by imaging device 201 in the embodiment) mounted on a business vehicle (e.g., realized by business vehicle 200) used for business; an identifying means 32 (e.g., realized by a position identifying unit 304 in the embodiment) that identifies the sign included in the inspection target image generated by the generating means 31; and a first captured image (e.g., The image processing device 30 includes a superimposing means 33 (realized by the superimposing unit 306 in the embodiment) that superimposes a first inspection target image generated using a captured image captured by the imaging device 201 mounted on the commercial vehicle 200 constituting the first formation shown in FIG. 2 ) on a second inspection target image generated using a second captured image different from the first captured image (for example, an image captured by the imaging device 201 mounted on the commercial vehicle 200 constituting the first formation shown in FIG. 2 on an occasion different from the first captured image, or an image captured by the imaging device 201 mounted on the commercial vehicle 200 constituting the fourth formation shown in FIG. 2 ), and an output means 34 (realized by the data output unit 308 in the embodiment) that outputs the inspection target image superimposed by the superimposing means 33. With this configuration, the image processing device 30 can acquire data for inspecting a structure without using a vehicle dedicated to inspection. Furthermore, since the image processing device 30 can acquire data while the commercial vehicle is in operation, the time required for inspection work can be reduced. Furthermore, the image processing device 30 can obtain highly accurate data by appropriately superimposing a plurality of images of the object to be inspected.

[0088] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Each embodiment can be combined with other embodiments as appropriate.

[0089] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0090] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0091] (Appendix 1) a generation means for generating an inspection target image including the structure, using an image captured by an imaging device mounted on a commercial vehicle used for business operations, of the structure to be inspected, on which a sign has been previously installed; an identification means for identifying the mark included in the inspection object image generated by the generation means; a superimposing means for superimposing a first inspection object image generated using a first captured image and a second inspection object image generated using a second captured image different from the first captured image, based on the position of the sign identified by the identifying means; and an output means for outputting the superimposed inspection object image. 1. An image processing device comprising:

[0092] (Appendix 2) the first captured image is captured by a first imaging device mounted on a commercial vehicle that constitutes a first train configuration, The second captured image is an image captured by a second imaging device mounted on a commercial vehicle that constitutes a second train formation different from the first train formation. 2. The image processing device of claim 1.

[0093] (Appendix 3) the imaging device is installed so as to capture an image in one of a plurality of predetermined imaging directions; The first captured image and the second captured image are captured images captured by an imaging device in the same imaging direction. 3. The image processing device according to claim 1 or 2.

[0094] (Appendix 4) the marking comprises a characteristic pattern; a similarity calculation means for calculating a similarity between the inspection target image and the characteristic pattern for each predetermined shift width; The identifying means identifies the marker as being present at a position having the highest similarity to the characteristic pattern. 4. An image processing device according to any one of claims 1 to 3.

[0095] (Appendix 5) The similarity calculation means calculates the similarity by cross-correlation processing. 5. The image processing device according to claim 4.

[0096] (Appendix 6) The generating means generating an inspection target image using a first traveling speed that is predetermined as a traveling speed of a commercial vehicle; When a difference between a second traveling speed calculated as the traveling speed of the commercial vehicle based on the installation position of the sign specified in the generated inspection target image and the image capturing time and the first traveling speed exceeds a predetermined threshold, the inspection target image is generated again using the second traveling speed. 6. An image processing device according to any one of claims 1 to 5.

[0097] (Appendix 7) The identification means changes the position of the marker identified in the first inspection target image based on the position of the marker identified in the second inspection target image; The generating means regenerates a predetermined area of ​​the first inspection target image that includes the sign whose position has been changed by the identifying means. 6. An image processing device according to any one of claims 1 to 5.

[0098] (Appendix 8) An imaging device mounted on a commercial vehicle used for business; an image processing device, The imaging device captures an image of a structure to be inspected on which a sign has been installed in advance, The image processing device includes: a generation means for generating an inspection object image including the structure using an image captured by the imaging device; an identification means for identifying the mark included in the inspection object image generated by the generation means; a superimposing means for superimposing a first inspection object image generated using a first captured image and a second inspection object image generated using a second captured image different from the first captured image, based on the position of the sign identified by the identifying means; and an output means for outputting the superimposed inspection object image. An inspection system characterized by:

[0099] (Appendix 9) The computer An image of the inspection target structure, which has a sign installed in advance, is generated using an image captured by an imaging device mounted on a commercial vehicle used for business operations, and the image of the inspection target structure including the structure is generated; Identifying the sign included in the generated inspection target image; superimposing a first inspection object image generated using the first captured image and a second inspection object image generated using a second captured image different from the first captured image based on the identified position of the marker; Output the superimposed inspection target image An image processing method comprising:

[0100] (Appendix 10) On the computer, a generation process for generating an inspection target image including the structure using an image captured by an imaging device mounted on a commercial vehicle used for business operations, the image capturing the inspection target structure on which a sign has been installed in advance; an identification process for identifying the sign included in the generated inspection target image; a superimposition process for superimposing a first inspection object image generated using a first captured image and a second inspection object image generated using a second captured image different from the first captured image, based on the identified position of the marker; Output processing to output the superimposed inspection target image An image processing program for executing the above.

[0101] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 7 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 8, 9, and 10 in the same dependency relationship as Supplementary Notes 2 to 7. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods. [Explanation of symbols]

[0102] 10,100 Inspection System 20,200 commercial vehicles 21,201 Imaging device 30,300 image processing devices 31 Generation means 32 Specific means 33 Superposition means 34 Output Method 202 Video data storage unit 203 Video data transmission unit 210 signs 301 Video data receiving unit 302 Image Generation Unit 303 Similarity calculation part 304 Location identification part 305 Imaging direction classification unit 306 Overlapping section 307 Abnormality determination section 308 Data Output Unit 309 Sign storage unit 1000 CPU 1001 Storage device 1002 memory

Claims

1. a generation means for generating an inspection target image including the structure, using an image captured by an imaging device mounted on a commercial vehicle used for business operations, of the structure to be inspected, on which a sign has been previously installed; an identification means for identifying the mark included in the inspection object image generated by the generation means; a superimposing means for superimposing a first inspection target image generated using a first captured image and a second inspection target image generated using a second captured image different from the first captured image, based on the position of the marker identified by the identifying means; and an output means for outputting the superimposed inspection object image.

1. An image processing device comprising:

2. the first captured image is captured by a first imaging device mounted on a commercial vehicle that constitutes a first train configuration, The second captured image is an image captured by a second imaging device mounted on a commercial vehicle that constitutes a second train configuration different from the first train configuration. The image processing device according to claim 1 .

3. the imaging device is installed so as to capture an image in one of a plurality of predetermined imaging directions; The first captured image and the second captured image are captured images captured by an imaging device in the same imaging direction.

3. The image processing device according to claim 1.

4. the marking comprises a characteristic pattern; a similarity calculation means for calculating a similarity between the inspection target image and the characteristic pattern for each predetermined shift width; The identifying means identifies the marker as being present at a position having the highest similarity to the characteristic pattern.

3. The image processing device according to claim 1.

5. The similarity calculation means calculates the similarity by cross-correlation processing.

5. The image processing device according to claim 4.

6. The generating means generating an inspection target image using a first traveling speed that is predetermined as a traveling speed of a commercial vehicle; When a difference between a second traveling speed calculated as the traveling speed of the commercial vehicle based on the installation position of the sign specified in the generated inspection target image and the image capture time and the first traveling speed exceeds a predetermined threshold, the inspection target image is generated again using the second traveling speed.

3. The image processing device according to claim 1.

7. the specifying means changes the position of the marker specified in the first inspection target image based on the position of the marker specified in the second inspection target image; The generating means regenerates a predetermined area of ​​the first inspection target image that includes the sign whose position has been changed by the identifying means.

3. The image processing device according to claim 1.

8. An imaging device mounted on a commercial vehicle used for business; an image processing device, The imaging device captures an image of a structure to be inspected on which a sign has been installed in advance, The image processing device includes: a generation means for generating an inspection target image including the structure using an image captured by the imaging device; an identification means for identifying the mark included in the inspection object image generated by the generation means; a superimposing means for superimposing a first inspection target image generated using a first captured image and a second inspection target image generated using a second captured image different from the first captured image, based on the position of the marker identified by the identifying means; and an output means for outputting the superimposed inspection object image. An inspection system characterized by:

9. The computer An image of the inspection target structure, which has a sign installed in advance, is generated using an image captured by an imaging device mounted on a commercial vehicle used for business operations, and the image of the inspection target structure including the structure is generated; Identifying the sign included in the generated inspection target image; superimposing a first inspection object image generated using the first captured image and a second inspection object image generated using a second captured image different from the first captured image based on the identified position of the marker; Output the superimposed inspection target image An image processing method comprising:

10. On the computer, a generation process for generating an inspection target image including the structure using an image captured by an imaging device mounted on a commercial vehicle used for business operations, the image capturing the inspection target structure on which a sign has been installed in advance; an identification process for identifying the sign included in the generated inspection target image; a superimposition process for superimposing a first inspection target image generated using a first captured image and a second inspection target image generated using a second captured image different from the first captured image, based on the identified position of the marker; Output processing to output the superimposed inspection target image An image processing program for executing the above.

Images