Image processing device and inspection system

Abstract

To provide an image processing device capable of correcting a time difference even between pieces of image data acquired by individually imaging from distant positions, and an inspection system with the image processing device.SOLUTION: An image processing device 50 comprises: an image information acquisition unit 60 which acquires a plurality of pieces of image information GI obtained by adding time information TI to image data GD obtained by imaging different positions; and a time correction unit 72 which corrects a time deviation from a correlation function CF between the pieces of image information GI, on the basis of an index IX indicative of the state of an imaging environment.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device for performing an analysis process on an image captured of an inspection target, for example, and an inspection system including the image processing device. [Background technology]

[0002] For example, known image processing techniques include one in which a common control unit is provided for multiple cameras (see Patent Document 1) and one in which the same subject is photographed by multiple cameras (see Patent Document 2).

[0003] However, for example, the aspect of Patent Document 1 may not be able to adequately handle cases where it is desired to independently control cameras located at distant locations, while the aspect of Patent Document 2 may not be able to adequately handle cases where it is desired to have different imaging ranges for each camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-216736 A [Patent Document 2] Patent No. 3958638 Summary of the Invention

[0005] The present invention has been made in consideration of the above-mentioned points, and aims to provide an image processing device and an inspection system equipped with the same that can correct the time difference between image data captured individually from distant positions.

[0006] An image processing device for achieving the above-mentioned objective includes an image information acquisition unit that acquires multiple pieces of image information in which image data captured at different positions is added with time information, and a time correction unit that corrects the time difference from a correlation function between the image information based on an index indicating the state of the shooting environment.

[0007] In the image processing device, the image information acquisition unit can acquire image data captured separately and independently at different positions, and the time correction unit can correct the time shift (time difference) from the correlation function between image information in which time information is added to image data based on an index indicating the state of the shooting environment. In other words, a process equivalent to time synchronization at the time of shooting between the acquired multiple image data can be performed after the fact.

[0008] In a specific aspect of the present invention, a distance defined between consecutively captured images is included as an index indicating the state of the photographing environment. In this case, for example, a time lag (time difference) between image information can be corrected by comparing the change in the distance defined between the images using a correlation function.

[0009] In another aspect of the present invention, the brightness of an image is included as an index indicating the state of the shooting environment. In this case, for example, a time lag (time difference) between image information can be corrected by comparing changes in the brightness of the images using a correlation function.

[0010] In yet another aspect of the present invention, the correlation function is a function that indicates a correlation between time change points related to image information for time change points determined from a preset threshold value for an index indicating a state of the shooting environment. In this case, the correlation function based on the time change points corresponding to the index enables quick and accurate calculation of the time shift (time difference).

[0011] In yet another aspect of the present invention, the time change point is binary data for the image information based on a threshold, and the correlation function is calculated from the binary data. In this case, the binary data enables simple and rapid processing.

[0012] An inspection system for achieving the above-mentioned objective is an inspection system equipped with the above-mentioned image processing device, and includes a plurality of imaging units that generate image information by adding time information to image data obtained by imaging the equipment to be inspected from different positions, and an image information output unit that outputs the image information generated by the plurality of imaging units to the image processing device.

[0013] In the inspection system, the image processing device can correct time lag (time difference) for image information generated by a plurality of imaging units and output to the image processing device by an image information output unit. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1A is an oblique view for explaining a usage mode of an inspection system according to one embodiment, FIG. 1B is an image diagram conceptually showing image data captured by an imaging unit constituting the inspection system, and FIG. 1C is a diagram showing the result of image synthesis. [Diagram 2] 1 is a block diagram showing an example of a configuration of an inspection system including an image processing device according to an embodiment. [Diagram 3] FIG. 2 is a diagram for explaining processing contents in an image processing device. [Figure 4] FIG. 1 is a block diagram for explaining a configuration example of an inspection system. [Diagram 5] 4 is a flowchart of a process in the image processing device. [Figure 6] 11 is a flowchart for explaining an example of calculation of an index among processes in the image processing device. [Figure 7] FIG. 10 is a diagram illustrating an example of calculation of a hash value. [Figure 8] Graph (A) shows how the distance between images changes, and graph (B) shows how the average brightness in the images changes. [Figure 9] 11 is a flowchart for explaining an example of calculation of a time difference between cameras among processes in the image processing device. [Figure 10] 1 is a graph for explaining an example of binarization of data. [Figure 11] 13 is a graph showing an example of a correlation function. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an example of an image processing device according to an embodiment and an inspection system including the same will be described with reference to FIG. 1 and the like. FIGS. 1(A) to 1(C) are diagrams showing an example of a usage mode of the inspection system 100 as a premise for explaining the inspection system 100 according to the present embodiment. For example, FIG. 1(A) is a perspective view showing a state in which an example of an inspection target CO in the inspection system 100 is a train facility TE. Also, FIG. 1(B) is an image diagram conceptually showing images G (video) captured continuously by a plurality of cameras 10 (cameras 1, 2, ...) as an imaging unit constituting the inspection system 100 as exemplified in FIG. 1(A), and FIG. 1(C) is a diagram showing a composite image SG as a result of combining images.

[0016] FIG. 2 is a block diagram showing an example of the configuration of an inspection system 100 including an image processing device 50 according to this embodiment, and FIG. 3 is a diagram for explaining the processing contents in the image processing device 50.

[0017] First, in the example shown in FIG. 1(A), the multiple cameras 10 constituting the inspection system 100 are attached and fixed at predetermined positions on the roof of the train TR, and capture an inspection target CO present in the traveling direction D1 of the train TR. The inspection system 100 judges whether or not there is an abnormality in the inspection target CO based on the captured images. Meanwhile, the train equipment TE as the inspection target CO in the illustrated example is a member in which an iron pole is installed so as to straddle the track, and the multiple cameras 10 capture the entire train equipment TE from different positions, and the captured images are pasted together (combined) as illustrated in FIG. 1(C) to capture the train equipment TE as the inspection target CO. That is, the inspection system 100 performs various checks, such as whether or not there is an abnormality in the train equipment TE, based on the image combined as described above.

[0018] As described above, in the example usage mode shown in Figures 1(A) to 1(C), each camera 10 mounted on the train TR continuously captures images of one train facility TE while moving from different positions, thereby acquiring multiple images G as shown in Figure 1(B). From the acquired images G, appropriate ones, i.e., those that include the train facility TE (for example, those that include partial images TE1i, TE2i), are selected and combined, making it possible to form a composite image SG that includes partial images TEi showing the train facility TE as exemplified in Figure 1(C).

[0019] Here, when the inspection system 100 is used in the above-mentioned manner, it is desirable that the imaging timing of each of the cameras 10 is synchronized. In other words, it is desirable that, for example, one control unit (a common control unit) centrally controls the operation of the multiple cameras 10 so that time information indicating the time when the images were captured is common to each of the cameras 10. However, when the distance between the cameras 10 is large, such as in the above-mentioned example in which multiple cameras 10 are mounted on a train TR to capture images, it is difficult to adopt such a manner, and each camera 10 is controlled independently.

[0020] In the inspection system 100 according to this embodiment or the image processing device 50 constituting the inspection system 100 (see FIG. 2), the multiple cameras 10 constituting the inspection system 100 are independently controlled (perform imaging operations), and even if a time lag (time difference) may occur between the cameras 10, this can be corrected. This makes it possible to accurately capture the CO to be inspected.

[0021] Hereinafter, with reference to Fig. 2, a configuration example of an inspection system 100 including an image processing device 50 will be described. As shown in the figure, the inspection system 100 includes an image information generating device 40 including a plurality of cameras 10, and an image processing device 50. In the example shown in the figure, the image information generating device 40 is an on-board facility mounted on a train TR, and is composed of train rooftop equipment 40A provided on the roof (outdoors) of the train TR, and in-train equipment 40B provided indoors. The train rooftop equipment 40A includes a camera 10 and lighting (strobe) 11. The in-train equipment 40B includes a control unit 20, an operation unit 21, an external storage medium 22, an image information output unit 30, and the like. In addition, in the illustrated example, one train roof-mounted device 40A and one corresponding in-train device 40B form a set, and the image information generating device 40 is made up of a plurality of such combinations. However, in the following, for the sake of simplicity, the explanation will focus on one set of train roof-mounted device 40A and in-train device 40B.

[0022] As described above, the camera 10 is fixed at a predetermined position and posture, and image data GD generated by imaging with the camera 10 is output to the control unit 20. Note that one image data GD corresponds to one image G illustrated in FIG. 1(B). In this example, the camera 10 is a near-infrared camera, and the lighting 11 irradiates near-infrared light according to the camera 10. By using near-infrared light that is invisible to the eye, it is possible to suppress the influence on the surroundings during imaging. Specifically, even in the case of imaging for equipment inspection on the track in the middle of the night, for example, even if the lighting 11 irradiates a large amount of light, the possibility of causing light pollution to the neighborhood is suppressed. Note that the camera 10 and the lighting 11 operate according to a command signal from the control unit 20.

[0023] The control unit 20 is composed of, for example, a CPU or various circuits, and is connected to each unit to control various operations related to imaging. For example, the control unit 20 is connected to an operation unit 21 as an input device, receives various command signals based on the operation of the operation unit 21 by a user, and performs various operations in response to the signals. Note that, although the illustrated example illustrates a case where one camera 10 is controlled, a case where multiple cameras 10 are controlled is also conceivable. Variations in this regard will be described later with reference to FIG. 4.

[0024] As described above, the control unit 20 controls the operation of the camera 10 and the lighting 11, and in particular assigns time to the image data GD from the camera 10. That is, the control unit 20 functions as a time assignment unit TA that adds time information TI to the image data GD sequentially output from the camera 10. Hereinafter, information in which the time information TI has been added to the image data GD is referred to as image information GI. The control unit 20 stores the sequentially generated image information GI in the external storage medium 22.

[0025] The external storage medium 22 is composed of various storage devices, etc., and stores programs necessary for various operations, as well as various information related to the images acquired as described above, in particular, multiple pieces of image information GI composed of multiple pieces of image data GD and time information TI.

[0026] The image information output section 30 outputs to the image processing device 50 a plurality of pieces of image information GI generated by the cameras 10 which are a plurality of imaging sections.

[0027] In this case, the image information generating device 40, which is composed of multiple sets of train roof equipment 40A and train interior equipment 40B, has multiple image information output units 30, so the image processing device 50 receives image information GI, each of which includes consecutive image data GD, from the multiple image information output units 30.

[0028] The image processing device 50 includes an image information acquisition unit 60, a main control unit 70, and a storage unit 80 in order to perform various image processes on the received image information GI.

[0029] The image information acquisition unit 60 is an interface unit that receives various information output from each image information output unit 30, in particular, a plurality of pieces of image information GI.

[0030] The main control unit 70 is composed of, for example, a CPU or various circuits, and is a main body of the image processing device 50 that performs various image processing, and also performs data processing. For this reason, in this embodiment, the main control unit 70 includes or functions as an image analysis unit 71 and a time correction unit 72. The image analysis unit 71 performs various image processing on the image information GI, and in particular, in this embodiment, calculates an index indicating the state of the shooting environment in each image information GI (image data GD) in order to be able to calculate the time difference between the image information GI transmitted from different image information output units 30. More specifically, calculation is performed on the distance specified between consecutively captured images in the image information GI, which is one of the indexes indicating the state of the shooting environment, and the brightness of the image. In addition, the time correction unit 72 corrects the time difference between the image information GI based on the index indicating the state of the shooting environment calculated in the image analysis unit 71, i.e., the distance and brightness.

[0031] The memory unit 80 is composed of various storage devices, etc., and in accordance with instructions from the main control unit 70, stores, for example, image information GI received by the image information acquisition unit 60, as well as various data as the results of various processing by the main control unit 70.

[0032] Hereinafter, the processing contents in the image processing device 50 will be described with reference to the conceptual diagram shown in Fig. 3. In the configuration described with reference to Fig. 2 and the like, the flow of various data such as image data GD is as shown in Fig. 3. To be more specific, a plurality of image data GD indicated as image data 1, 2, 3, ... in the figure are acquired as a result of imaging performed from various positions, and time information TI is respectively assigned to these pieces of image data GD, which are then accepted as image information GI in the image information acquisition section 60 of the image processing device 50. That is, a plurality of image information GI indicated as image information 1, 2, 3, ... in the figure are taken in on the image processing device 50 side, and each piece of image information 1, 2, 3, ... is composed of image data 1, 2, 3, ... and time information 1, 2, 3, ... corresponding thereto.

[0033] Each piece of image information 1, 2, 3, ... is made up of consecutive images, and as shown partially enlarged and surrounded by a dashed line, for example, image information 1 is made up of multiple (many) pieces of image information 1-1, 1-2, 1-3, .... The same is true for other pieces of image information 2, etc. In other words, from a different perspective, each piece of image information GI can be said to be an image information group made up of multiple (many) pieces of image information.

[0034] However, between these pieces of image information GI (for example, between image information 1 and image information 2), the time stamping is not necessarily synchronized, and there is a possibility that a time discrepancy occurs. For example, the cause of this discrepancy may be an error in the clock frequency of the control device for each camera 10 (see FIG. 2, etc.) used for capturing images.

[0035] In response to this, the image processing device 50 is configured to correct the time difference from the correlation function CF between the image information GI in the time correction unit 72 based on the index IX indicating the state of the shooting environment. Here, attention is paid to the distance and brightness between images as the index IX indicating the state of the shooting environment between the cameras 10 (see FIG. 2, etc.), and in particular, the degree of the time difference is calculated by capturing changes therein from a plurality (a large number) of image information 1-1, 1-2, 1-3, ... and using the correlation function CF related to the changes. A specific example of such index IX and correlation function CF will be described later with reference to FIG. 5, etc.

[0036] In addition, various configurations are possible for the installation of the image processing device 50 that performs the above-mentioned processing. For example, the image processing device 50 may be configured to exist independently of the camera 10 (see FIG. 2, etc.) in order to perform image processing separately from the generation operation of the image information GI by the camera 10 (see FIG. 2, etc.). However, even in the configuration in which the camera 10 and the image processing device 50 are disposed separately, the image information GI may be acquired by, for example, using wireless communication, so that the generation of the image information GI and the processing such as image analysis thereof may be performed simultaneously in parallel. On the other hand, the image processing device 50 may be configured to be mounted on the train together with the in-train device 40B (see FIG. 2), and the image processing device 50 may be connected to each part to perform processing simultaneously and in parallel.

[0037] A variation of the above embodiment will now be described with reference to an inspection system 100 shown in block diagram form as FIG.

[0038] First, in the above example, the train roof equipment 40A and the train interior equipment 40B are configured with one camera 10, one light, one control unit 20, etc., but this is not limiting and various combinations are possible. For example, as shown by camera 1 in FIG. 4, the train roof equipment 40A may be configured without a light. Of course, as shown by camera 2, etc., or as in the above-mentioned embodiment, the train roof equipment 40A may be configured with one camera 10 (camera 2 in the illustrated example) and one light 11 (light 1 in the illustrated example). In addition to this, for example, it is also possible to configure two cameras 3 and 4 to be controlled by one control unit 3 provided on the train interior equipment 40B side. Furthermore, it is also possible to configure two cameras 5 and 6 to have corresponding lights 2 and 3, respectively, and to control them by one control unit 4.

[0039] As for the image processing device 50, in the example of FIG. 4, the central server CV that manages various data includes the function of the image processing device 50. Specifically, the central server CV includes a memory unit ME that is composed of an external memory medium 1A that stores various information including image information GI from the external memory medium 1 in the train equipment 40B in correspondence with the above-mentioned various aspects of the external memory medium 1, and also includes a control unit MP that controls the whole system, an image processing unit GP, an operation unit (input device) OU, a display unit (output device) DS, etc. In the central server CV, various information stored in the external memory medium 1A of the memory unit ME is analyzed and processed in the image processing unit GP according to various command signals from the control unit MP, and the results are displayed on the display unit DS that is composed of a display, etc. In addition, the control unit MP receives various operation signals from an operator via the operation unit OU and performs various processes in response to the signals. As described above, the central server CV also functions as the image processing device 50.

[0040] Hereinafter, with reference to FIG. 5 etc., a detailed description will be given of an example of time lag correction using image processing according to this embodiment.

[0041] First, as shown in the flowchart of FIG. 5, in this embodiment, the processing in the image processing device 50 is roughly divided into two processes: calculation of an index (step S1) and correction of the time difference between the cameras (step S2).

[0042] Of these, the first step S1 will be described in detail with reference to Figures 6 to 8. The second step S2 will be described in detail with reference to Figures 9 to 11.

[0043] First, an example of a more detailed processing procedure for step S1 in Fig. 5 will be described with reference to the flowchart in Fig. 6. Note that, here, the main control unit 70 of the image processing device 50 illustrated in Fig. 2 functions as the image analysis unit 71.

[0044] The main control unit 70 as the image analysis unit 71 first designates a target camera 10 from among the multiple cameras 10 (step S101). That is, it determines which camera 10 the image information GI from which is to be processed. Note that in this case, the selected image information GI is assumed to be composed of multiple (a large number of) pieces of image information 1-1, etc., as exemplified for the image information 1 in FIG. 3.

[0045] Next, the main control unit 70 inputs information on one image from the image information GI (for example, image data 1-1 constituting image information 1-1 in FIG. 3) (step S102), calculates a hash value for the image information (one piece of image data) (step S103), and further uses the hash value calculated in step S103 to calculate the distance between images, which is one of the indices (step S104). Note that a specific example of the calculation of the hash value and the calculation of the distance between images based thereon will be described later with reference to FIG. 7 etc.

[0046] Next, the main control unit 70 calculates the average luminance as another index for the image information (one piece of image data) input in step S102 (step S105). A detailed example of a method for calculating the average luminance will be described later. The calculation of the average luminance may be performed simultaneously in parallel with the processing of steps S103 and S104.

[0047] When the main control unit 70 has completed the processing from step S102 to step S105, it checks whether the processing has been completed for all image information (image data) that constitutes the image information GI selected in step S101 (step S106). If the processing has not been completed (step S106: No), the main control unit 70 inputs information for a new image from the image information GI (for example, image data 1-2 that constitutes the image information 1-2 in Figure 3) (step S102) and performs the same processing.

[0048] On the other hand, in step S106, when it is confirmed that processing has been completed for all image information (image data) contained in the target image information GI (step S106: Yes), the main control unit 70 stores in the memory unit 80 the distance between images and the average brightness calculated for the image information GI selected in step S101 (for example, image information 1 in Figure 2) (step S107).

[0049] When the main control unit 70 completes the data storage process in step S107, it checks whether the processing has been completed for all of the multiple cameras 10, i.e., whether the processing has been completed for all image information GI (step S108), and if it is not completed (step S108: No), the main control unit 70 specifies a new camera 10 from the multiple cameras 10 (step S101), i.e., inputs the image information GI corresponding to that camera 10 (e.g., image information 2), and performs the same processing.

[0050] On the other hand, in step S108, if it is confirmed that the processing has been completed for all of the target cameras 10 (step S108: Yes), the main control unit 70 ends the operation as the image analysis unit 71.

[0051] The distance between the images and the average brightness obtained as described above are used as indices for correcting the time difference between the cameras shown in step S2 of FIG.

[0052] A specific example of the calculation of the hash value described above will be described below with reference to Fig. 7. First, as shown in the figure, for image information GI including an image group of consecutive images, some of the consecutive images that make up this information are designated as images G1, G2, and G3. For each of these, the hash value of an 8x8 matrix as shown in the figure is calculated by the following operational steps (1) to (5). (1) Reduce images G1, G2, and G3. (2) If images G1, G2, and G3 are color, convert them to grayscale. (3) Calculate the periodic components of the image using discrete cosine transform processing. (4) Extract low-frequency components from the periodic components. (5) Binarize the signal based on whether the low-frequency components are greater than the average.

[0053] As a result, as shown in the figure, for example, a hash value that is an 8×8 matrix can be obtained. By vectorizing this, a 64-dimensional vector is obtained. In the above embodiment, the value of each element that constitutes the vector is 0 or 1 due to binarization. In this case, the hash values ​​obtained as described above will also be similar if the original images are similar (the arrangement of 0s and 1s will be similar). Therefore, by defining the difference in hash value between images as a distance, it is possible to capture the state of change in the images.

[0054] Next, calculation of the distance between images using the hash value obtained as described above will be described. In the illustrated example, three consecutive images G1, G2, and G3 are shown, and in this case, the distance between two images adjacent in time, that is, the distance between image G1 and image G2, and the distance between image G2 and image G3, are obtained. In other words, the distance between the input image and the previous input image is calculated. The hash value of the input image is expressed as x=(x1, x2, ..., x 64 ), and the hash value of the previous input image is y=(y1,y2,…,y 64 ), then the distance d(x,y) between images is, for example, TIFF0007609629000001.tif13168. For the above three images G1, G2, and G3, when the hash values ​​of 0 or 1 shown in the 8×8 matrix corresponding to the images G1 and G2 in the figure are applied to the above formula to calculate, the distance between them is 10, and similarly, the distance between images G2 and G3 is 30. As described above, the more similar the images are, the smaller the distance becomes, so in the above case, it can be seen that images G1 and G2 are relatively similar images, while images G2 and G3 are relatively different images. As described above, in the above, the distance specified between consecutively captured images can be included as an index indicating the state of the shooting environment.

[0055] In the flowchart of Fig. 6, in the process for the first image, there is no previous input image. In this case, the distance may be calculated by comparing with a predetermined default value (for example, all 64 elements are set to 0). Alternatively, the distance may not be calculated for the first image.

[0056] Consider the characteristics of the distance d(x, y) as described above. For example, when the train TR (see FIG. 1, etc.) carrying the camera 10 is stopped, the imaging environment does not change or changes very little, so the distance between successive images is considered to be small or close to zero. On the other hand, the moment the train TR starts moving, a large change in the distance value is considered to occur, and such a change is considered to occur simultaneously in all the cameras 10 mounted on the train TR. Therefore, by capturing this change and using it as a time reference, it is possible to correct the time difference between the cameras 10. In other words, in cases where the camera 10 is sometimes stopped, the above distance d(x, y) can be used as an index indicating the state of the imaging environment.

[0057] The distance d(x, y) is an example of an index showing distance, and various definitions other than the above can be adopted for the distance as long as it is possible to show the magnitude relationship according to the difference in hash values. Also, various methods other than the above can be adopted for the hash values.

[0058] Next, calculation of the average brightness, that is, calculation of the average brightness of the entire image, will be described. For example, in each of the three images G1, G2, G3, etc., if the width of the image is W, the height is H, the brightness of the pixel at coordinates (x, y) is I(x, y), and the average brightness is m, the average brightness m is expressed as follows: It is represented by TIFF0007609629000002.tif14168. In this case, it is considered that the average luminance m changes significantly when the shooting environment changes, that is, when the surrounding brightness changes.

[0059] Consider the characteristics of the average brightness m as described above. For example, when a train TR (see FIG. 1, etc.) carrying a camera 10 enters or exits a tunnel, it is considered that the brightness will change drastically. This is also considered to be the case, for example, at night, when a light 11 (see FIG. 2, etc.) that emits near-infrared light is used. In addition, such a change is considered to occur simultaneously or almost simultaneously in all of the cameras 10 carried on the train TR. Therefore, by capturing this change and using it as a time reference, it is possible to correct the time difference between the cameras 10. That is, in a case where an instantaneous change in brightness occurs, the average brightness m can be used as an index indicating the state of the shooting environment. In a case where multiple cameras 10 are arranged at different positions in the traveling direction, such as when they are located at the front and rear ends of the train TR, some time difference will occur. In this case, it is possible to make a correction by taking into account, for example, the arrangement of each camera and the speed of the train TR when entering a tunnel.

[0060] The above average luminance m is one example of an index of luminance. Various other definitions of luminance can be adopted as long as they can accurately indicate the change in luminance. For example, it is possible to adopt the difference between the minimum and maximum luminance values ​​in an image as an index.

[0061] FIG. 8(A) is a graph showing the change in the distance d between images as one of the indices, where the horizontal axis indicates time and the vertical axis indicates the value of the distance d between images. Here, the graph shows the results of an analysis of image information GI (image information 1, 2) captured by two cameras 10 (e.g., cameras 1 and 2) during the time when a train TR equipped with multiple cameras 10 stops from a running state and starts running again. Note that the cameras 1 and 2 are not synchronized in terms of time, and the time is based on the time information TI included in each piece of image information GI. Also, a curve P1 shown by a solid line indicates the distance between images in one camera 10 (camera 1), and a curve P2 shown by a dashed line indicates the distance between images in the other camera 10 (camera 2).

[0062] While train TR is stopped, the images from cameras 1 and 2 do not change, so the distance between the images is small for both cameras. In other words, the distance between the images from both cameras 1 and 2 remains small over a certain section on the graph. However, in the example of FIG. 8(A), the section where the distance is zero or closer to zero for camera 2 is shifted to the left compared to camera 1. This difference corresponds to the time difference between the two.

[0063] FIG. 8(B) is a graph showing the change in the average brightness m in an image, which is another aspect as one of the indices, and in the figure, the horizontal axis indicates time, and the vertical axis indicates the value of the average brightness m. Here, the results are shown of an analysis of image information GI (image information 3, 4) captured by two cameras (e.g., cameras 3, 4) during the time when a train TR equipped with multiple cameras 10 enters a tunnel and then exits the tunnel. Note that the cameras 3 and 4 are not synchronized in terms of time, and the time is based on the time information TI included in each image information GI. Also, a curve Q1 shown by a solid line indicates the average brightness in one camera 10 (camera 3), and a curve Q2 shown by a dashed line indicates the average brightness in the other camera 10 (camera 4).

[0064] The average brightness of the images from cameras 1 and 2 is low from when the train TR enters the tunnel until it leaves the tunnel. However, in the example of Fig. 8(B), the section where the brightness of camera 4 is 0 or close to 0 is shifted to the right more than that of camera 3. This difference corresponds to the time difference between the two.

[0065] By using the above-mentioned distance and brightness as indicators of the state of the shooting environment, the time lag (time difference) occurring between the cameras 10 can be accurately corrected.

[0066] Next, an example of a more detailed processing procedure for step S2 in Fig. 5 will be described with reference to the flowchart in Fig. 9. Note that, in this example, the main control unit 70 of the image processing device 50 illustrated in Fig. 2 functions as the time correction unit 72.

[0067] Also, here, as an example, the time information of one camera 10 (e.g., camera 1) among the multiple cameras 10 in question is used as a reference, and the degree of deviation of the time information of other cameras 10 (camera 2, etc.) relative to camera 1 is calculated.

[0068] The main control unit 70 as the time correction unit 72 first selects two cameras out of the multiple cameras 10 and reads indices to correct the relative time difference between them (step S201). That is, one camera 10 (camera 1) and another camera 10 (e.g. camera 2) are selected, and data on indices (distance d, average brightness m) related to the image information (image information 1, 2) for these cameras is extracted from the storage unit 80 and reads them. More specifically, data such as that shown in the graph of FIG. 8(A) is read.

[0069] Next, the main control unit 70 performs binarization processing to read change points in the data read in step S201 (step S202). That is, the main control unit 70 creates binarized data that specifies the points where significant changes in values ​​have occurred, as shown in Fig. 8(A). An example of this will be described in detail later with reference to Fig. 10.

[0070] Next, the main control unit 70 calculates a correlation function based on the binarized data in step S202 (step S203). Furthermore, the main control unit 70 calculates the peak position of the calculated correlation function (step S204) and outputs it (step S205). This output result corresponds to the degree of deviation of the time information to be obtained, that is, the amount of correction for the time deviation. Note that an example of the calculation of the correlation function using the binarized data and the extraction of the peak position of the correlation function will be described later with reference to FIG. 11.

[0071] When the main control unit 70 has completed the processing from step S201 to step S205, it checks whether or not processing has been completed for all combinations of cameras 10, that is, if camera 1 is used as the reference as described above, whether processing has been completed for all combinations of camera 1 and other cameras (step S206). If processing has not been completed (step S206: No), the main control unit 70 performs similar processing for the new combination.

[0072] On the other hand, if it is confirmed in step S206 that the processing has been completed for all of the target combinations (step S206: Yes), the main control unit 70 ends the operation as the time correction unit 72.

[0073] An example of binarization of data, i.e., the creation of the above-mentioned binarized data, will be described below with reference to the graph in Fig. 10. In the figure, the upper part α shows the graph illustrated in Fig. 8(A), i.e., the change in the distance d between the images of two cameras 10 (e.g., cameras 1 and 2), and the lower part β shows an example of binarization of the upper part α. Note that in the lower part β, the horizontal axis is time, as in the upper part α, and the vertical axis is the binarized value, i.e., a value expressed as 0 or 1. Also, the curve R1 shown by a solid line in the lower part β corresponds to the curve P1 shown in the upper part α, and the curve R2 shown by a dashed line in the lower part β corresponds to the curve P2.

[0074] Here, as shown in the upper row α of the figure, a threshold is set in advance to determine the change point in the distance between images as an index, and the time when the threshold is exceeded is set as the change point. The binarized data shown in the lower row β has a value of 1 when the distance falls below the threshold, and is 0 otherwise.

[0075] As described above, for the binarized data calculated for the two cameras 10 (cameras 1 and 2), a correlation function as shown in Fig. 11 is calculated. Specifically, when the data corresponding to the curves R1 and R2 in the binarized data shown in Fig. 10 are p1(t) and p2(t), the correlation function R(s) is expressed as follows: If we define it as TIFF0007609629000003.tif12168, the correlation function R(s) will be graphed as shown in Figure 11. This correlation function R(s) reaches its maximum value at the point where there is a time lag between the two cameras 1 and 2. In other words, in the figure, the value T of s when the correlation function R(s) reaches its peak is the time lag to be found.

[0076] In the above, the correlation function R(s) is a function that indicates the correlation between the time change points of the image information GI from the two cameras 1 and 2, for the time change points determined from a threshold value previously set for the distance between images, which is an index showing the state of the shooting environment. In the above case, the time change points are indicated by binary data (binary data change points) for the image information based on the threshold value, and the correlation function R(s) is calculated from the binary data. This enables simple, fast, and accurate processing.

[0077] As mentioned above, by similarly calculating the time shift based on the correlation function as described above for all combinations of the reference camera 1 and other cameras, the degree of time shift in the other cameras relative to the reference camera 1 can be calculated and corrections can be made based on this, and ultimately temporal synchronization can be established for all of the acquired image information.

[0078] In addition, for the other index, average luminance m, it is possible to calculate the time lag by performing a similar binarization process and calculation based on a correlation function for the graph shown in FIG. 8(B).

[0079] In addition, regarding the calculation of the time shift, it is possible to use both the distance d between the images and the average brightness m. However, it is also possible to select the most appropriate one each time, taking into account, for example, the running section of the train TR.

[0080] As described above, the image processing device 50 according to this embodiment includes an image information acquisition unit 60 that acquires a plurality of pieces of image information GI in which time information TI is added to image data GD captured at different positions, and a time correction unit 72 that corrects a time shift from a correlation function CF between the pieces of image information GI based on an index IX indicating the state of the shooting environment. In the image processing device 50 configured as described above, the image information acquisition unit 60 can acquire image data GD captured separately and independently at different positions, and the time correction unit 72 can correct a time shift (time difference) from a correlation function CF between the pieces of image information GI in which time information TI is added to image data GD based on an index IX indicating the state of the shooting environment. In other words, a process equivalent to time synchronization at the time of shooting between the acquired plurality of pieces of image data GD can be performed after the fact.

[0081] The inspection system 100 according to the present embodiment includes an image processing device 50, and includes cameras 10 as imaging units that generate image information GI by adding time information TI to image data GD obtained by imaging an inspection target facility (for example, train facility TE as inspection target CO) from different positions, and an image information output unit 30 that outputs the image information GI generated by the multiple cameras 10 to the image processing device 50. In this case, the image processing device 50 can correct a time lag (time difference) for the image information GI generated by the multiple cameras 10 and output by the image information output unit 30 to the image processing device 50.

[0082] 〔others〕 The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit and scope of the present invention.

[0083] First, in the example described above, one camera 10 (e.g., camera 1) is used as a reference and the degree of time shift of other cameras is calculated, but this is not limited thereto. For example, without setting a reference, calculations may be made using a correlation function for all camera combinations, and the time shift may be corrected based on these results.

[0084] Furthermore, in the above, two indices are used, one relating to the distance between images and the other relating to the brightness of the images, but it is also possible to adopt a mode in which only one of the two is used as the indices.

[0085] Furthermore, in the above, the correlation function is calculated from binarized data, but it is also possible to calculate the correlation function from non-binarized data (for example, data in the form of curves P1 and P2 shown in FIG. 8).

[0086] Further, in the above, near-infrared light is used in the camera 10 and the lighting 11, but this is not limiting, and for example, visible light may be used.

[0087] In addition, although the above embodiment is described as being mounted on a train TR, it is also possible to mount the camera on other moving bodies, and for example, it is possible to mount the camera on one or more vehicles to perform inspections of road facilities. [Explanation of symbols]

[0088] 10...camera, 11...lighting, 12...image information, 20...control unit, 21...operation unit, 22...external storage medium, 30...image information output unit, 40...image information generating device, 40A...train roof equipment, 40B...train interior equipment, 50...image processing device, 60...image information acquisition unit, 70...main control unit, 71...image analysis unit, 72...time correction unit, 80...memory unit, 100...inspection system, CF...correlation function, CO...inspection target, CV...central server, D1 ...direction of travel, DS...display unit, G,G1,G2,G3...image, GD...image data, GI...image information, GP...image processing unit, IX...index, ME...storage unit, MP...control unit, OU...operation unit, P1,P2,Q1,Q2,R1,R2...curve, R(s)...correlation function, SG...synthetic image, T...value, TA...time assignment unit, TE...train equipment, TI...time information, TR...train, d...distance (index), m...average brightness (index), α...upper row, β...lower row

Claims

1. An image information acquisition unit that acquires multiple pieces of image information, each of which is image data of an inspection target facility captured from different positions and time information added thereto, from outside the device; a time correction unit that corrects a time difference from a correlation function between the image information based on an index indicating a state of the shooting environment; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the index indicating the state of the photographing environment includes a distance defined between successively photographed images.

3. The image processing device according to claim 1 , wherein the index indicating the state of the photographing environment includes a brightness of the image.

4. The image processing device according to any one of claims 1 to 3, wherein the correlation function is a function that indicates a correlation between time change points related to the image information, the time change points being determined from a threshold value that is preset for an index indicating the state of the shooting environment.

5. the change point in time is binarized data of the image information based on the threshold value, The image processing device according to claim 4 , wherein the correlation function is calculated from the binarized data.

6. An inspection system comprising the image processing device according to any one of claims 1 to 5, A plurality of imaging units that generate the image information by adding the time information to the image data obtained by imaging the inspection target equipment from different positions; an image information output unit that outputs the image information generated by the plurality of imaging units to the image processing device; An inspection system equipped with:

Images