Image processing device, image processing method, and program

Abstract

To provide an image processing apparatus for accurately determining an anomaly even if a target to be imaged is moving at a high speed.SOLUTION: An image processing apparatus is configured to: identify correspondence between identical feature points in a base image and an evaluation image, the base image being obtained by imaging a moving body moving in a predetermined one moving direction, the evaluation image being obtained by imaging the moving body at a timing different from a timing at which the base image is captured; specify clip section in the evaluation image corresponding to one section obtained by segmenting the base image in the moving direction at predetermined intervals, based on the change in difference of positions of the identical feature points having the correspondence and appearing in the base image and the evaluation image; and generate a new evaluation image in which images of the clip section are sequentially arranged in the moving direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for comparing a reference image of an object with a newly captured image of the object, and determining whether there is an abnormality in the object captured in the newly captured image.

[0003] In Patent Document 1, a method is described in which a comparison reference image and an image on an inspection date are used to extract a plurality of feature points from each image, and the feature points of one image are associated with those of the other image, and the associated feature points are compared with the image. between It is shown that the other image is projectively changed so that the deviation of the feature points of the first image is minimized (Patent Document 1, paragraphs 0047 to 0051, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-3574 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, in the technology for determining abnormalities in an object by comparing a reference image with an image taken at a different time, there is a demand for technology that can accurately determine abnormalities even when the object being photographed is moving at high speed.

[0006] An object of this disclosure is to provide an image processing device, an image processing method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0007] According to a first aspect of this disclosure, an image processing device includes a correspondence relationship identification means for identifying the correspondence relationship between identical feature points in a basic image captured of a moving object moving in a predetermined movement direction and an evaluation image captured of the moving object at a timing different from the timing of capturing the basic image; a section identification means for identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position in the movement direction of the identical feature point captured in the basic image and the evaluation image; and a new evaluation image generation means for generating a new evaluation image in which images of the cut-out section are arranged sequentially in the movement direction.

[0008] According to a second aspect of this disclosure, an image processing method identifies correspondences between identical feature points in a basic image captured of a moving object moving in a predetermined direction of movement and an evaluation image captured of the moving object at a timing different from the timing of capturing the basic image, identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the direction of movement based on changes in the difference in position in the direction of movement of the identical feature points captured in the basic image and the evaluation image, and generates a new evaluation image in which images of the cut-out section are arranged sequentially in the direction of movement.

[0009] According to a third aspect of this disclosure, the program causes a computer of an image processing device to function as: a correspondence relationship identification means for identifying the correspondence relationship between identical feature points in a basic image captured of a moving object moving in a predetermined movement direction and an evaluation image captured of the moving object at a timing different from the timing of capturing the basic image; a section identification means for identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position in the movement direction of the identical feature points captured in the basic image and the evaluation image; and a new evaluation image generation means for generating a new evaluation image in which images of the cut-out section are arranged sequentially in the movement direction. [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 1 is a first diagram illustrating a configuration of an image processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a hardware configuration diagram of an image processing device according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a functional block diagram of an image processing device according to an embodiment of the present disclosure. [Figure 4] 1A and 1B are diagrams illustrating examples of a base image and an evaluation image according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a diagram illustrating a processing flow of an image processing device according to an embodiment of the present disclosure. [Figure 6] FIG. 10 is a diagram illustrating a relationship between feature points in a base image and an evaluation image according to an embodiment of the present disclosure. [Figure 7] FIG. 10 is a first diagram illustrating an example of a correspondence relationship specifying process according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a second diagram illustrating an example of a correspondence relationship specifying process according to an embodiment of the present disclosure. [Figure 9] FIG. 10 is a diagram illustrating an overview of a correspondence removal process according to an embodiment of the present disclosure. [Figure 10] FIG. 1 is a first diagram illustrating an overview of a section identification process according to an embodiment of the present disclosure. [Figure 11] FIG. 10 is a second diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure. [Figure 12] FIG. 10 is a third diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure. [Figure 13] FIG. 10 is a diagram illustrating another configuration of an image processing device according to an embodiment of the present disclosure. [Figure 14] FIG. 10 is a diagram illustrating another processing flow of the image processing device according to an embodiment of the present disclosure. [Figure 15] FIG. 10 is a diagram illustrating another configuration of an image processing device according to an embodiment of the present disclosure. [Figure 16] FIG. 10 is a diagram illustrating another processing flow of the image processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] FIG. 1 is a first diagram showing the configuration of an image processing system according to an embodiment of the present disclosure. The image processing system 100 is configured by a communication connection between an image processing device 1 and a line scan camera 2. The line scan camera 2 is installed so as to capture an image of a moving object 5 moving in a predetermined direction. In FIG. 1, the line scan camera 2 is installed in a fixed position and captures an image of the moving object 5 moving to the right. The moving object 5 may be, for example, a train. The line scan camera 2 repeatedly captures images of the moving object 5 at predetermined short time intervals. With each capture, the image captured by the line scan camera 2 becomes a long, narrow image. The line scan camera 2 sequentially transmits the images generated by the capture to the image processing device 1. By using the line scan camera 2, it is possible to capture the entire moving object 5, which is long in the direction of movement.

[0012] The image processing device 1 synthesizes images acquired from the line scan camera 2 to generate an image capturing a predetermined range of the moving object 5. The image of the predetermined range may be the entire moving object 5, or if the moving object 5 is a train with multiple cars, it may be one car or a predetermined number of cars of the train. The image processing device 1 may compare each image (in this disclosure, a basic image and an evaluation image) generated each time the same moving object passes in front of the line scan camera 2 and output the comparison result. The comparison result may be the presence or absence of an abnormality in the moving object 5, etc.

[0013] Here, the image processing device 1 generates one of the images each time the same moving object 5 passes in front of the line scan camera 2 as a base image and the other as an evaluation image. The image processing device 1 may determine whether there is a difference (such as an abnormality) between the evaluation image and the base image. The base image may be an image taken when the moving object 5 is normal, and the evaluation image may be an image taken when the moving object 5 is inspected after a period of time has passed.

[0014] In the present disclosure, the image processing device 1 performs the following processing to generate a base image and an evaluation image. Specifically, the image processing device 1 identifies the correspondence between the same feature points in the base image and the evaluation image. Based on changes in the difference in position between the same feature points that correspond to each other in the base image and the evaluation image, the image processing device 1 identifies clipped sections in the evaluation image that correspond to sections obtained by dividing the base image at predetermined intervals in the movement direction. The image processing device 1 reduces or enlarges each clipped section in the evaluation image so that each clipped section in the evaluation image matches the length of an image in each section of the base image. A new evaluation image is generated by arranging the images of each section after the clipped sections have been reduced or enlarged in order in the movement direction.

[0015] FIG. 2 is a hardware configuration diagram of an image processing device according to an embodiment of the present disclosure. As shown in FIG. 2, the image processing device 1 is a computer equipped with various hardware components, such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, other storage devices 104, and a communication module 105.

[0016] FIG. 3 is a functional block diagram of an image processing device according to an embodiment of the present disclosure. The CPU 101 of the image processing device 1 executes an image processing program, which causes the image processing device 1 to perform the functions of an acquisition unit 11, an image generation unit 12, a correspondence relationship identification unit 13, a correspondence relationship removal unit 14, a section identification unit 15, an image conversion unit 16, a new evaluation image generation unit 17, and an anomaly detection unit 18.

[0017] The acquisition unit 11 acquires image data from the line scan camera 2 . The image generating unit 12 generates a base image and an evaluation image. The correspondence specifying unit 13 specifies the correspondence between the same feature points in the base image and the evaluation image. The correspondence removal unit 14 identifies identical feature points that have a correspondence relationship between the base image and the evaluation image and that result in an outlier due to a change in the difference in position of the identical feature points, and removes the correspondence relationship for the identical feature points from the correspondence identification result of the correspondence identification unit 13. The section identification unit 15 identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the direction of movement of the moving object, based on the change in the difference in position of the same feature point that has a corresponding relationship between the basic image and the evaluation image. The image conversion unit 16 reduces or enlarges each of the cut-out sections in the evaluation image so that the length of each cut-out section in the evaluation image matches the length of one section of each of the basic images. The new evaluation image generating unit 17 generates a new evaluation image by sequentially arranging the images of each section after reducing or enlarging the cut-out section in the moving direction of the moving object. The anomaly detection unit 18 compares the base image with the new evaluation image to detect anomalies in the moving object.

[0018] The image processing device 1 may be configured such that each of the above functions is performed by a single computer, or such that multiple computers perform the above functions together. When multiple computers form the image processing device 1, any one of the computers performs any of the above functions, so that the functions are performed as a whole. The function of the anomaly detection unit 18 does not have to be included in the image processing device 1. The present disclosure may be capable of generating a new evaluation image to be compared with a basic image based on at least the evaluation image generated by the image generation unit 12.

[0019] FIG. 4 is a diagram showing an example of a base image and an evaluation image according to one embodiment of the present disclosure. The base image and evaluation image are images generated at different times. The base image is an image generated by combining the image data captured by the line scan camera 2. Similarly, the evaluation image is also an image generated by combining the image data captured by the line scan camera 2.

[0020] The line scan camera 2 is fixedly installed at a position where it can capture the moving object 5, and captures the moving object 5 by releasing the shutter at high speed as the moving object 5 passes by the line scan camera 2. As a result, the multiple pieces of captured data output by the line scan camera 2 are image data that include a portion of the moving direction of the moving object 5. By sequentially combining these pieces of captured data so that they match the moving direction of the moving object 5 captured in the image, it is possible to generate a basic image and an evaluation image that include the entire moving direction of the moving object 5. The moving speed and change in moving speed of the moving object 5 when the captured data used to generate the basic image may be different from the moving speed and change in moving speed of the moving object 5 when the captured data used to generate the evaluation image are captured. Furthermore, the shutter speeds when capturing the captured data used to generate the basic image and the evaluation image may be different. As a result, the lengths of the basic image and the evaluation image, which capture the entire moving direction of the moving object 5, are different, as shown in FIG. 4.

[0021] FIG. 5 is a diagram showing a processing flow of an image processing device according to an embodiment of the present disclosure. Next, the processing flow of the image processing device 1 will be explained step by step. The acquisition unit 11 outputs multiple image data acquired from the line scan camera 2 to the image generation unit 12 at the timing of generating a basic image. The image generation unit 12 synthesizes each piece of shooting data to generate a basic image (step S101). Similarly, the acquisition unit 12 outputs multiple image data acquired from the line scan camera 2 to the image generation unit 12 at the timing of generating an evaluation image. The image generation unit 12 synthesizes each piece of shooting data to generate an evaluation image (step S102). As a result, a basic image and an evaluation image as shown in FIG. 4 are generated. Note that since the basic image and the evaluation image are generated at different times, they may be stored in a storage unit or the like.

[0022] The correspondence specifying unit 13 detects the timing of generating a new evaluation image (step S103). For example, the correspondence specifying unit 13 may detect the timing of generating a new evaluation image based on a user operation. When the correspondence specifying unit 13 detects the timing of generating a new evaluation image, it acquires the basic image and evaluation image generated by the image generating unit 12 from the storage unit or the image generating unit 12.

[0023] The correspondence identification unit 13 detects feature points of the moving object 5 that appear in the basic image (step S104). Feature points indicate points where edges, such as the corners of a specific part of the moving object 5 or the corners of screws, stand out. The correspondence identification unit 13 extracts the boundaries between the parts that make up the moving object 5 that appear in the basic image and other adjacent parts or objects using edge processing, and detects feature points where the edges stand out. The feature points may be part of a specific part that makes up the moving object 5. In this case, the correspondence identification unit 13 may identify the specific part using pattern matching and identify a part of that part as the feature point. The correspondence identification unit 13 identifies multiple feature points throughout the entire basic image. The more feature points, the better. For example, the correspondence identification unit 13 identifies multiple feature points within a section that is set in advance in the basic image. The correspondence identification unit 13 similarly identifies feature points in the evaluation image (step S105).

[0024] In identifying feature points, the correspondence identification unit 13 may identify feature points that do not have similar objects within a predetermined range nearby. This reduces erroneous determinations in the subsequent matching process for identical feature points between the base image and the evaluation image. In other words, this eliminates the difficulty of identifying the correspondence between feature points in the base image and feature points in the evaluation image when there are many similar objects nearby. This process is an example of a process in which the correspondence identification unit 13 determines whether there are any similar objects to a specific object within a predetermined range including the specific object, and if there are no similar objects, detects the specific object as a candidate for identical feature points.

[0025] The correspondence specifying unit 13 compares the feature points specified in the base image with the feature points specified in the evaluation image to perform matching of identical feature points (step S106). Specifically, if the degree of coincidence of the edge patterns indicated by the feature points is equal to or greater than a predetermined threshold, the feature points are determined to be identical feature points, and the correspondence between the feature points in the base image and the feature points in the evaluation image is specified.

[0026] FIG. 6 is a diagram showing the relationship between feature points in a base image and an evaluation image according to an embodiment of the present disclosure. In both images 6A and 6B shown in FIG. 6, the top row shows the base image 51 and the bottom row shows the evaluation image 52. When lines are drawn to connect the feature points in the base image 51 with the corresponding feature points in the evaluation image 52, the lines appear as shown in FIG. 6. Image 6B shows a case in which more feature points are identified than in image 6A. For convenience of explanation, FIG. 6 uses an example in which the lengths of the base image 51 and the evaluation image 52 are the same. However, in this disclosure, the lengths of the base image 51 and the evaluation image 52 are different, as shown in FIG. 4. However, in other processes, the lengths of the base image 51 and the evaluation image 52 may be matched first. When the lengths of the base image and the evaluation image are equal or nearly equal, as shown in FIG. 6, and the moving speed of the moving object 5 in the base image and the moving speed of the moving object 5 in the evaluation image are the same, the line connecting the feature point in the base image and the corresponding feature point in the evaluation image will be perpendicular to the upper limit. On the other hand, if the moving speed or change in moving speed of the moving object 5 in the base image and the evaluation image are different, the corresponding identical feature points will be shifted in the direction of movement of the moving object 5, as shown in Figure 6, and the line connecting those feature points will be diagonal.

[0027] Each time a new evaluation image is generated, the correspondence relationship identification unit 13 identifies feature points in each of the base image and the evaluation image (the most recently generated new evaluation image) and performs matching of identical feature points. If feature points in the base image are identified in advance and an attempt is made to find identical feature points in the evaluation image that correspond to those feature points, it will be impossible to find the corresponding feature points in the evaluation image if the identical feature points are not clearly visible in the evaluation image due to the influence of shadows or the like. Therefore, by identifying feature points in each of the base image and the evaluation image each time a new evaluation image is generated, it becomes possible to identify identical feature points from the base image and the evaluation image in a manner that is robust to changes in shooting conditions.

[0028] In some cases, the correspondence between the feature points in the base image and the evaluation image cannot be adequately identified in the correspondence identification process described above. Therefore, the image processing device 1 adds the following process to the correspondence identification process. FIG. 7 is a first diagram illustrating an example of the correspondence relationship specifying process. 7 shows four process summaries of the correspondence identification process: a first process summary 91, a second process summary 92, a third process summary 93, and a fourth process summary 94. The first process summary 91 shows how three feature points 51a, 51b, and 51c are identified in the base image 51, and three feature points 52a, 52b, and 52c are identified in the evaluation image 52. Now, in the processing of the correspondence identification unit 13, it should be determined that the feature points 51a and 52a are the same feature point, the feature points 51b and 52b are the same feature point, and the feature points 51c and 52c are the same feature point. However, it is assumed that the same feature point in the evaluation image corresponding to the feature point 51b is not detected, and the relationship between the feature points 51a and 52a and the relationship between the feature points 51a and 52b are both determined to be the same feature point.

[0029] That is, the first processing outline 91 indicates that the feature point 51a of the base image has been identified as having a relationship of the same feature point with each of the two feature points 52a and 52c of the evaluation image. In this case, the relationship between the feature point 51a and the feature point 52a is a relationship of the same feature point, but the relationship between the feature point 51a and the feature point 52b is not a correct correspondence relationship of the same feature point, and therefore an error has occurred in the identification. If a similar feature point exists in a position near the moving object 5, errors may occur repeatedly in the identification of such correspondence relationships of the same feature points.

[0030] In this case, as shown in the second processing outline 92, the correspondence identification unit 13 identifies the range in which errors occur consecutively when identifying the correspondence between the same feature points in each of the basic image 51 and the evaluation image 52. The range in which the identified errors occur consecutively in the basic image 51 is set to 51x, and the range in which the identified errors occur consecutively in the evaluation image 52 is set to 52x.

[0031] As shown in a third processing outline 93, the correspondence identification unit 13 identifies a center point 51p of a range 51x where the identified errors continuously occur in the base image 51, and a center point 52p of a range 52x where the identified errors continuously occur in the evaluation image 52. When the correspondence identification unit 13 sets a range where errors continuously occur, as shown in a fourth processing outline 94, the correspondence identification unit 13 provisionally identifies center points 51p and 52p of each range within that range as points having a correspondence relationship of the same feature point, and generates correspondence data. Thereafter, the correspondence data is used to perform the outlier removal process by the correspondence removal unit 14 described above.

[0032] The processing of the correspondence identification unit 13 is an example of processing in which a range in each of the basic image and the evaluation image where errors occur in identifying the correspondence between the same feature points in the basic image and the evaluation image are identified, and a predetermined position in each of the identified ranges in the basic image and the evaluation image is identified as the correspondence between the same feature points.

[0033] According to the above processing, by avoiding as much as possible the generation of correspondence data that includes an error in the correspondence between the same feature points in the base image and the evaluation image, it is possible to prevent the occurrence of feature points that are not identified as outliers in the outlier removal processing in the correspondence removal unit 14 described below.

[0034] The range where errors occur consecutively may be specified by the image processing device 1 outputting a display screen showing the relationship between identical feature points on the base image and the evaluation image by superimposing lines, and receiving a user's specification of the range where errors occur consecutively on the display. Alternatively, the correspondence specifying unit 13 of the image processing device 1 may automatically specify the range where errors in the correspondence between identical feature points occur consecutively in each of the base image and the evaluation image.

[0035] FIG. 8 is a second diagram illustrating an example of the correspondence relationship specifying process. Next, an example will be described in which the image processing device 1 automatically identifies the range in which errors in the correspondence between the same feature points occur consecutively in each of the base image and the evaluation image. The image processing device 1 may perform the following processing in the correspondence identification process.

[0036] 8 shows four processing summaries: a first automatic processing summary 111, a second automatic processing summary 112, a third automatic processing summary 113, and a fourth automatic processing summary 114. The first automatic processing summary 111 shows how three feature points 51a, 51b, and 51c are identified in a base image 51, and three feature points 52a, 52b, and 52c are identified in an evaluation image 52. Now, in the processing of the correspondence identification unit 13, it is supposed that the feature points 51a and 52a should be determined to be the same feature point, the feature points 51b and 52b should be the same feature point, and the feature points 51c and 52c should be the same feature point. However, it is assumed that the same feature points in the evaluation image corresponding to the feature points 51b and 51c are not detected, and the relationship between the feature points 51a and 52a, the relationship between the feature points 51a and 52b, and the relationship between the feature points 51a and 52c are determined to be the same feature point.

[0037] That is, in the example of the first automatic processing outline 111, feature point 51a of the base image is identified as having the same feature point relationship as each of three feature points 52a, 52b, and 52c of the evaluation image. In this case, the relationship between feature point 51a and feature point 52a is the same feature point relationship, but the relationship between feature point 51a and feature point 52b and the relationship between feature point 51a and feature point 52c are not the same feature point correspondence relationships, and therefore an error has occurred in the identification. If similar feature points exist in positions near the moving object 5, errors may occur repeatedly in the identification of such correspondence relationships of the same feature points.

[0038] In this case, the correspondence identification unit 13 detects that multiple feature points in the evaluation image have been determined to be the same feature point as one feature point in the base image (first automatic processing overview 111). In this case, the correspondence identification unit 13 determines whether the distance between each of the multiple feature points in the evaluation image that have been determined to be the same feature point as one feature point in the original image is less than a threshold value and therefore close to each other (second automatic processing overview 112). If the distance between each of the multiple feature points in the evaluation image is less than a threshold value and therefore close to each other, the correspondence identification unit 13 identifies a range that includes those feature points 52a, 52b, and 52c (third automatic processing overview 113).

[0039] The correspondence specifying unit 13 specifies the center point of the range specified in the evaluation image as the feature point 52p in the evaluation image that corresponds to the feature point 51a in the basic image.Then, the feature point 51a in the basic image and the feature point 52p specified in the evaluation image are specified as the same feature point, and correspondence data including the relationship therebetween is generated.

[0040] The correspondence identification unit 13 may display information about a range in the evaluation image that includes feature points 52a, 52b, and 52c, allowing the user to modify that range. In this case, the correspondence identification unit 13 acquires information about the range modified by the user (fourth automatic processing outline 114). The correspondence identification unit 13 identifies the center point of that range as feature point 52p in the evaluation image that corresponds to feature point 51a in the base image.

[0041] According to the above process, the image processing device 1 can automatically identify the range in which errors in the correspondence relationship between the same feature points occur consecutively.

[0042] FIG. 9 is a diagram showing an overview of the correspondence removal process according to an embodiment of the present disclosure. The correspondence identification unit 13 outputs correspondence data holding multiple correspondences between identical feature points identified in the base image and the evaluation image to the correspondence removal unit 14. The correspondence data is data including multiple correspondences between the coordinates of identical feature points identified in the base image and the evaluation image in each image. The correspondence removal unit 14 reads the coordinates of the identical feature points in the base image and the evaluation image included in the correspondence data, and calculates the difference (xDiff) in the coordinates in the movement direction of the moving object 5 in the image (x-axis direction in this disclosure). The correspondence removal unit 14 calculates the difference (xDiff) in the x-coordinate in the movement direction for each correspondence of identical feature points. The correspondence identification unit 13 identifies outliers with a steep change in the difference (xDiff) between the x-coordinate value of the feature point identified in the base image and the x-coordinate of the identical feature point in the corresponding evaluation image (step S107).

[0043] Here, the moving object 5 being photographed is moving in the real world, but due to the constraints of inertial force, it is assumed that the speed and acceleration of the moving object 5 can be approximated as being almost constant over an extremely short period of time. In other words, it is assumed that the speed of the moving object 5 changes linearly within a certain short period of time. The correspondence removal unit 14 utilizes this to remove outliers by linear approximation (step S108).

[0044] In graph 7A of FIG. 9, the correspondence removal unit 14 displays a two-dimensional graph showing the relationship between the x-coordinate value of a feature point identified in the base image and the difference (y-axis) between the x-coordinate of the same feature point in the corresponding evaluation image. For example, in graph 7A of FIG. 9, p1 and p2 are outliers. The correspondence identification unit 13 identifies such outliers using a known technique such as the RANSAC (RANdom SAmple Consensus) algorithm. This generates correspondence data (7B) from which the outliers have been removed. This process is an example of the process in which the correspondence removal unit 14 identifies identical feature points that have a correspondence relationship between the base image and the evaluation image, where changes in the difference in position between the identical feature points are outliers, and removes the correspondence between the identical feature points.

[0045] There is a possibility that information that has been mistakenly identified as the same feature point may be included in the result of identifying the correspondence of the same feature point by the correspondence identification unit 13. If such erroneous information is left, it will cause problems in the calculations of the subsequent processing by the section identification unit 15 and the image conversion unit 16, and therefore the processing by the correspondence removal unit 14 is required.

[0046] FIG. 10 is a first diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure. FIG. 11 is a second diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure. FIG. 12 is a third diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure. The correspondence removal unit 14 outputs the correspondence data from which the outliers have been removed to the section identification unit 15. Here, in the basic image, one section is predetermined to have a width of 20px (pixels) in the direction of movement of the moving object 5. The width (20px) of one section set in the basic image is determined so that image processing for one section does not take much time, depending on the shutter speed of the line scan camera 2 and the processing capacity of the image processing device 1. The section identification unit 15 identifies the correspondence of one section of the evaluation image corresponding to one section of the basic image (step S109). Now, as shown in FIG. 10, the difference in the x-coordinate direction between each of the sections in the evaluation image corresponding to each of the three sections in the basic image, each 20px long, is unknown.

[0047] Therefore, the section identification unit 15 identifies the correspondence between each section of the evaluation image corresponding to each of the multiple sections set in the basic image using the correspondence data acquired from the correspondence removal unit 14. The correspondence data 71 shown in FIG. 11 indicates the difference between the x-coordinate of a feature point identified in the basic image and the x-coordinate of a feature point identified in the evaluation image corresponding to that feature point. For example, the correspondence data 71 records that the x-coordinate value of feature point α identified in the basic image is 10.2 (px), and the difference in x-coordinate between feature point α and the corresponding feature point in the evaluation image is "-1.2." The correspondence data 71 also records that the x-coordinate value of feature point β identified in the basic image is 15.6 (px), and the difference in x-coordinate between feature point β and the corresponding feature point in the evaluation image is "-0.5." In the correspondence data 71, the x-coordinate value of the feature point γ identified in the basic image is 19.5 (px), and the difference in x-coordinate between the feature point γ and the corresponding feature point in the evaluation image is 2.2 ". Furthermore, in the correspondence data 71, the x-coordinate value of the feature point δ identified in the basic image in that basic image is recorded as 20.2 (px), and the difference in x-coordinate between that feature point δ and the corresponding feature point in the evaluation image is recorded as "2.9". Note that the numerical value of the x-coordinate is considered to correspond to the number of px (pixels). If the origin of the x-coordinate in the basic image and the evaluation image is the left end, when the difference between the feature point in the basic image and the feature point in the basic image that corresponds to that feature point is negative, the x-coordinate position of the feature point in the evaluation image is closer to the origin than the x-coordinate position of the feature point in the corresponding basic image.

[0048] Now, let's look at the evaluation image corresponding to the end of the first section of the base image (x coordinate 20.0 (px)) determined by the width of 20px. OkeruThe x coordinate is calculated using the correspondence data 71. In this case, since the feature point of the x coordinate indicating x coordinate = 20.0 (px) in the basic image is not identified in the basic image, attention is focused on feature point γ (x coordinate = 19.5) and feature point δ (x coordinate = 20.2), which are neighboring points before and after x coordinate 20.0 (px). In this case, the section identification unit 15 can calculate the virtual feature point in the evaluation image corresponding to the virtual feature point equivalent to x coordinate 20.0 in the basic image using the following linear interpolation formula.

[0049] <Linear interpolation formula> difference in x coordinates =2.2+( ( 2.9-2.2 ) ÷ ( 20.2-19.5 ) )×(20.0-19.5) = 2.7

[0050] Similarly, the section identification unit 15 calculates the x-coordinate in the evaluation image corresponding to the position of the end of the second section of the basic image (x-coordinate 40.0 (px)), the x-coordinate in the evaluation image corresponding to the position of the end of the third section (x-coordinate 60.0 (px)), etc., and then calculates the positional difference between the x-coordinate of the basic image at the end of each section and the x-coordinate of the corresponding evaluation image using a linear interpolation formula.

[0051] The section identification unit 15 generates cut-out section identification data 81 indicating, for each section, the difference between the x-coordinate of the end of the section set in the base image and the x-coordinate in the evaluation image corresponding to that x-coordinate (step S110). The section identification unit 15 outputs the cut-out section identification data 81 to the image conversion unit 16.

[0052] The image conversion unit 16 identifies the coordinates from the start to the end of one section in the evaluation image based on the cut-out section identification data 81 (step S111). 10As shown in A), the image width from the start coordinate to the end coordinate of one section in the evaluation image is reduced or enlarged to the width (20px) of one section set in the basic image to generate cut-out section data (step S112). As a result, the image width from the start coordinate to the end coordinate of one section in the evaluation image is reduced or enlarged in the direction of movement of the moving object shown in the image. As a result, the length of one section in the basic image and one section in the evaluation image match.

[0053] As shown in FIG. 6, the length of the original image of the evaluation image acquired from the line scan camera 2 may be converted in advance so that the lengths of the base image and the evaluation image match. In this case, the image conversion unit 16 converts the length of the original image of the evaluation image acquired from the line scan camera 2 in advance as shown in FIG. 10 As shown in B), the image conversion unit 16 first determines the width of the corresponding original image based on the length from the start coordinate to the end coordinate of one section in the evaluation image based on the cut-out section identification data 81 and the conversion rate when the original image of the evaluation image is adjusted to the length of the basic image. (10 B ) The width of the original image of the evaluation image identified by the above is converted to the width (20px) of one section set in the base image to generate cut-out section data (Figure 12( 10 C).

[0054] The image conversion unit 16 outputs each of the image data (cut-out section data) for one section generated from the evaluation image to the new evaluation image generation unit 17. The new evaluation image generation unit 17 generates a new evaluation image by arranging and combining the image data (cut-out section data) for each section generated from the evaluation image in order (step S113). The length of the new evaluation image in the direction of movement of the moving object 5 matches the length of the moving object 5 in the direction of movement of the basic image.

[0055] According to the above process, even if the subject photographed by the line scan camera 2 is a moving object 5, a new evaluation image can be generated that matches the length of the moving object 5 in the movement direction of the basic image.

[0056] Furthermore, according to the above-described processing, it is possible to generate a new evaluation image in which the same feature points appear at positions in the direction of movement that roughly correspond to the feature points of the basic image, even if the subject, ie, the moving object 5, moves at high speed and the speed of the moving object 5 changes as it passes by the line scan camera 2. This makes it easy to compare and evaluate two images generated based on images taken at different times by the line scan camera 2, because the same feature points will be in roughly the same positions.

[0057] Furthermore, according to the above-described processing, even if the moving object 5 is long in the direction of movement, such as a train, and the evaluation image becomes longer in the direction of movement depending on the shutter speed, it is possible to generate a new evaluation image that matches the length of the basic image. Therefore, when performing evaluation such as detecting abnormal points in the entire new evaluation image, the number of image processes for each section obtained by dividing the new evaluation image into sections of a predetermined size can be reduced, thereby reducing the labor required for such image processing.

[0058] Furthermore, the above-described processing identifies many relationships between the same feature points in the base image and the evaluation image, so that even if the change in speed of the moving object 5 when photographed to generate the base image does not match the change in speed of the moving object 5 when photographed to generate the evaluation image, it is possible to generate a new evaluation image in which the same feature points appear at approximately the same positions in the direction of movement.

[0059] FIG. 13 is a diagram showing another configuration of an image processing device according to an embodiment of the present disclosure. FIG. 14 is a diagram showing another processing flow of the image processing device according to an embodiment of the present disclosure. The image processing device 1 shown in FIG. 13 performs the functions of a correspondence relationship specifying means, a section specifying means, an image conversion means, and a new evaluation image generating means. The correspondence determination means determines the correspondence between the same feature points in a basic image captured of a moving object moving in a predetermined direction of movement and an evaluation image captured of the moving object at a timing different from the timing of capturing the basic image (step S1301). The section identification means identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on the change in the difference in position of the same feature point that has a corresponding relationship between the basic image and the evaluation image (step S1302). The image conversion means converts each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having a length of one section in each of the basic images (step S1303). The new evaluation image generating means generates a new evaluation image by arranging the images of each section after the conversion of the cut-out sections in order in the direction of movement of the moving object shown in the images (step S1304).

[0060] FIG. 15 is a diagram showing another configuration of an image processing device according to an embodiment of the present disclosure. FIG. 16 is a diagram showing another processing flow of the image processing device according to an embodiment of the present disclosure. The image processing device 1 shown in FIG. 15 performs the functions of a correspondence relationship specifying means, a section specifying means, and a new evaluation image generating means. The correspondence determination means determines the correspondence between the same feature points in a basic image captured of a moving object moving in a predetermined direction of movement and an evaluation image captured of the moving object at a timing different from the timing of capturing the basic image (step S1501). The section identification means identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on the change in the difference in position of the same feature point that has a corresponding relationship between the basic image and the evaluation image (step S1502). The new evaluation image generating means generates a new evaluation image by sequentially arranging the images of the cutout section in the movement direction (step S1503).

[0061] The image processing device 1 of this disclosure has been described above, but this disclosure is not limited to the above-described embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of this disclosure within the scope of this disclosure.

[0062] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0063] (Appendix 1) a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image; a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having the length of one section in each of the basic images; a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of each of the cut-out sections after reduction or enlargement in the movement direction; An image processing device comprising:

[0064] (Appendix 2) a correspondence removal means for identifying identical feature points whose change in the difference in position of the identical feature points in the movement direction between the basic image and the evaluation image is an outlier, and removing the correspondence between the identical feature points; The section specifying means specifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in a difference in position of the same feature point in the movement direction between the basic image and the evaluation image after the removal. 2. The image processing device according to claim 1.

[0065] (Appendix 3) The correspondence specifying means specifies, in each of the basic image and the evaluation image, a range in which errors occur in specifying the correspondence between the same feature points in the basic image and the evaluation image, and specifies predetermined positions in each of the specified ranges in the basic image and the evaluation image as the correspondence between the same feature points. 3. The image processing device according to claim 2.

[0066] (Appendix 4) an image generating means for generating the basic image and the evaluation image by combining images of the moving object captured by a line scan camera; 2. The image processing device according to claim 1, comprising:

[0067] (Appendix 5) The correspondence relationship specifying means specifies a plurality of correspondence relationships between the same feature points in the basic image and the evaluation image in the movement direction. 2. The image processing device according to claim 1.

[0068] (Appendix 6) The correspondence specifying means detects predetermined objects in each of the base image and the evaluation image as candidates for the same feature point. 6. The image processing device according to claim 5.

[0069] (Appendix 7) The correspondence specifying means determines whether or not there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, detects the predetermined object as a candidate for the same feature point. 7. The image processing device according to claim 6.

[0070] (Appendix 8) an abnormality detection means for detecting an abnormality in the moving object by comparing the basic image with the new evaluation image; 2. The image processing device according to claim 1, comprising:

[0071] (Appendix 9) Identifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined one moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image; identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; shrinking or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having a length of one section in each of the basic images; A new evaluation image is generated by sequentially arranging the images of each section after the reduction or enlargement of the cut-out section in the movement direction. Image processing methods.

[0072] (Appendix 10) identifying an identical feature point whose change in the difference in position of the identical feature point in the movement direction between the basic image and the evaluation image is an outlier, and removing the correspondence relationship for the identical feature point; Based on a change in the difference in the position of the same feature point in the movement direction between the basic image and the evaluation image after the removal, a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction is identified. 10. The image processing method according to claim 9.

[0073] (Appendix 11) A range in which errors occur continuously in identifying the correspondence between the same feature points in the basic image and the evaluation image is identified in each of the basic image and the evaluation image, and predetermined positions in each of the identified ranges in the basic image and the evaluation image are identified as the correspondence between the same feature points. 11. The image processing method according to claim 9 or 10.

[0074] (Appendix 12) The images of the moving object captured by the line scan camera are combined to generate the base image and the evaluation image. 12. The image processing method according to any one of Supplementary Note 9 to Supplementary Note 11, comprising:

[0075] (Appendix 13) Identifying a plurality of correspondences between the same feature points in the basic image and the evaluation image in the movement direction. 13. An image processing method according to any one of claims 9 to 12.

[0076] (Appendix 14) Detecting predetermined objects in each of the base image and the evaluation image as candidates for the same feature point. 14. An image processing method according to any one of claims 9 to 13.

[0077] (Appendix 15) It is determined whether there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, the predetermined object is detected as a candidate for the same feature point. 15. An image processing method according to any one of claims 9 to 14.

[0078] (Appendix 16) The base image is compared with the new evaluation image to detect abnormalities in the moving object. 16. An image processing method according to any one of claims 9 to 15.

[0079] (Appendix 17) The computer of the image processing device, a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from that of photographing the basic image; a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having the length of one section in each of the basic images; a new evaluation image generating means for generating a new evaluation image by sequentially arranging images of each section after the reduction or enlargement of the cut-out section in the movement direction; A program that functions as a

[0080] (Appendix 18) A computer of the image processing device, identifying identical feature points whose change in the difference in position of the identical feature points in the movement direction between the basic image and the evaluation image is an outlier, and causing the device to function as correspondence relationship removal means for removing the correspondence relationship between the identical feature points; The section specifying means specifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in a difference in position of the same feature point in the movement direction between the basic image and the evaluation image after the removal. 17. The program described in Appendix 17.

[0081] (Appendix 19) The correspondence specifying means specifies, in each of the basic image and the evaluation image, a range in which errors occur in specifying the correspondence between the same feature points in the basic image and the evaluation image, and specifies predetermined positions in each of the specified ranges in the basic image and the evaluation image as the correspondence between the same feature points. 17. The program of claim 16.

[0082] (Appendix 20) A computer of the image processing device, an image generating means for generating the basic image and the evaluation image by combining images of the moving object captured by a line scan camera; 19. The program according to claim 17, wherein the program functions as follows:

[0083] (Appendix 21) The correspondence relationship specifying means specifies a plurality of correspondence relationships between the same feature points in the basic image and the evaluation image in the movement direction. 21. A program according to any one of claims 17 to 20.

[0084] (Appendix 22) The correspondence specifying means detects predetermined objects in each of the base image and the evaluation image as candidates for the same feature point. 22. A program according to any one of claims 17 to 21.

[0085] (Appendix 23) The correspondence specifying means determines whether or not there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, detects the predetermined object as a candidate for the same feature point. 23. A program according to any one of claims 17 to 22.

[0086] (Appendix 24) A computer of the image processing device, 24. The program according to claim 17, which functions as an abnormality detection means for detecting an abnormality in the moving object by comparing the basic image with the new evaluation image.

[0087] (Appendix 25) a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image; a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of the cutout section in the movement direction; An image processing device comprising:

[0088] (Appendix 26) an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having the length of one section in each of the basic images; The new evaluation image generating means generates a new evaluation image by sequentially arranging images of each of the sections after the reduction or enlargement of the cut-out sections in the movement direction. 26. The image processing device of claim 25.

[0089] (Appendix 27) Identifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined one moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image; identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; A new evaluation image is generated by sequentially arranging the images in the cutout section in the direction of movement. An image processing method comprising:

[0090] (Appendix 28) The computer of the image processing device, a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from that of photographing the basic image; a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of the cutout section in the movement direction; A program that functions as a [Explanation of symbols]

[0091] 1. Image processing device 2. Line scan camera 5. Mobile 11...Acquisition part 12. Image generation unit 13. Correspondence identification unit 14. Correspondence removal part 15 Section identification section 16. Image conversion section 17. New evaluation image generation unit 18. Abnormality detection unit

Claims

1. a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image; a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that the image length of each of the cut-out sections in the evaluation image matches the image length of the corresponding section in the basic image; a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of each of the cut-out sections after reduction or enlargement in the movement direction; An image processing device comprising:

2. a correspondence removal means for identifying identical feature points whose change in the difference in position of the identical feature points in the movement direction between the basic image and the evaluation image is an outlier, and removing the correspondence between the identical feature points; The section specifying means specifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in a difference in position in the movement direction of the same feature point appearing in the basic image and the evaluation image after the removal. The image processing device according to claim 1 .

3. The correspondence specifying means specifies a range in each of the basic image and the evaluation image where errors occur in specifying the correspondence between the same feature points in the basic image and the evaluation image, and specifies a predetermined position in the range specified in the basic image and a predetermined position in the range specified in the evaluation image as the correspondence between the same feature points. The image processing device according to claim 2 .

4. an image generating means for generating the basic image and the evaluation image by combining images of the moving object captured by a line scan camera; The image processing device according to claim 1 , comprising:

5. The correspondence relationship specifying means specifies a plurality of correspondence relationships between the same feature points in the basic image and the evaluation image in the movement direction. The image processing device according to claim 1 .

6. The correspondence specifying means detects predetermined objects in each of the base image and the evaluation image as candidates for the same feature point. The image processing device according to claim 5 .

7. The correspondence specifying means determines whether or not there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, detects the predetermined object as a candidate for the same feature point. The image processing device according to claim 6 .

8. Identifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined one moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image; identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction; reducing or enlarging each of the cutout sections in the evaluation image so that the image length of each of the cutout sections in the evaluation image matches the image length of the corresponding section in the basic image; A new evaluation image is generated by sequentially arranging the images of each section after the reduction or enlargement of the cut-out section in the movement direction. Image processing methods.

9. The computer of the image processing device, a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from that of photographing the basic image; a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the movement direction between the basic image and the evaluation image; an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that the image length of each of the cut-out sections in the evaluation image matches the image length of the corresponding section in the basic image; a new evaluation image generating means for generating a new evaluation image by sequentially arranging images of each of the sections after the reduction or enlargement of the cut-out sections in the movement direction; A program that functions as a

Images