Displacement measuring device, its system, displacement measuring method and its program

Abstract

To achieve alignment of images and measurement of displacement with high precision using images picked up at different times.SOLUTION: An image correction unit acquires an image representing two or more reference markers used as a reference of an amount of displacement and a measurement marker representing a pattern spatially repeated at a fixed pitch from an imaging unit for each frame to correct a position of the measurement marker so as to compensate for displacement between frames of the reference marker. An amount of displacement calculation unit calculates the amount of displacement of the measurement marker from a phase difference between frames of a moire image generated from a pattern of the measurement marker.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a displacement measuring device using image displacement measurement. ,So system, displacement measurement method Law The present invention relates to, for example, structural health monitoring and image processing. [Background technology]

[0002] Displacement detection has become an urgent issue in countermeasures against the deterioration of public facilities (infrastructure) consisting of structures such as bridges. Ring displacement meters, Doppler sensors, fixed cameras, and other fixed displacement measurement devices are sometimes used for displacement detection. Optical methods such as sampling moiré method and digital image correlation (DIC) method are sometimes used using images captured by fixed cameras. Installing fixed displacement measurement devices can be difficult depending on the topography (hilly terrain, waterside, etc.) and the arrangement of buildings and structures around the object to be measured.

[0003] For these reasons, attention is being paid to displacement measurement using mobile displacement measurement devices installed on mobile objects such as unmanned aerial vehicles (drones) and robots. Attempts have been made to equip unmanned aerial vehicles with various displacement measurement devices, such as laser Doppler vibrometers and global positioning systems (GPS). However, laser Doppler vibrometers are generally expensive and difficult to implement economically. GPS sometimes does not provide the accuracy required for infrastructure maintenance and management.

[0004] In recent years, vision-based structural health inspection systems using images captured by drone-mounted cameras have been proposed. For example, Non-Patent Documents 1 and 2 each relate to measuring infrastructure displacement using the DIC method. The DIC method can measure displacement with an accuracy of less than 1 mm. However, because the DIC method uses speckles as markers, accuracy is prone to significant degradation due to noise. Furthermore, the image capture management method described in Patent Document 1 is characterized by capturing an image when the overlap rate with the land portion of the image captured immediately before reaches a predetermined overlap rate. Therefore, this method is not suitable for continuously measuring minute displacements of infrastructure. The inspection method described in Patent Document 2 is characterized by detecting the corrosion state of live power transmission lines during inspection. Therefore, this method is also not suitable for measuring minute displacements of infrastructure. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-15704 [Patent Document 2] Patent Publication No. 2021-107992 [Patent Document 3] Patent No. 4831703 [Patent Document 4] Patent No. 6565037 [Non-patent literature]

[0006] [Non-Patent Document 1] Reagan, D., Sabato, A. and Niezrecki, C.; Feasibility of using digital image correlation for unmanned aerial vehicle structural health monitoring of bridges, Structural Health Monitoring, 2018, 17 (5), pp.1056-1072. [Non-patent document 2] Kalaizakis, M., Vitzilaios, N., Rizos, DC and Sutton, MA, Drone-Based Stereo DIC: System Development, Experimental Validation and Infrastructure Application, Experimental Mechanics, 2021, 61, pp.981-996. Summary of the Invention [Problem to be solved by the invention]

[0007] Patent Documents 3 and 4 describe a method for measuring infrastructure displacement using a sampling moiré method with images recorded over a short period of time with a fixed camera. However, if the camera cannot be installed in a fixed position, the position and shape of the object being measured as seen in the image change as the camera moves, making it impossible to obtain an accurate displacement amount. In other words, it has been difficult to measure displacement with high accuracy using images captured using cameras mounted on mobile objects such as drones, robots, and ships, or cameras built into mobile devices such as mobile phones. Furthermore, when monitoring displacement over a long period of time, the position of a camera that is supposed to be fixed may be disturbed by vehicle traffic, wind, etc. This can also be a factor in reducing the accuracy of the measured displacement.

[0008] The present invention has been made in view of the above points, and provides a displacement measuring device that realizes highly accurate image registration and displacement measurement using images captured at different times. ,So system, displacement measurement method Law The objective of the project is to provide the necessary training and programs. [Means for solving the problem]

[0009] (1) The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is a method for acquiring, from an imaging unit for each frame, images representing a plurality of markers each representing a spatially repeated pattern at a constant pitch; Between a reference frame, which is a frame used as a reference for the displacement of the marker, and a measurement frame, which is a frame used as a measurement target for the displacement of the marker The marker Displacement of The image is corrected to compensate for the difference with the accuracy of the pixel interval, and converted into a first corrected image. A moiré image is generated from the pattern of the markers appearing in the first corrected image so that the period of the phase is enlarged more than the period of the luminance. The reference frame and the measurement frame with an image corrector for correcting the first corrected image to convert it into a second corrected image so as to compensate for a phase difference between the first corrected image and the second corrected image; The reference frame and the measurement frame with a displacement amount calculation unit that calculates the displacement amount of the marker from the phase difference between the Two markers that form part of the plurality of markers are reference markers that serve as a reference for the amount of displacement, and markers that form another part of the plurality of markers are measurement markers, and in the image of the reference frame, the positions of the measurement markers are internal division points between the two reference markers, and the image correction unit corrects the coordinates of the positions of the measurement markers in the first corrected image and the second corrected image of the measurement frame by subtracting the coordinates of the positions of the internal division points in the measurement frame. It is a displacement measuring device.

[0011] (2) Another aspect of the present invention may be a program for causing a computer to function as the displacement measuring device of (1).

[0012] (3) Another aspect of the present invention may be a system including an imaging unit and the displacement measuring device of (1).

[0013] (4) Another aspect of the present invention is a method for measuring a displacement of a subject, the method comprising: acquiring, for each frame, an image representing a plurality of markers each representing a spatially repeated pattern at a constant pitch; Between a reference frame, which is a frame used as a reference for the displacement of the marker, and a measurement frame, which is a frame used as a measurement target for the displacement of the marker The marker Displacement of The image is corrected to compensate for the difference with the accuracy of the pixel interval, and converted into a first corrected image. A moiré image is generated from the pattern of the markers appearing in the first corrected image so that the period of the phase is enlarged more than the period of the luminance. The reference frame and the measurement frame with an image correction step of correcting the first corrected image to convert it into a second corrected image so as to compensate for a phase difference between the first corrected image and the second corrected image; The reference frame and the measurement frame witha displacement amount calculation step of calculating a displacement amount of the marker from a phase difference between the two markers constituting a part of the plurality of markers are reference markers used as a reference for the amount of displacement, and another marker constituting a part of the plurality of markers is a measurement marker, and the position of the measurement marker in the image of the reference frame is an internal division point between the two reference markers, and the image correction step corrects the coordinates of the positions of the measurement markers in the first corrected image and the second corrected image of the measurement frame by subtracting the coordinates of the positions of the internal division point in the measurement frame. This is a displacement measurement method. [Effects of the Invention]

[0015] According to the present invention, it is possible to achieve highly accurate image registration and displacement measurement using images captured at different times. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a side view for explaining an overview of a displacement measuring system according to an embodiment of the present invention. [Figure 2] 1 is a front view for explaining an overview of a displacement measuring system according to an embodiment of the present invention. [Figure 3] 1 is a schematic block diagram illustrating an example of the functional configuration of a displacement measurement system according to an embodiment of the present invention. [Figure 4] 5 is a flowchart showing a first example of a displacement measurement process according to the present embodiment. [Figure 5] 10A and 10B are diagrams illustrating examples of the arrangement of markers before and after deformation of a measurement object. [Figure 6] 10A and 10B are diagrams illustrating examples of changes in the positional relationship between reference markers due to deformation. [Figure 7] 10 is a flowchart showing a second example of the displacement measurement process according to the present embodiment. [Figure 8] 10 is a display example of an image of each frame. [Figure 9] 10A and 10B are diagrams illustrating examples of changes in displacement amount over time at each marker. [Figure 10] FIG. 1 illustrates an example of an experimental setup. [Figure 11] FIG. 10 is a diagram illustrating an example of changes in the position of each marker over time. [Figure 12] FIG. 10 is a diagram showing an example of the trajectory of each marker. [Figure 13] 10A and 10B are diagrams illustrating an example of a change over time in the positional relationship between reference markers. [Figure 14] FIG. 10 is a diagram showing an example of changes in the position of an observed marker over time. [Figure 15] FIG. 10 is a diagram showing an example of observed changes in deflection over time. [Figure 16] FIG. 10 is a diagram illustrating an example of a trajectory of a displacement amount. [Figure 17] FIG. 1 is a diagram showing an experimental optical system for displacement measurement by aerial photography. [Figure 18] FIG. 10 is a diagram showing an example of displacement measurement by aerial photography. [Figure 19] FIG. 10 is an explanatory diagram for explaining homography transformation. [Figure 20] 10 is a flowchart showing a third example of the displacement measurement process according to the present embodiment. [Figure 21] FIG. 1 is a schematic block diagram illustrating an example of the functional configuration of an image correction system according to an embodiment of the present invention. [Figure 22] FIG. 10 is an explanatory diagram showing a first example of image alignment. [Figure 23] FIG. 10 is an explanatory diagram showing a second example of image alignment. [Figure 24] 10A and 10B are explanatory diagrams illustrating experimental results of image alignment correction. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 and 2 are a side view and a front view for explaining an overview of a displacement measurement system 1 according to this embodiment. In the example of FIG. 1, the displacement measurement system 1 is used to measure the displacement of a bridge Br1 as a measurement object. Two reference markers Mk-A and Mk-B and a measurement marker Mk-C are arranged in that order in the longitudinal direction on the side of the bridge Br1. The reference markers Mk-A and Mk-B are each used as a reference for the amount of displacement. The measurement marker Mk-C is installed at a predetermined measurement point as a target position for measuring the amount of displacement.

[0018] The imaging unit 20 is supported by a drone Dn flying in the air at a position facing the side of the bridge Br1. At this position, the imaging unit 20 captures an image showing the reference markers Mk-A and Mk-B and the measurement marker Mk-C for each frame. The imaging unit 20 may be positioned so that the reference markers Mk-A, Mk-B, and the measurement marker Mk-C are included in its field of view. A road is laid on the surface of bridge Br1. When an inspection vehicle Vc passes over the road, the weight of the inspection vehicle Vc causes deformation of bridge Br1. Under this setting, the displacement measurement system 1 can measure the amount of displacement at measurement marker Mk-C that accompanies deformation of bridge Br1. The displacement measurement system 1 may also acquire images captured over a longer period of time to observe changes in the shape of bridge Br1 over time. In the following description, the "reference marker" and "measurement marker" may be collectively referred to simply as "marker."

[0019] Next, an overview of the method for measuring the amount of displacement using the displacement measurement system 1 will be explained using Figure 2. The displacement measurement system 1 illustrated in Figure 2 compensates for changes in the positions of the reference markers Mk-A and Mk-B that occur between frames captured at different times. The horizontal and vertical directions of the captured image correspond to the x and y directions, respectively. The x direction corresponds to the longitudinal direction of the bridge girder in the three-dimensional space in which the bridge Br1 is actually installed (sometimes referred to as "subject space" in the following explanation), and the y direction corresponds to the vertical direction in the subject space. Furthermore, the space on the image of each frame is sometimes referred to as "image space" to distinguish it from the subject space.

[0020] FIG. 2(a) shows an image captured at time t1. At this point, no deformation has occurred in the bridge Br1. This image is used as a reference image. The reference markers Mk-A, Mk-B, and measurement markers Mk-C are arranged in a straight line on the left and right of the drawing. That is, the position of the measurement marker Mk-C (x C ,y C ) is the position of the reference marker Mk-A (x A ,y A ) and the position of the reference marker Mk-B (xB ,y B ) corresponds to the dividing point of the line passing through it.

[0021] FIG. 2(b) illustrates an image captured at time t2. The shape of the bridge Br1 that appears in the image at time t2 is deformed from that that appears in the image at time t1. The position (x A ',y A ') is the position (x A ,y A ) at time t2. On the other hand, the position of the reference marker Mk-B at time t2 (x B ',y B ') is the position (x B ,y B ) is displaced upward. However, the deformation of the bridge Br1 that appears in the image at time t2 includes apparent deformation due to changes in the position and orientation of the imaging unit 20. The apparent deformation is observed as "blur" in the image, which causes measurement errors.

[0022] Therefore, as illustrated in FIG. 2(c), the displacement measurement system 1 calculates the corrected position (x A * ,y A * ) and the corrected position of the reference marker Mk-B (x B * ,y B * ) are the positions (x A ,y A ) and the position of the reference marker Mk-B (x B ,y B ) in the measurement frame. The displacement measurement system 1 determines, for example, coordinate transformation parameters for correcting the image at time t2 by coordinate transformation. The displacement measurement system 1 determines the position (x C* ,y C * ) can be calculated by performing coordinate transformation. As a coordinate transformation method, for example, a linear transformation such as an affine transformation can be used.

[0023] The displacement measurement system 1 then generates moiré images of the measurement marker Mk-C for each of the corrected image captured at time t2 and the image captured at time t1. The displacement measurement system 1 calculates the amount of displacement of the measurement marker Mk-C installed on the bridge Br1 based on the phase difference between the frames of the generated moiré images. Even when the imaging unit 20 moves, changes in the position of the measurement marker Mk-C that appear in the image of the measurement frame due to the movement are compensated for. Therefore, the amount of displacement of the measurement marker Mk-C can be measured with high accuracy.

[0024] Each marker may have any pattern as long as the luminance is regularly distributed. In the examples of Figures 1 and 2, the luminance of each marker forms a grid pattern that varies at a constant period (pitch) in both the horizontal and vertical directions. The imaging unit 20 may be installed in a position where the pitch of the marker pattern appearing in the captured image is at least two pixels.

[0025] When measuring displacement in a certain direction (for example, the vertical direction), a marker is used that displays a stripe image arranged in a direction that intersects that direction. The marker does not have to be a dedicated object primarily for displacement measurement; a regular pattern displayed on the surface of the structure to be measured may also be used. For example, a pattern formed by the boundaries of individual window frames arranged on the surface of a building, or bricks or tiles arranged on a wall, may be used. The marker may be applied to measuring displacement on any scale, including bridges, buildings, dams, tunnel interiors, and small test pieces as well as other objects to be measured.

[0026] However, the positions of the reference markers are set at positions that can be considered stationary over time, or at positions that fluctuate relatively less than the positions of the measurement markers. In the examples of Figures 1 and 2, the parts of the bridge piers where reference markers Mk-A and Mk-B are set fluctuate relatively less due to external forces than the parts of the bridge girder where measurement marker Mk-C is set. External forces acting on a bridge include, for example, the movement of vehicles and pedestrians, wind, water currents, earthquakes, etc. Furthermore, the number of reference markers is not limited to two and can be three or more. The number of measurement markers is not limited to one and can be two or more.

[0027] Next, a description will be given of an example of the functional configuration of the displacement measuring system 1 according to this embodiment. Fig. 3 is a schematic block diagram showing an example of the functional configuration of the displacement measuring system 1 according to this embodiment. The displacement measuring system 1 includes a displacement measuring device 10 and an imaging unit 20. The displacement measuring device 10 includes a parameter input unit 12, a calculation processing unit 14, and a display unit 16.

[0028] Various parameters are input to the parameter input unit 12. These parameters include parameters used for identifying the position of each fiducial marker, measuring the displacement of a measurement marker, generating a moiré image from the fiducial markers and the measurement markers, calculating the amount of displacement from the phase difference of the moiré image, and the like. These parameters will be described together with the functions of the calculation processing unit 14. The parameter input unit 12 may be configured to include a data input interface, or may be configured to include input devices such as a mouse, touch sensor, keyboard, etc. that input various information in response to user operations.

[0029] The arithmetic processing unit 14 performs processing for enabling the functions of the displacement measuring device 10 and processing for controlling these functions. The arithmetic processing unit 14 may be configured as a computer system including, for example, a processor such as a CPU (Central Processing Unit) and a storage medium such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The processor may implement the functions of each functional unit by reading a predetermined control program pre-stored in the storage medium and executing processing instructed by instructions written in the read control program. The arithmetic processing unit 14 includes, as functional units, a marker detection unit 142, a first image correction unit 144, a first displacement amount calculation unit 146, a second image correction unit 148, a second displacement amount calculation unit 150, and a displacement amount output unit 152.

[0030] The marker detection unit 142 detects reference markers and measurement markers from the image of each frame shown in the image data input from the imaging unit 20. The marker detection unit 142 can detect markers by performing known image recognition processing. For example, the marker detection unit 142 uses a known machine learning model (e.g., a neural network) to calculate a reliability indicating the degree of likelihood that a known marker pattern will appear for each block, which is a partial region of the image to be processed. The marker detection unit 142 may be configured with pre-set pattern data related to a marker pattern, which is the form of a reference marker, and calculate the degree of similarity (similarity) between a partial region of the image to be processed and the marker pattern as the reliability. The marker detection unit 142 can determine a predetermined number of regions whose calculated reliability is higher than a predetermined reliability threshold as marker regions (hereinafter, sometimes referred to as "marker regions"). The predetermined number corresponds to the sum of the number of reference markers and the number of measurement markers. The marker detection unit 142 outputs, to the first image correction unit 144, marker information including the center coordinates of the marker that indicates the marker area detected from the image of each frame.

[0031] The first image correction unit 144 identifies, for each frame, the marker area indicated in the marker information input from the marker detection unit 142. The distribution of the identified marker area may differ between frames due to differences in the position and orientation of the imaging unit 20 at each imaging time. The first image correction unit 144 calculates coordinate transformation parameters for correcting the image indicated in the image data input from the imaging unit 20 so that differences in the position of the reference marker are compensated for between frames. The first image correction unit 144 can calculate the coordinate transformation parameters, for example, so that the position of the reference marker in a measurement frame of interest as the measurement target for measuring the displacement of the measurement marker coincides with the position of the reference marker in a reference frame used as the reference for the displacement of the measurement marker. For example, an image captured at a time when no deformation occurs may be used as the image of the reference frame. The first image correction unit 144 outputs first corrected image data indicating an image corrected by coordinate transformation using the calculated coordinate transformation parameters to the first displacement amount calculation unit 146 and the second image correction unit 148. The first corrected image data includes the corrected image of the measurement frame. The first corrected image data may include an image of the reference frame that has not been corrected.

[0032] At this stage, the accuracy of the position of each reference marker is approximately the same as the spacing (pixel pitch) between adjacent pixels, so the accuracy of the position of the measurement marker in the corrected image is also approximately the same as the pixel pitch. In this application, the correction of the image or the position of the measurement marker on that image performed by the first image corrector 144 may be referred to as "coarse correction," and the accuracy of this correction may be referred to as "pixel accuracy."

[0033] The first displacement amount calculation unit 146 calculates, as the first displacement amount, the displacement amount of the measurement marker in the subject space shown in the image of each frame indicated by the first corrected image data input from the first image correction unit 144. That is, the first displacement amount calculation unit 146 calculates the displacement amount of the actual measurement marker based on the displacement between the position of the measurement marker in the reference frame and the position of the measurement marker in the corrected measurement frame.

[0034] The first displacement amount calculation unit 146 can use a sampling moiré method in calculating the first displacement amount. More specifically, the first displacement amount calculation unit 146 generates a moiré image of the measurement markers shown in the image of each frame. For example, the first displacement amount calculation unit 146 generates T thinned images by thinning a partial image including at least the area of ​​the measurement marker at a period T (T is a predetermined integer of 2 or greater indicating the number of pixels) that is the same as or different from the period P of the pattern of the measurement marker. The phase of each of the T thinned pixels (the coordinate of the pixel that is the starting point for thinning) is shifted to a value between 0 and T-1. The first displacement amount calculation unit 146 can generate a moiré image for each of the T thinned images by interpolating pixel values ​​between adjacent pixels after thinning. The first displacement amount calculation unit 146 performs a discrete Fourier transform on the phase-shifted T moiré images to calculate a phase distribution at a predetermined spatial frequency. The first displacement amount calculation unit 146 can also calculate phase distributions for other frames using a similar method. The first displacement amount calculation unit 146 calculates the displacement distribution by multiplying the period (pitch) p of the actual pattern of the phase difference distribution, which is the difference in phase distribution between frames, by a coefficient obtained by dividing the frequency by 2π. The first displacement amount calculation unit 146 then averages the displacement amounts for each pixel constituting the displacement distribution across the frequency and individual pixels, and determines the displacement amount of the measurement marker to which that pixel belongs as the first displacement amount. The first displacement amount calculation unit 146 outputs first displacement amount data indicating the determined first displacement amount to the displacement amount output unit 152. Thus, the first displacement amount calculation unit 146 can roughly calculate the displacement amount using an image corrected with pixel accuracy. The periods P, T, and p are preset in the first displacement amount calculation unit 146 as parameters for calculating the first displacement amount. The period P is the pitch of the lattice marker on the captured image, measured in units of, for example, pixels. The period p is the actual physical pitch of the lattice marker, measured in units of, for example, mm. The period T may differ from the period P, but it may be a value that is close to the period P. The reciprocal 1 / Q of the period Q of the phase of the moiré image corresponds to the absolute value of the difference between the reciprocal 1 / T of the period T and the reciprocal 1 / P of the period P, so the smaller the difference between the periods T and P, the larger the period Q of the phase of the moiré image.Therefore, the period T may be determined in advance according to a desired magnification of the period Q with respect to the period T. The sampling moiré method is described in more detail in, for example, International Publication No. 2017 / 138414.

[0035] The second image correction unit 148 identifies the marker region of each marker from the image for each frame indicated in the first corrected image data input from the first displacement amount calculation unit 146. The second image correction unit 148 generates a moiré image for each identified reference marker. The second image correction unit 148 can use the same method as the first displacement amount calculation unit 146 to generate the moiré image. The second image correction unit 148 calculates the phase distribution of the moiré image generated for a predetermined frequency for each frame. Similar to the first image correction unit 144, the second image correction unit 148 calculates coordinate transformation parameters for correcting the image indicated in the first corrected image data by coordinate transformation so that differences in the positions of the reference markers are compensated for between frames. Because the phase period of the moiré image is longer than the luminance period of the markers, the image correction at this stage is performed with a precision of less than one pixel (subpixel precision). The second image corrector 148 outputs second corrected image data representing an image corrected by coordinate transformation using the calculated coordinate transformation parameters to the second displacement amount calculator 150. The second corrected image data may include an image of the uncorrected reference frame in addition to the corrected measurement frame. In this application, the image correction performed by the second image corrector 148 may be referred to as "micro correction," and the accuracy of this correction may be referred to as "subpixel accuracy." The second image corrector 148 may use the same method as the first image corrector 144 for coordinate transformation.

[0036] The second displacement amount calculation unit 150 calculates the displacement amount of the measurement marker between frames as the second displacement amount using the sampling moiré method, similar to the first displacement amount calculation unit 146, based on the image of each frame indicated by the second corrected image data input from the second image correction unit 148. The second displacement amount calculation unit 150 outputs second displacement amount data indicating the determined second displacement amount to the displacement amount output unit 152. Therefore, the second displacement amount calculation unit 150 determines a minute displacement amount as the second correction amount using the image corrected with sub-pixel accuracy.

[0037] The displacement amount output unit 152 selects either the first displacement amount of each measurement marker indicated in the first displacement amount data input from the first displacement amount calculation unit 146, or the second displacement amount of each measurement marker indicated in the second displacement amount data input from the second displacement amount calculation unit 150, as the final displacement amount of the measurement marker. The displacement amount output unit 152 outputs information on the selected displacement amount to the display unit 16. The displacement amount output unit 152 may be set in advance to select either the first displacement amount or the second displacement amount, or may select either the first displacement amount or the second displacement amount according to an input signal input from an input device.

[0038] The display unit 16 displays information indicating the amount of displacement of the measurement marker input from the calculation processing unit 14. The display unit 16 may display the displacement in any manner. For example, the display unit 16 may represent the displacement numerically or may graphically show the change in magnitude. The display unit 16 may include, for example, a liquid crystal display, a dial, or the like.

[0039] The imaging unit 20 captures an image representing a subject within a predetermined field of view centered on the imaging direction. As illustrated in FIGS. 1 and 2, the imaging unit 20 may be installed at a position and orientation such that the field of view includes a subject on which multiple reference markers and at least one measurement marker are attached. In this embodiment, the position of the imaging unit 20 does not need to be fixed. As illustrated in FIGS. 1 and 2, the imaging unit 20 may be installed or supported on a mobile object such as a drone. The mobile object is not limited to a drone, but may also be a robot, a cart, a vehicle, a ship, or a living thing (e.g., handheld photography by a human). The imaging unit 20 may also be included in a device whose primary function is not necessarily imaging, such as a multi-function mobile phone (smartphone) or a microscope.

[0040] The imaging unit 20 is a digital video camera that sequentially captures one frame of image every predetermined time (for example, 1 / 960 to 1 / 15 seconds). A typical digital video camera has pixels arranged in a two-dimensional imaging plane, and can capture two-dimensional images. A plurality of still images captured sequentially constitute a moving image. The imaging unit 20 has an output data interface that outputs image data representing the captured images to the displacement measuring device 10 wirelessly or via a wired connection.

[0041] Next, an example of displacement measurement processing based on the positions of reference markers will be explained using Figures 4 to 6. However, two reference markers and one or more measurement markers are used (see Figures 1 and 2). At the reference time (reference frame), the measurement markers are placed on a line with the two reference markers as its end points. However, this example uses an affine transformation for coordinate transformation.

[0042] (Affine transformation) Fig. 4 is a flowchart showing a first example of a displacement measurement process according to this embodiment. When executing the process illustrated in Fig. 4, the calculation processing unit 14 determines a reference frame that serves as a reference for the amount of displacement. For example, an image representing the measurement object before displacement is used as the image of the reference frame. Typically, an image at the beginning of observation is determined as the reference frame, and frames captured thereafter are used as measurement frames to be corrected.

[0043] (Step S102) The marker detection unit 142 detects a marker for each frame shown in the image data input from the imaging unit 20. The marker detection unit 142 determines the coordinates of the center of gravity of each detected marker as the central coordinates. (Step S104) The first image corrector 144 analyzes the translation, rotation, and scaling factor from the reference frame to the measurement frame using the arrangement of the two reference markers in the reference frame and the arrangement of the two reference markers in a measurement frame different from the reference time. The translation, rotation, and scaling factor correspond to coordinate transformation parameters by affine transformation from the reference frame to the measurement frame. The first image corrector 144 corrects the image of the measurement frame with pixel accuracy (coarse correction) to compensate for the translation at the analyzed translation amount, the rotation at the rotation amount, and the magnification at the scaling factor. (Step S106) The first displacement amount calculation unit 146 uses the sampling moiré method to analyze the phase difference between the moiré images of the measurement markers in the reference frame and the measurement frame, and calculates the actual displacement amount of the measurement marker as the first displacement amount from the analyzed phase difference.

[0044] (Step S108) For each reference marker, the second image corrector 148 analyzes the phase difference between the moiré image in the reference frame and the moiré image in the measurement frame after position correction with pixel accuracy. Based on the analyzed phase difference, the second image corrector 148 analyzes the translation amount, rotation amount, and scale ratio from the reference frame to the measurement frame. The second image corrector 148 re-corrects the image of the corrected measurement frame with subpixel accuracy (micro-correction) so as to compensate for the translation at the analyzed translation amount, the rotation at the rotation amount, and the magnification at the scale ratio. (Step S110) The second displacement amount calculation unit 150 uses the sampling moiré method to analyze the phase difference between the moiré images of the measurement markers in the reference frame and the measurement frame, and calculates the displacement amount of the actual measurement marker as the second displacement amount from the analyzed phase difference. (Step S112) The displacement amount output unit 152 selects either the first displacement amount obtained in step S106 or the second displacement amount obtained in step S110 as the final displacement amount. The displacement amount output unit 152 causes the display unit 16 to display information on the selected displacement amount.

[0045] FIG. 5(a) shows an example of the arrangement of reference markers Mk-A and Mk-B and measurement marker Mk-C on a reference frame captured before deformation of the measurement object. The position of measurement marker Mk-C corresponds to the internal division point of the two reference markers Mk-A and Mk-B. The marker detection unit 142 obtains the coordinates (x A ,y A ), (x B ,y B ), (x C ,y C ) can be specified in pixel units (pixel accuracy). FIG. 5(b) shows an example of the arrangement of reference markers Mk-A, Mk-B and measurement marker Mk-C on a measurement frame captured after deformation of the measurement object. The marker detection unit 142 detects the coordinates (x A ',y A '), (x B ',y B '), (x C ',y C ') can be specified in pixel units (pixel accuracy).

[0046] Factors that cause the positions of the fiducial markers Mk-A and Mk-B to vary between frames include translation, scale (change in size, deviation in magnification), and rotation. The first image corrector 144 corrects the displacement of any one of the reference markers between frames, for example, the displacement from the center of gravity A to the center of gravity A′ (x A '-x A ,y A '-y A ) can be calculated as the translation amount (Δx, Δy) of the reference marker Mk-A. The first image correction unit 144 can calculate the magnification r' / r from the distance r between the two reference markers Mk-A and Mk-B in the reference frame to the distance r' between the two reference markers A and Mk-B in the measurement frame as a scale factor Δs (magnification difference, see FIG. 6(b)). The distance r is calculated as √(x B -x A ) 2 +(y B -y A ) 2 , the distance r' is √(x B '-x A ') 2 +(y B '-y A ') 2 This becomes:

[0047] The first image corrector 144 can calculate the angle between a line AB passing through centers of gravity A and B in the reference frame and a line A'B' passing through centers of gravity A' and B' in the measurement frame as the rotation amount Δθ (rotational deviation, see FIG. 6(a)). The rotation amount Δθ can be calculated as the inverse cosine function value (arccos) of the cosine value (cos) obtained by normalizing the dot product of a vector AB from center of gravity A to center of gravity B and a vector A'B' from center of gravity A' to center of gravity B' by the absolute values ​​of the vectors AB and A'B'. FIG. 6(c) shows a situation in which rotational deviation and magnification deviation overlap. Generally, in addition to rotational deviation and magnification deviation, translation may also be included.

[0048] The calculated scale, rotation, and translation amount correspond to coordinate transformation parameters for converting an arbitrary position on the reference frame onto the measurement frame. The first image corrector 144 can correct the position of the measurement marker on the measurement frame using the calculated scale, rotation, and translation amount to perform coordinate transformation that matches the position of the reference marker between the reference frame and the measurement frame.

[0049] (AB correction) The position of the measurement marker can also be realized using a simpler method. The method described below is also called AB correction (Absolute Blurring Compensation). The first image corrector 144 can correct the position of the measurement marker in the measurement frame using the positions of two reference markers in the measurement frame, without explicitly calculating the scale, rotation amount, and translation amount.

[0050] In the AB correction, the center of gravity C of the measurement marker Mk-C in the reference frame is set on a line passing through the centers of gravity A and B of the two reference markers Mk-A and Mk-B. The first image correction unit 144 calculates the inter-frame displacement (x A '-x A ,y A '-y A ), (x B '-x B ,y B '-y B ) is calculated by taking the weighted average value of the center of gravity (x C ',y C However, if the center of gravity C of the measurement marker Mk-C is the dividing point between the centers of gravity A and B of the two reference markers Mk-A and Mk-B, the displacement (x A '-x A ,y A '-y A ), (x B '-x B ,y B '-y B ) can be used as a weighting coefficient for the distance between the centers of gravity A and B and the center of gravity C. For example, the distance between the centers of gravity A and C and the distance between the centers of gravity C and B can be expressed as L A , L B Then, the first image correcting unit 144 calculates the corrected coordinate value y of the measurement marker Mk-C in the measurement frame as C * Similarly, the first image correcting unit 144 calculates the corrected coordinate value x of the measurement marker Mk-C in the x direction in the measurement frame. C * can be calculated according to equation (2).

[0051]

number

[0052]

number

[0053] In particular, when the center of gravity C of the measurement marker Mk-C is the midpoint of the centers of gravity A and B of the two reference markers Mk-A and Mk-B, the first image correction unit 144 corrects the coordinate values ​​of the centers of gravity in the measurement frames of the reference markers Mk-A and Mk-B by (x A ',y A '), (x B ',y B The simple average value of the coordinates of the center of gravity of the measurement marker Mk-C (x C ',y C ') can be subtracted to correct for this.

[0054] The measurement frame can also be considered to be formed by in-plane deformation of the reference frame image with the center of gravity C as the base point. In this case, rotation, magnification (scale), and lens aberration appear point-symmetrically with respect to the center of gravity C. If centers of gravity A and B are in the horizontal direction (x direction), rotation displaces centers of gravity A and B in the y and -y directions, respectively, while a change in scale displaces centers of gravity A and B in the -x and +x directions. Therefore, if center of gravity C is the dividing point between centers of gravity A and B, the displacements occurring at centers of gravity A and B are canceled out by a weighted average based on the reciprocal of the distance to each center of gravity A and B. Furthermore, because centers of gravity A, B, and C all displace in the same direction by the same amount due to translation, the displacement due to translation is canceled out by subtracting the coordinates of the dividing point between centers of gravity A and B from the coordinates of center of gravity C.

[0055] In addition, if the center of gravity C of the measurement marker Mk-C is the external dividing point of the centers of gravity A and B of the two reference markers Mk-A and Mk-B, the displacement (x A '-x A ,y A '-yA ), (x B '-x B ,y B '-y B ) can be used as a weighting coefficient for the distance from each of the centers of gravity A and B to the external division point. In this case, too, parallel translation, rotation, and changes in scale can be compensated for by simple calculations. However, because errors due to lens aberration are not canceled out, the accuracy is lower than when the center of gravity C of the measurement marker Mk-C is the internal division point of the centers of gravity A and B of the two reference markers Mk-A and Mk-B.

[0056] The sampling moiré method cannot measure displacements greater than half the period of the pattern displayed on the measurement marker. This is because the sampling moiré method uses the phase difference of the moiré generated from the pattern, making it impossible to distinguish between displacements whose difference is an integer multiple of the pattern period. This method corrects the displacement between frames with pixel accuracy, and based on the displacement between the center of gravity position C' of the measurement marker Mk-C in the corrected measurement frame and the center of gravity position C in the reference frame, the displacement of the measurement marker Mk-C between frames can be kept within half the period of the pattern pitch of the measurement marker Mk-C. Therefore, the first displacement amount calculation unit 146 can use the sampling moiré method to measure the actual displacement of the measurement marker Mk-C between frames after correction with pixel accuracy as the first displacement amount.

[0057] Furthermore, the second image corrector 148 can correct the position of the measurement frame in the measurement frame with subpixel accuracy by using AB correction on the position of the reference frame in the measurement frame corrected with pixel accuracy. When performing correction with subpixel accuracy, the second image corrector 148 generates a moiré image for each marker in the measurement frame and determines the phase distribution of the generated moiré image. The second image corrector 148 generates a phase distribution for each reference marker in the measurement frame. The second image corrector 148 calculates the phase distribution obtained by weighting the phase distribution for each reference marker as the correction amount for the measurement marker. On the other hand, the second image corrector 148 subtracts the correction amount from the phase distribution in the measurement marker in the measurement frame to generate a corrected phase distribution. The second displacement amount calculator 150 can measure the second displacement amount with subpixel accuracy using a sampling moiré method based on the phase difference between the corrected phase distribution in the measurement marker in the measurement frame and the phase distribution in the measurement marker in the reference frame.

[0058] Next, an example of displacement measurement processing using AB correction will be described with reference to FIG. 7. FIG. 7 is a flowchart showing a method using AB correction as a second example of displacement measurement processing according to this embodiment. When executing the processing illustrated in FIG. 7, similarly to the example of FIG. 4, the calculation processing unit 14 sets a reference frame that serves as a reference for the amount of displacement, and other frames are used as measurement frames to be corrected. However, in the reference frame, these measurement markers and reference markers are placed so that the center coordinate of the measurement marker is on a line that passes through the center coordinates of the two reference markers, that is, the internal or external division point.

[0059] (Step S152) The marker detection unit 142 detects a marker for each frame shown in the image data input from the imaging unit 20. The marker detection unit 142 determines the coordinates of the center of gravity of each detected marker as the central coordinates. (Step S154) The first image corrector 144 calculates the inter-frame displacement of the center coordinates of each fiducial marker between the reference frame and the measurement frame. The first image corrector 144 corrects the center coordinates of the measurement markers in the measurement frame based on the weighted average value of the inter-frame displacement of the center coordinates of each fiducial marker according to equations (1) and (2). This corrects the positions of the measurement markers with pixel accuracy (coarse correction). (Step S156) The first displacement amount calculation unit 146 uses the sampling moiré method to analyze the phase difference between the moiré images of the measurement marker in the reference frame and the measurement marker in the measurement frame whose position has been corrected, and calculates the displacement amount of the actual measurement marker as the first displacement amount from the analyzed phase difference.

[0060] (Step S158) The second image corrector 148 analyzes the phase difference between the moiré image in the reference frame and the moiré image in the measurement frame after the position of each reference marker has been corrected with pixel accuracy. The analyzed phase difference corresponds to the inter-frame displacement with sub-pixel accuracy. The second image corrector 148 corrects the center coordinates of the measurement marker after the correction with pixel accuracy in the measurement frame based on the weighted average value of the inter-frame displacement of the center coordinates of each reference marker with sub-pixel accuracy, in accordance with equations (1) and (2). This corrects the position of the measurement marker with sub-pixel accuracy (fine correction). (Step S160) The second displacement amount calculation unit 150 uses the sampling moiré method to analyze the phase difference between the moiré image of the measurement marker in the reference frame and the moiré image of the measurement marker in the measurement frame whose position has been corrected with sub-pixel accuracy, and calculates the displacement amount of the actual measurement marker as the second displacement amount from the analyzed phase difference. (Step S162) Either the first displacement amount obtained in step S156 or the second displacement amount obtained in step S160 is selected. The displacement amount output unit 152 causes the display unit 16 to display information on the selected displacement amount.

[0061] (simulation) Next, a description will be given of a simulation performed on the displacement measuring device 10 according to this embodiment. In the simulation, a known amount of displacement is used to verify the measurement. In the first simulation, image a shown in FIG. 8(a) was used as the image of the reference frame before displacement, and image b shown in FIG. 8(b) was used as the image of the measurement frame after displacement. The coordinates of the measurement markers in image b were then corrected, and the coordinates of the corrected measurement markers in image c (FIG. 8(c)) were calculated. Image c was obtained by transforming the coordinates in image b so that the positions of the reference markers matched those in image a. The size of images a and c was 4000 pixels x 1600 pixels.

[0062] Image a is a composite image that simulates the front of a bridge on which seven markers AG are installed. The center coordinates (X, Y) of the markers AG are as follows: A(400,360), D(1200,960), C(2000,360), E(2800,360), B(3600,360), F(400,960), G(3600,960) Markers A and G each have a grid pattern in which the brightness fluctuates over six periods in the horizontal and vertical directions at a constant pitch P. The brightness value at the coordinates (i, j) of each marker is set to Bias+Amp(cos(2π(i / P+X)+cos(2π(j / P+Y))). Note that the pitch P is 20 pixels, the bias value Bias is 128, and the amplitude Amp is 50. Of the markers AG, markers A and B were used as reference markers, and marker CE was used as a measurement marker. The position of marker CE corresponds to the internal division point of the line passing through markers A and B. The position of marker C corresponds to the midpoint between markers A and B.

[0063] Image b was created by applying known displacements to the markers CE shown in image a, and then performing an affine transformation on the images obtained by applying the displacements, adding translation, scaling, and rotation. The displacements applied to the markers CE are as follows: D(0.025,-0.05), C(0,-0.1), E(0,-0.01) The parallel shift amounts Δx and Δy were set to 6.6 pixels and 13.2 pixels, respectively, and the rotation angle Δθ was set to 2.2°.

[0064] For comparison, image b', obtained by applying a known displacement before the affine transformation, was used as the post-deformation measurement frame image, and the displacement of marker CE was calculated using the sampling moiré method. Here, the actual pitch p of the pattern displayed on the marker was set to 100 mm, and the thinning period T was set to 20 pixels. To separate the x- and y-directional pattern components from the two-dimensional grid pattern, a low-pass filter with a filter size of 43 pixels and a cutoff frequency of 0.1 was applied. The size of the displacement averaging region was set to 30 pixels x 30 pixels. As a result, the displacements of markers D, C, and E in the y-direction before and after deformation were 5.0, 10.0, and 1.021 mm, respectively.

[0065] In the second simulation, the displacements of markers D, C, and E in the y direction were determined using the following approach: Unless otherwise specified, AB correction was used as the image correction method. (1) The coordinate values ​​(pixel accuracy) of markers D, C, and E in image c obtained after rough correction were corrected based on the coordinate values ​​of markers A and B and the internal division ratio, and the displacement in the y direction was determined using the sampling moiré method. As a result, the displacement amounts in the y direction of markers D, C, and E before and after deformation were 5.007, 9.992, and 1.020 mm, respectively. (2) Using the coordinate values ​​(subpixel accuracy) of markers D, C, and E in image c obtained after coarse and fine correction, the displacement in the y direction was determined using the sampling moiré method. As a result, the displacements of markers D, C, and E in the y direction before and after deformation were 5.064, 10.034, and 1.071 mm, respectively. (3) The coordinate values ​​of markers D, C, and E in image c obtained after coarse and fine correction were corrected based on the coordinate values ​​of markers A and B and the internal division ratio, and the displacement in the y direction was determined using the sampling moiré method. As a result, the displacement amounts in the y direction of markers D, C, and E before and after deformation were 5.019, 9.988, and 1.024 mm, respectively.

[0066] Comparing (1) and (2), the error in the displacement obtained from (1) is smaller than that from (2). This shows that even if the displacement is obtained with high accuracy using the sampling moiré method, the accuracy of the correction for the imaging direction simulated by the affine transformation is not necessarily high. Comparing (1) and (3), (1) has a slightly smaller error than (3). However, the error in the displacement is similar between (1) and (2), and there is no significant difference.

[0067] In the third simulation, the displacement amount at the marker CE was calculated for each measurement frame after displacement, assuming the following cases (i) to (v). The period for each case was set to 20 frames. (i) When the position of the imaging unit 20 is stationary (stationary, no transformation), (ii) when it is parallel to the x direction (horizontal direction) (Δx), (iii) when it is moving in the y direction (Δy), (iv) when it is rotating (Δθ), (v) when the magnification changes (Δs).

[0068] In (i), to simulate camera shot noise, random noise with an amplitude equivalent to 2% of the amplitude of the brightness representing the marker pattern was added to the brightness value of each marker shown in the reference frame image shown in Figure 8(a), and the position was varied for each measurement frame. In (ii), the initial value of the amount of translation in the x direction added to the image obtained by adding random noise to the coordinate value of each marker was set to 0, and it was increased by 0.1 pixel for each frame, and after reaching a maximum value of 1 pixel, it was decreased by 0.1 pixel for each frame. In (iii), the initial value of the y-direction translation amount added to the image obtained by adding random noise to the coordinate values ​​of each marker was set to 0, and it was increased by 0.1 pixels for each frame. After reaching a maximum value of 1 pixel, it was decreased by 0.1 pixels for each frame. In (iv), the initial value of the rotation amount added to the image obtained by adding random noise to the coordinate values ​​of each marker was set to 0, and it was increased by 0.001 degrees for each frame. After reaching a maximum value of 0.01 degrees, it was decreased by 0.01 degrees for each frame. The reference point for rotation was set to the center of the image. In (v), the initial magnification value added to the image obtained by adding random noise to the coordinate values ​​of each marker was 1 (normal magnification), and it was increased by 0.0001 for each frame. After reaching a maximum value of 1.001, it was decreased by 0.001 for each frame. The reference point for magnification was the center of the image.

[0069] Figures 9(a) and (b) show the displacement of marker AC in the x and y directions in each measurement frame. These displacements were obtained using the sampling moiré method without correction. The conditions were: the thinning period T was 20, the marker pitch period p in the subject space was 100 mm, the cutoff frequency of the low-pass filter was 0.1, and the size of the averaging area used to average the phase difference of the moiré image was 40 × 40 pixels. As shown in Figures 9(a) and (b), the displacements in the x and y directions fluctuate significantly due to coordinate changes that mimic changes in the imaging direction. Figure 9(c) shows the x- and y-direction displacements of marker C after correction in each measurement frame, using circles and black circles, respectively. Correction is performed by subtracting the average y-coordinates of markers A and B on the image in the measurement frame. As a result, in all cases (i)-(v), the average values ​​in the x and y directions are 0.017 and -0.017, and the standard deviations are 0.011 and 0.012, respectively, which is not a significant difference. The simulation results in Figure 9(c) show that measurement accuracy can be obtained in cases where translation, rotation, or scale changes, similar to that in cases where no changes occur.

[0070] (experiment) Next, an experiment carried out on the displacement measuring device 10 according to this embodiment will be described. The primary purpose of the first experiment was to verify the alignment of images captured by a moving camera. In this experiment, images of three markers, Mk-A to Mk-C, installed on a structure were captured at regular intervals using a camera mounted on a drone. Markers Mk-A to Mk-C were arranged at equal intervals horizontally. Figure 10(a) shows an example of an image captured facing the front of the structure. Figure 10(b) is an enlarged view of the area in Figure 10(a) showing markers Mk-A to Mk-C. In the experiment, of markers Mk-A to Mk-C, marker Mk-C was used as the measurement marker, and markers Mk-A and Mk-B were used as reference markers. No external force was applied to the structure during drone flight. Therefore, ideally, the displacement of each marker would be zero.

[0071] Fig. 11 shows, with pixel accuracy, the x- and y-coordinates of the center points of markers Mk-A, Mk-C, and Mk-B in images captured at each time. In Fig. 11, each column corresponds to a marker, and each row corresponds to a coordinate. While markers Mk-A, Mk-C, and Mk-B are actually stationary, the changes in position shown in Fig. 11 indicate changes in the position of the camera used as the imaging unit 20 (image blur).

[0072] Figure 12 shows the trajectories of the centroids of markers Mk-A, Mk-C, and Mk-B in the images from the start to the end of observation, with pixel accuracy in the left column and subpixel accuracy in the right column. In Figure 12, each column corresponds to a marker. The shapes of the trajectories of each marker are similar for both pixel and subpixel accuracy. This indicates that displacements common to the entire image are significant, and that local displacements may be hidden if not corrected. Furthermore, correcting the position of each marker with subpixel accuracy allows for smoother trajectories. It is expected that subpixel accuracy measurements will enable the displacement of each marker to be obtained with high accuracy.

[0073] 13(a), (b), (c), and (d) show the x-direction translation amount Δx, the y-direction translation amount Δy, the rotation amount Δθ, and the scale factor Δs calculated from the measurement images captured at each time. Note that the initial frame (time t=0) is used as the reference frame, and frames at each subsequent time t are used as measurement frames. The translation amounts Δx, Δy, rotation amount Δθ, and scale factor Δs are calculated from the center of gravity of the fiducial marker in the measurement frame and the reference frame at each time.

[0074] Figures 14(a) and (b) show the x and y coordinates of the center of gravity of marker Mk-A at each time. The values ​​obtained with pixel accuracy and with sub-pixel accuracy are shown by thin and thick lines, respectively. With pixel accuracy, the value changes in a stepped manner over time, whereas with sub-pixel accuracy, the value changes smoothly over time. This stepped change is most noticeable in the y coordinate value, which changes relatively little over time. This shows that quantization of the position of marker Mk-A in pixel units (pixel accuracy) has a significant impact on the error caused by minute displacements.

[0075] Figure 15 shows the deflection values ​​of marker Mk-C at each time with subpixel accuracy. The deflection values ​​shown correspond to the displacement of marker Mk-C in the y direction corrected using the positions of markers Mk-A and Mk-B. The average and standard deviation of the deflection values ​​shown in Figure 15 were 0.004 mm and 0.048 mm, respectively. This result demonstrates that deflection values ​​can be measured with a high accuracy of 0.1 mm or less, even when using images captured by a camera mounted on a drone flying (hovering) in front of a structure, regardless of fluctuations in position and orientation due to flight.

[0076] FIG. 16 shows the amount of image blur at each time. The illustrated amount of image blur corresponds to the amount of displacement of the coordinates of the center of gravity of marker Mk-C at each subsequent time, based on the coordinates of the center of gravity of marker Mk-C at the initial time. FIG. 16(a) shows the trajectory of the amount of image blur obtained without correcting the position of marker Mk-C. The trajectory starts from the origin, moves approximately 20 pixels to the right, moves approximately 40 pixels to the left, and then moves toward the origin. Fig. 16(b) shows the locus of the image blur obtained by correcting the position of the marker Mk-C with pixel accuracy. After correction, the image blur converges to the vicinity of the origin. Figure 16(c) is an enlarged view of Figure 16(b). The amount of image blurring falls within a range of approximately ±3 pixels in the x direction and ±1 pixel in the y direction. Fig. 16(d) shows the trajectory of the image blur obtained by correcting the position of the marker Mk-C with sub-pixel accuracy. With sub-pixel accuracy, the image blur converges closer to the origin than with pixel accuracy. Figures 16(e) and (f) are both enlarged views of Figure 16(d). The amount of image blurring falls within the range of approximately ±0.2 pixels in the x direction and ±0.03 pixels in the y direction. 16, according to this embodiment, even when the imaging unit 20 is moving, image blurring caused by the movement is corrected with high accuracy, thereby improving the accuracy of the amount of displacement measured using the captured images.

[0077] The second experiment primarily aims to verify the displacement measured using images captured by a camera moved using a drone. Figure 17 shows the experimental optical system for aerial displacement measurement. In the example shown in Figure 17, displacement in the y direction was measured (deflection measurement) using images captured by a camera mounted on a drone. However, for comparison, displacement in the y direction was measured using images (Figure 17(a)) captured by a digital camera mounted on a tripod with a fixed-focus lens with a focal length of 35 mm. Figure 17(a) shows three markers AC. Markers A and B on the left and right of the drawing were used as reference markers, and marker C in the center was used as the measurement marker. The pitch of the patterns of markers AC in the x and y directions was 50 mm. Marker C was fixed to a moving stage, and the height of the moving stage (displacement in the y direction (opposite to the vertical direction, sometimes called "upward")) was controlled by the operator using a stage controller. The distance from the camera or drone to marker C was approximately 7 m, and the distance between markers A and C and between markers B and C were both 3.1 m. This is expected to measure deflection at the center of a bridge with a total length of 6.2 m.

[0078] In this experiment, a video (resolution: 3840 pixels x 2160 pixels) was captured at a height of approximately 2.5 m using a camera mounted on a small drone. The duration of each capture was approximately 50 seconds. For the first 10 seconds, there was no displacement in the y direction, and then the drone was raised over 5 seconds until the displacement in the y direction reached 5 mm. The height was then maintained for approximately 20 seconds, and the drone was raised over another 5 seconds until the displacement in the y direction reached 10 mm, after which the height was maintained for 10 seconds. Figures 17(b) and 17(c) show an image taken with a displacement of 0 at the start of imaging (0 seconds) while the small drone was flying, and an image taken after deformation with a displacement of 5 mm 25 seconds after imaging began. In this case, the camera is positioned facing downward from above, rather than directly in front of, the object being measured, marker C, and a tilted video is taken. The positions of the left and right arrows in Figures 17(b) and (c) indicate that the image position is shifted up, down, left, and right. The difference in the position of the central arrow in Figures 17(b) and (c) includes the effect of parallax due to differences in the height of the drone. This shows that the drone does not necessarily have to be positioned directly in front of the structure being photographed.

[0079] Figure 18 shows the experimental results of deflection measurement using aerial photography. Figure 18(a) shows the trajectories of each marker's position in the xy plane measured with subpixel accuracy. These trajectories show that the position changes over time due to the hovering of a small drone. Figures 18(b) and 18(c) show the time series of displacement in the y direction measured using images captured by a camera mounted on a drone and a camera fixed to a tripod, respectively. The known displacement amounts are 0 mm from the start of imaging until 10 seconds after the start of imaging, 5 mm from 15 to 35 seconds after the start of imaging, and 10 mm after 40 seconds. The displacement amounts using images captured using a drone-mounted camera using this embodiment (Figure 18(b)) were 0.095 mm from the start of imaging until 10 seconds after the start of imaging, 4.744 mm from 15 to 35 seconds after the start of imaging, and 10.540 mm after 40 seconds. The displacement amount analyzed using the conventional method using images captured with a camera fixed to a tripod (Figure 18(c)) was 0.034 mm for 10 seconds after the start of imaging, 5.035 mm for 15 to 35 seconds, and 10.051 mm for 40 seconds or more. These experimental results confirmed that even when using images captured with a moving camera according to this embodiment, accurate displacement amounts can be obtained, just as when using images captured with a fixed camera. This experiment verified that this embodiment can achieve high-precision deflection measurement of structures using images captured by aerial drone photography.

[0080] (When there are three or more reference markers) Next, an embodiment using three or more reference markers will be described. The above-described AB correction requires that a measurement marker be placed on a straight line between two reference markers in the reference frame. Depending on the placement of the reference markers, it may not be possible to place the measurement marker at a position where the amount of displacement is to be measured. Furthermore, the larger the angle between the direction of the measurement marker from the imaging unit 20 and the direction of the optical axis of the imaging unit 20 (i.e., the imaging direction), the more significant the distortion of the shape of the measurement marker that appears in the captured image tends to be. Therefore, it is desirable to relax the constraints on the position of the imaging unit 20 and the placement of each marker.

[0081] Therefore, the first image correction unit 144 or the second image correction unit 148 uses an image representing three or more fiducial markers in each frame to calculate transformation parameters for coordinate transformation of the coordinates in the measurement frame so as to align the positions of the individual fiducial markers between the reference frame and the measurement frame. The first image correction unit 144 or the second image correction unit 148 corrects the positions of the measurement markers by performing coordinate transformation using the calculated transformation parameters. When three or more fiducial markers are used, the positions of the measurement markers may be any positions in the image of each frame as long as they are different from the positions of the fiducial markers. This relaxes the constraints on the positions of the measurement markers. More specifically, the positions of the measurement markers do not have to be on a straight line between each pair of fiducial markers as in AB correction, nor do they have to be within a polygon whose vertices are the positions of three or more fiducial markers. Furthermore, even if the direction of the measurement markers from the imaging unit 20 is obliquely intersecting the frontal direction, shear distortion (skew, shear) of the shape of the measurement markers appearing in the captured image is compensated for, thereby suppressing a decrease in measurement accuracy. When the number of reference markers is three, the first image correction unit 144 or the second image correction unit 148 can use a linear transformation such as an affine transformation as a coordinate transformation method, in which shear distortion is also taken into account in addition to translation, rotation, and scale.

[0082] When the number of reference markers is four, the first image correcting unit 144 or the second image correcting unit 148 can use homography transformation as the coordinate transformation method. Homography transformation is a linear transformation that converts the four vertices P1-P4 of a quadrangle P into vertices Q1-Q4 of a corrected quadrangle Q, each of which has a different shape (see FIG. 19), and is also called trapezoidal transformation. Point P5 in quadrangle P is transformed into Q5 in quadrangle Q by homography transformation. When point P5 is transformed into point Q5, the positional relationship with each vertex is maintained, and displacement due to more complex deformation is compensated for by a simple calculation.

[0083] The first image correcting unit 144 or the second image correcting unit 148 may determine coordinate transformation parameters to be used for the homography transformation so that the positions Q1 to Q4 of the four reference markers Mk-1 to Mk-4 in the corrected images of each frame are common between frames. The first image correcting unit 144 or the second image correcting unit 148 can perform homography transformation on the coordinates (x, y) indicating the position P5 of the measurement marker Mk-5 before correction using the determined coordinate transformation parameters, and calculate the coordinates (u, v) of the position Q5 of the measurement marker Mk-5 after correction.

[0084] For example, when an initial frame is used as a reference frame, the positions of the four reference markers Mk-1 to Mk-4 shown in the reference frame are set as reference positions in the first image corrector 144 or the second image corrector 148. The first image corrector 144 or the second image corrector 148 sets each of the other frames as measurement frames and determines coordinate transformation parameters for converting the positions of the four reference markers Mk-1 to Mk-4 shown in each measurement frame to the reference positions. Using the determined coordinate transformation parameters, the first image corrector 144 or the second image corrector 148 determines the corrected position of measurement marker Mk-5 by applying equation (3) to the position of measurement marker Mk-5 shown in the measurement frame.

[0085] Equation (3) shows coordinate transformation from coordinates (x, y) to coordinates (u, v). In equation (3), a to h are real numbers corresponding to coordinate transformation parameters. Equation (4) may be used instead of equation (3). The coordinate transformation parameters a to h satisfy both the equalities of equations (5) and (6).

[0086]

number

[0087]

number

[0088]

number

[0089]

number

[0090] Therefore, the first image correcting unit 144 or the second image correcting unit 148 can calculate coordinate transformation parameters a to h between the coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) indicating the positions of the four fiducial markers in the measurement frame and the coordinates (u1, v1), (u2, v2), (u3, v3), and (u4, v4) of the corresponding reference positions so as to satisfy equations (5) and (6). More specifically, the first image correcting unit 144 or the second image correcting unit 148 can solve eight simultaneous equations (equation (7)) obtained by substituting the coordinates before and after correction for each fiducial marker into equations (5) and (6), and calculate the coordinate transformation parameters a to h using a technique such as Gaussian elimination.

[0091]

number

[0092] Next, an example of displacement measurement processing using homography transformation will be described. In the following description, the position of a reference marker in a measurement frame is corrected with sub-pixel accuracy, and the position of the measurement marker in the measurement frame is corrected based on coordinate transformation from the position of the reference marker in the reference frame to the position of the reference marker in the corrected measurement frame.

[0093] Fig. 20 is a flowchart showing a technique using homography transformation as a third example of the displacement measurement process according to this embodiment. When executing the process shown in Fig. 20, similarly to the example in Fig. 4, the calculation processing unit 14 sets a reference frame that is used as a reference for the amount of displacement, and other frames are used as measurement frames to be corrected. (Step S202) The marker detection unit 142 detects a marker for each frame indicated in the image data input from the imaging unit 20. The marker detection unit 142 determines the coordinates of the center of gravity of each detected marker as the central coordinates. (Step S204) The first image corrector 144 determines coordinate transformation parameters for homography transformation from the position of each reference marker in the measurement frame to the position of a corresponding reference marker in the reference frame, using the arrangement of the four reference markers in the reference frame and the arrangement of the four reference markers in a measurement frame different from the reference time. The first image corrector 144 performs homography transformation on the positions of the measurement markers in the measurement frame using the determined coordinate transformation parameters, and determines the corrected positions with pixel accuracy (coarse correction). (Step S206) The first displacement amount calculation unit 146 uses the sampling moiré method to analyze the phase difference between the moiré image of the measurement marker in the reference frame and the measurement marker whose position in the measurement frame has been corrected, and calculates the displacement amount of the actual measurement marker as the first displacement amount from the analyzed phase difference.

[0094] (Step S208) For each reference marker, the second image correcting unit 148 analyzes the phase difference between the moiré image of the reference frame and the moiré image in the measurement frame after the position has been corrected with pixel accuracy. Based on the analyzed phase difference, the second image correcting unit 148 calculates coordinate transformation parameters for homography transformation from the measurement frame to the reference frame. Using the calculated coordinate transformation parameters, the second image correcting unit 148 performs homography transformation on the position of the measurement marker in the measurement frame after the correction with pixel accuracy, and determines the position of the measurement marker after the correction with subpixel accuracy. (Step S210) The second displacement amount calculation unit 150 uses the sampling moiré method to analyze the phase difference between the measurement marker in the reference frame and the moiré image of the measurement marker whose position in the measurement frame has been corrected with sub-pixel accuracy, and calculates the displacement amount of the actual measurement marker as the second displacement amount from the analyzed phase difference. (Step S212) The displacement amount output unit 152 selects either the first displacement amount obtained in step S206 or the second displacement amount obtained in step S210. The displacement amount output unit 152 causes the display unit 16 to display information on the selected displacement amount.

[0095] (Image correction device) In the above description, the displacement measuring system 1 and the displacement measuring device 10 are exemplified as embodiments, but they may also be implemented in the form of an image correction system 3 and an image correction device 30. In the following description, differences from the above embodiment will be mainly discussed, and unless otherwise specified, the same reference numerals will be used and the above description will be applied.

[0096] FIG. 21 is a schematic block diagram showing an example of the functional configuration of an image correction system 3 according to this embodiment. The image correction system 3 includes an image correction device 30 and an imaging unit 20. The image correction device 30 includes a parameter input unit 12, a calculation processing unit 34, and a display unit 16. The calculation processing unit 34 includes a marker detection unit 142, a first image correction unit 144, and a second image correction unit 148. The calculation processing unit 34 illustrated in FIG. 21 is similar to the calculation processing unit 14 illustrated in FIG. 3 except that the first displacement amount calculation unit 146, the second displacement amount calculation unit 150, and the displacement amount output unit 152 are omitted. The first image correction unit 144 converts the coordinates of each position in the measurement frame to compensate for the displacement of each fiducial marker between the measurement frame and the reference frame, thereby obtaining an image of the measurement frame whose position has been corrected with pixel accuracy. The second image correction unit 148 converts the coordinates of each position in the measurement frame so that the displacement corresponding to the phase difference between the moiré image of the reference marker in the image of the measurement frame whose position has been corrected with pixel accuracy and the moiré image of the reference marker in the image of the reference frame is compensated for, thereby obtaining an image of the measurement frame whose position has been corrected with sub-pixel accuracy.

[0097] This configuration allows for image registration so that the positions of multiple reference markers match between frames with subpixel accuracy, even when the imaging unit 20 is mounted on a moving object. Registration can be applied to monitoring various surface conditions based on the time-varying luminance distribution in a predetermined evaluation area of ​​the captured image. Furthermore, the image is not limited to a visible image based on visible light emitted from the subject, but may also be an infrared image, an ultraviolet image, an X-ray image, or the like. Furthermore, by applying a mechanoluminescent material to the evaluation area in advance, the pattern that appears due to mechanoluminescence (a mechanoluminescent image) can be used to analyze stress distribution.

[0098] Next, an example of image alignment by the image correction device 30 will be described. FIG. 22 is an explanatory diagram of a first example of image alignment. In the example of FIG. 22, four reference markers Mk-A to Mk-D are placed on the surface of a subject within the imaging region (field of view) of the imaging unit 20, and the example is applied to observing changes in the shape of cracks (fissures) on a wall surface as an example of an observation object that appears on the surface within an evaluation region included in the imaging region. In this case, the time evolution of the length, width, branching, etc. of the cracks in the evaluation region can be observed using images captured for each frame. The evaluation region may be any region that is included in the imaging region in each frame during the observation period.

[0099] FIG. 23 is an explanatory diagram of a second example of image alignment. FIG. 23 shows an application example of connecting images of multiple frames into one image frame. The first image correction unit 144 and the second image correction unit 148 acquire images of multiple frames and correct the positions within the acquired frames so that the positions of two or more reference frames commonly included in the acquired images match. This makes it possible to spatially connect images of multiple frames with subpixel accuracy. FIG. 23(a) illustrates two images of frames captured at different times. Image A of one frame represents field of view A including markers Mk-A1, Mk-B1, Mk-B2, and Mk-A2, and image B of the other frame represents field of view B including markers Mk-C1, Mk-B1, Mk-B2, and Mk-C2. The first image correction unit 144 and the second image correction unit 148 can form an image with a larger area by connecting image A and image B so that the positions of markers Mk-B1 and Mk-B2, which are common markers to both frames, match. The multiple frames may be captured simultaneously using separate imaging units 20, or may be captured at different times using a single imaging unit 20. Furthermore, the first image correction unit 144 and the second image correction unit 148 may sequentially connect pairs of two-frame images that commonly include two or more reference markers in their fields of view, and form an image with a larger area in one frame from images of three or more frames.

[0100] Figure 24 is an explanatory diagram illustrating the experimental results of image alignment correction. In the experiment, images captured aerially at regular intervals using an imaging unit 20 mounted on a small drone during flight were used for alignment. Figure 24(a) shows an image obtained by simply overlaying and synthesizing the first frame image and the 90th frame image. Figure 24(a) shows that the positions of common figures and characters appearing in two images captured at different times before correction are significantly different. Figure 24(b) shows an image obtained by synthesizing a corrected image obtained by performing AB correction on the image of the later frame of two images using two markers Mk-A and Mk-B appearing on the left and right ends of each frame image with an image captured in an earlier frame. Figure 24(b) shows that there is no difference in the placement of common figures and characters between the two images captured at different times, and the patterns in the two images almost perfectly overlap. This allows for blur correction of images captured by a drone, and as an example, makes it possible to easily detect changes in image information (the text "NEW NORMAL!" in the example of FIG. 24) in a display area that the user desires to observe (for example, an evaluation area). This embodiment can be applied not only to images captured using visible light with a general camera, but also to images captured using an infrared camera and mechanoluminescent images, and can be used for various image evaluations in the evaluation and diagnosis of the soundness of structures.

[0101] As described above, the displacement measuring device 10 according to this embodiment includes image correction units (e.g., first image correction unit 144, second image correction unit 148) that acquire, for each frame, images representing two or more reference markers used as a basis for the amount of displacement and measurement markers representing patterns that are spatially repeated at a constant pitch from the imaging unit 20, and correct the positions of the measurement markers to compensate for the displacement of the reference markers between frames.The displacement measuring device 10 also includes displacement amount calculation units (first displacement amount calculation unit 146, second displacement amount calculation unit 150) that calculate the amount of displacement of the measurement markers from the phase difference between frames of moiré images generated from the patterns of the measurement markers. With this configuration, even if the position or orientation of the imaging unit 20 fluctuates from frame to frame, the change in each coordinate in the captured image is compensated for, so the amount of displacement of the part where the measurement marker is placed can be measured without losing accuracy from images captured at different times.

[0102] In addition, in an image of a reference frame, which is a frame used as a reference for the amount of displacement, the position of the measurement marker is on a straight line passing between the two reference markers, and the image correction unit may analyze the scale, rotation, and translational movement amount of the image of the measurement frame from the reference frame based on the respective positions of the two reference markers in the reference frame and the respective positions of the two reference markers in the image of the measurement frame, which is a frame used to measure the amount of displacement, and correct the position of the measurement marker in the measurement frame based on the scale, rotation, and translational movement amount. This configuration compensates for changes in size, rotation, and translation of the subject image between frames based on the positions of the two reference markers, thereby compensating for fluctuations in the position and orientation of the imaging unit 20 through simple calculations.

[0103] In addition, in an image of a reference frame, which is a frame used as a reference for the amount of displacement, the position of the measurement marker is an internal division point between two reference markers, and the image correction unit may correct the position of the measurement marker in an image of a measurement frame, which is a frame used to measure the amount of displacement, based on the position of the internal division point in the measurement frame. This configuration compensates for changes in the position of the measurement marker between frames based on the positions of the two reference markers, thereby compensating for fluctuations in the position and orientation of the imaging unit 20 with simple calculations without sacrificing accuracy.

[0104] Alternatively, the number of reference markers may be three or more, and the image corrector may calculate transformation parameters for coordinate transformation that causes the positions of the reference markers to match between frames, and correct the positions of the measurement markers using the transformation parameters. With this configuration, changes in the position of the measurement marker that occur between frames are compensated for using the positions of three or more reference markers as references. The position of the measurement marker can be set arbitrarily within the field of view of the imaging unit 20, and changes in the position of the measurement marker are compensated for even if the direction from the imaging unit 20 to the measurement marker intersects with the imaging direction of the imaging unit 20. This reduces the degree of freedom in the installation positions of the measurement marker and the imaging unit 20.

[0105] In addition, the number of reference markers is four, and the coordinate transformation may be a homography transformation. With this configuration, more complex image deformation can be corrected based on the positions of the reference markers than when the number of reference markers is three. Therefore, correction with higher tolerance to noise, the optical system of the imaging unit 20, etc. can be achieved.

[0106] In addition, each reference marker represents a pattern that is spatially repeated at a constant pitch, and an image correction unit (e.g., second image correction unit 148) may further correct the position of the measurement marker to compensate for the phase difference between frames of the moire image generated from the reference marker pattern. According to this configuration, by using a moiré image in which the brightness distribution representing the pattern of the reference marker is enlarged, the position of the reference marker can be corrected with precision finer than a pixel, thereby improving the precision of the displacement amount of the measured reference marker.

[0107] The displacement measuring system 1 may also include a moving body (for example, a drone) on which the imaging unit 20 is installed. According to this configuration, even in an environment where the imaging unit 20 cannot be fixed, the amount of displacement at the site where the measurement marker is placed can be measured with high accuracy using the image captured by the imaging unit 20.

[0108] (Variation) The displacement measuring system 1 and the image correction system 3 according to the above-described embodiments may be modified as follows: The imaging unit 20 and the displacement measuring device 10 or the image correction device 30 may be connected via a wired or wireless network. The displacement measuring device 10 or the image correction device 30 does not necessarily have to be integrated with the parameter input unit 12 and the display unit 16. In the displacement measuring device 10, one or both of the parameter input unit 12 and the display unit 16 may be omitted. Furthermore, the displacement measuring device 10 or the image correcting device 30 may include the imaging unit 20 and be configured as a single displacement measuring device 10 or image correcting device 30 .

[0109] In the above description, the first image corrector 144 and the second image corrector 148 in the displacement measuring device 10 or the image corrector 30 mainly correct the positions of the measurement markers in the measurement frames so that the positions of the individual reference markers in the measurement frames coincide with the positions of the corresponding reference markers in the reference frames, but this is not limiting. The first image corrector 144 and the second image corrector 148 may treat each time frame as a measurement frame and correct the positions of the measurement markers for each measurement frame so that the positions of the individual reference markers for each measurement frame coincide with the positions of the corresponding reference markers in a reference frame separate from the measurement frame.

[0110] Although the first displacement amount calculation unit 146 and the second displacement amount calculation unit 150 use luminance values ​​as the signal values ​​for each pixel in the above example, the present invention is not limited to this. The first displacement amount calculation unit 146 and the second displacement amount calculation unit 150 may use color signal values, such as signal values ​​for each color such as red, green, and blue, or a combination of these signal values, as the signal values ​​for each pixel. The first image correcting section 144 and the second image correcting section 148 may perform AB correction using two predetermined reference markers out of the three or more reference markers. In the above description, the center of gravity is used as an example of a representative point representing the position of each marker, but this is not limiting. The representative point may be a predetermined vertex, for example, the vertex at the bottom left corner of the drawing.

[0111] In the displacement measuring device 10 or the image correcting device 30, the first image correcting section 144 and the second image correcting section 148 may be integrated into a single image correcting section. In the displacement measuring device 10, the first displacement amount calculation section 146 and the second displacement amount calculation section 150 may be integrated into a single displacement amount calculation section.

[0112] In addition, the second image corrector 148 and the second displacement amount calculator 150 may be omitted from the displacement measuring device 10. In that case, correction with sub-pixel accuracy is not performed. The signal values ​​of the pattern displayed on the reference marker do not necessarily have to fluctuate regularly. The first displacement amount calculator 146 outputs information on the first displacement amount calculated with pixel accuracy to the display unit 16 as information indicating the displacement amount of the measurement marker. In addition, one or both of the parameter input unit 12 and the display unit 16 may be omitted. Various parameters used to calculate the displacement amount may be set in advance or may be input from another device. Information on the first displacement amount calculated by the first displacement amount calculation unit 146 and the displacement amount acquired by the displacement amount output unit 152 may be stored in the device itself or output to another device.

[0113] In the above explanation, it is assumed that the pitch of the repeating pattern displayed on each marker is the same for all markers, but it may be different for each marker. For example, the pitch may be larger for measurement points located farther away from the image capture unit 20. In this case, the period of the pattern displayed on the image does not become smaller even if the distance from the image capture unit 20 increases, so it is possible to prevent or mitigate degradation of measurement accuracy due to the distance from the image capture unit 20.

[0114] Note that a portion of the displacement measuring device 10 or image correction device 30 in the above-described embodiments, such as the arithmetic processing unit 14, may be implemented by a computer. In this case, a program for implementing this control function may be recorded on a computer-readable recording medium and loaded into a computer system for execution. The term "computer system" as used herein refers to a computer system built into the displacement measuring device 10 or image correction device 30, including hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be a program for implementing part of the above-described functions, or may be a program that can be implemented in combination with a program already stored in the computer system. Furthermore, part or all of the displacement measuring device 10 or image correction device 30 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the displacement measuring device 10 or image correction device 30 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0115] The embodiments of the present invention have been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope of the gist of the present invention. [Explanation of symbols]

[0116] 1...displacement measurement system, 3...image correction system, 10...displacement measurement device, 12...parameter input section, 14, 34...arithmetic processing section, 16...display section, 30...image correction device, 142...marker detection section, 144...first image correction section, 146...first displacement amount calculation section, 148...second image correction section, 150...second displacement amount calculation section, 152...displacement amount output section

Claims

1. acquiring, from an imaging unit, images representing a plurality of markers each representing a spatially repeated pattern at a constant pitch for each frame; correcting the image to convert it into a first corrected image so as to compensate for the displacement of the marker between a reference frame, which is a frame used as a reference for the displacement of the marker, and a measurement frame, which is a frame used to measure the displacement of the marker, with accuracy of pixel spacing; an image correction unit that generates a moiré image from the pattern of the marker appearing in the first corrected image so that a phase period is enlarged compared to a luminance period, and corrects the first corrected image to convert it into a second corrected image so as to compensate for a phase difference between the reference frame and the measurement frame of the moiré image; a displacement amount calculation unit that calculates a displacement amount of the marker from a phase difference between the reference frame and the measurement frame of a moire image appearing in the second corrected image, two markers constituting a part of the plurality of markers are reference markers serving as a reference for a displacement amount, and another marker constituting a part of the plurality of markers are measurement markers; In the image of the reference frame, the position of the measurement marker is an internal division point between two of the reference markers, The image correction unit corrects coordinates of the positions of the measurement markers in the first corrected image and the second corrected image of the measurement frame by subtracting coordinates of the positions of the internal division points in the measurement frame. Displacement measuring device.

2. On the computer, In order to function as the displacement measuring device according to claim 1, program.

3. The imaging unit and the displacement measuring device according to claim 1 are provided. system.

4. a moving body on which the imaging unit is installed; The system of claim 3.

5. The displacement measuring device Acquiring an image representing a plurality of markers each representing a spatially repeated pattern at a constant pitch for each frame; correcting the image to convert it into a first corrected image so as to compensate for the displacement of the marker between a reference frame, which is a frame used as a reference for the displacement of the marker, and a measurement frame, which is a frame used to measure the displacement of the marker, with accuracy of pixel spacing; an image correction step of generating a moiré image from the pattern of the markers appearing in the first corrected image so that the period of the phase is enlarged more than the period of the luminance, and correcting the first corrected image so as to compensate for the phase difference between the reference frame and the measurement frame of the moiré image, thereby converting it into a second corrected image; a displacement amount calculation step of calculating a displacement amount of the marker from a phase difference between the reference frame and the measurement frame of a moire image appearing in the second corrected image; Run two markers constituting a part of the plurality of markers are reference markers serving as a reference for a displacement amount, and another marker constituting a part of the plurality of markers are measurement markers; In the image of the reference frame, the position of the measurement marker is an internal division point between two of the reference markers, The image correction step corrects coordinates of the positions of the measurement markers in the first corrected image and the second corrected image of the measurement frame by subtracting coordinates of the positions of the internal division points in the measurement frame. Displacement measurement method.

Images