Correction parameter calculation method, displacement amount calculation method, correction parameter calculation device, and displacement amount calculation device

Abstract

The correction parameter calculation method includes: a first acquisition step (S11) of acquiring first image data obtained by imaging an object (60) from a first imaging device (10); a second acquisition step (S12) of acquiring second distance data from the object (60) and second image data obtained by imaging the object (60) from a second imaging device (20); a third acquisition step (S21) of acquiring displacement direction information indicating the direction of displacement of the object in three dimensions; a correspondence establishment step (S13) of establishing correspondence of the positions of the object (60) in the first and second image data; an estimation step (S14) of estimating the position of the first imaging device (10); a distance calculation step (S15) of calculating first distance data from the first imaging device (10) to the object (60); and a parameter calculation step (S16) of calculating a correction parameter for converting a pixel displacement amount of the object (60) into an actual size displacement amount using the first distance data and the displacement direction information.

Description

Technical Field

[0001] The present disclosure relates to a correction parameter calculation method, a displacement amount calculation method, a correction parameter calculation device, and a displacement amount calculation device for calculating a correction parameter for measuring an actual size value representing the displacement of an object's motion, namely, an actual size displacement amount, using an image. Background Art

[0002] Conventionally, an imaging device has been disclosed that uses image data obtained by capturing an object with a camera and distance measurements to the object from a distance measuring device such as a laser distance meter to non-contactly measure the state of the object's surroundings (see Patent Document 1). For example, if the object is a bridge, the state may be the amount of deflection of the bridge.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 5281610 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] However, in order to accurately measure the state of an object, the imaging device of Patent Document 1 requires the camera and the distance measuring device to be arranged so that the optical axis of the camera is parallel to the optical axis of the distance measuring device. Therefore, Patent Document 1 discloses the use of a mounting fixture for mounting the camera and the distance measuring device so that the optical axis of the camera is parallel to the optical axis of the distance measuring device. Thus, the imaging device of Patent Document 1 requires the preparation of a mounting fixture to accurately measure the state of an object, making it difficult to accurately and easily measure the actual dimensional value of displacement.

[0008] Therefore, the present disclosure relates to a correction parameter calculation method and the like that can easily and accurately convert the displacement amount into an actual size value when measuring displacement using an image.

[0009] Means for solving problems

[0010] A correction parameter calculation method according to one embodiment of the present disclosure calculates correction parameters for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing the displacement of an object's motion, the correction parameter calculation method comprising: a first acquisition step of acquiring, from a first camera device, first image data obtained by photographing the object; a second acquisition step of acquiring, from the second camera device, second distance data from a second camera device disposed at a position different from that of the first camera device to the object, and second image data obtained by photographing the object; a third acquisition step of acquiring displacement direction information representing the direction of displacement of the object in three dimensions; and a corresponding step of converting the first image data into a second image data. a step of estimating the position of the first imaging device relative to the second imaging device based on the correspondence result and the second distance data; a step of calculating the first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; and a step of calculating the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information. The pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first imaging device at different timings.

[0011] A correction parameter calculation method according to one embodiment of the present disclosure calculates correction parameters for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation method comprising: a first acquisition step of acquiring, from a first imaging device, first image data obtained by imaging the object; a second acquisition step of acquiring, from the second imaging device, second distance data from a second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; a third acquisition step of acquiring a distance data representing the object in three dimensions. a step of detecting the shape of the first imaging device in the second image data; an estimating step of estimating the position of the first imaging device based on the detection result; a distance calculating step of calculating first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; and a parameter calculating step of calculating the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information. The pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first imaging device at different timings.

[0012] In addition, a correction parameter calculation method according to one embodiment of the present disclosure calculates a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing the displacement of a motion of an object, the correction parameter calculation method comprising: a first acquisition step of acquiring, from a first imaging device, first image data obtained by imaging the object; a second acquisition step of acquiring, from the second imaging device, second distance data from a second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; and a third acquisition step of acquiring data representing the displacement of the object in three dimensions. a step of detecting the shape of the first camera device in the second image data; an estimating step of estimating the position and posture of the first camera device based on the detection result; a distance calculating step of calculating first distance data from the first camera device to the object based on the position and posture of the first camera device and the second distance data; and a parameter calculating step of calculating the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information. The pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first camera device at different timings.

[0013] The displacement calculation method involved in one embodiment of the present invention includes: a fourth acquisition step of acquiring the correction parameter calculated by using the above-mentioned correction parameter calculation method; a fifth acquisition step of acquiring the two or more third image data; and a conversion step of converting the pixel displacement in the two or more third image data into the actual size displacement based on the correction parameter.

[0014] A correction parameter calculation device according to one embodiment of the present disclosure calculates correction parameters for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing the displacement of a motion of an object, the correction parameter calculation device comprising: a first acquisition unit for acquiring, from a first camera device, first image data obtained by photographing the object; a second acquisition unit for acquiring, from the second camera device, second distance data from a second camera device disposed at a position different from that of the first camera device to the object, and second image data obtained by photographing the object; a third acquisition unit for acquiring displacement direction information representing the direction of displacement of the object in three dimensions; a matching unit for matching the first image data with the second image data; a position estimating unit that estimates the position of the first imaging device relative to the second imaging device based on the correspondence result and the second distance data; a distance calculating unit that calculates first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; and a parameter calculating unit that calculates the correction parameter for converting a pixel displacement amount of a measurement point of the object into an actual size displacement amount using the first distance data and the displacement direction information, the pixel displacement amount of the measurement point of the object being based on two or more third image data captured by the first imaging device at different timings.

[0015] In addition, a correction parameter calculation device according to one embodiment of the present disclosure calculates a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation device comprising: a first acquisition unit for acquiring, from a first imaging device, first image data obtained by imaging the object; a second acquisition unit for acquiring, from the second imaging device, second distance data from a second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; a third acquisition unit for acquiring a distance representing the object in three dimensions. a position estimating unit that detects the shape of the first imaging device in the second image data and estimates the position of the first imaging device based on the detection result; a distance calculating unit that calculates first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; and a parameter calculating unit that calculates the correction parameter for converting a pixel displacement amount of a measurement point of the object into an actual size displacement amount using the first distance data and the displacement direction information. The pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first imaging device at different timings.

[0016] A displacement amount calculation device involved in one embodiment of the present disclosure includes: a fourth acquisition unit, which acquires the correction parameter calculated by using the above-mentioned correction parameter calculation device; a fifth acquisition unit, which acquires the two or more third image data; and a conversion unit, which converts the pixel displacement in the two or more third image data into the actual size displacement based on the correction parameter.

[0017] Effects of the Invention

[0018] According to the correction parameter calculation method and the like according to one embodiment of the present disclosure, the displacement amount can be easily converted into an actual size value with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram showing a schematic configuration of a displacement measurement system according to the first embodiment.

[0020] Figure 2 This is a block diagram showing the functional configuration of the displacement measurement system according to the first embodiment.

[0021] Figure 3 This is a flowchart showing the operation of the displacement measurement system according to the first embodiment.

[0022] Figure 4This is a diagram for explaining feature point matching in the matching unit according to the first embodiment.

[0023] Figure 5 This is a diagram for explaining a method of converting displacement into actual size according to the first embodiment.

[0024] Figure 6 This is a flowchart showing the operation of the displacement measurement system according to the modification of the first embodiment.

[0025] Figure 7A This is a first diagram for explaining displacement correction taking the displacement direction into consideration according to a modification of the first embodiment.

[0026] Figure 7B FIG. 2 is a second diagram for explaining displacement correction taking the displacement direction into consideration according to a modification of the first embodiment.

[0027] Figure 8 This is a diagram for explaining a method of converting displacement into actual size according to a modification of the first embodiment.

[0028] Figure 9 This is a diagram showing a schematic configuration of a displacement measurement system according to the second embodiment.

[0029] Figure 10 This is a block diagram showing the functional configuration of a displacement measurement system according to the second embodiment.

[0030] Figure 11 This is a diagram showing an example of a mark attached to the first imaging device according to the second embodiment.

[0031] Figure 12 This is a flowchart showing the operation of the displacement measurement system according to the second embodiment. DETAILED DESCRIPTION

[0032] (Process leading to this disclosure)

[0033] As described above, in order to accurately measure the state of an object, the imaging device described in Patent Document 1 requires the camera and distance measuring device to be mounted so that the optical axis of the camera is parallel to the optical axis of the distance measuring device. In Patent Document 1, a dedicated mounting device is used to maintain the optical axis of the camera parallel to the optical axis of the distance measuring device.

[0034] However, when using images to measure displacement, it is desirable to be able to easily and accurately measure the actual dimensional value of a physical quantity such as the displacement amount. For example, it is desirable to be able to accurately measure displacement even when the optical axis of the camera is not parallel to the optical axis of the distance measuring device, as described in Patent Document 1. Furthermore, it is desirable to be able to easily measure the actual dimensional value of displacement without, for example, requiring specialized mounting equipment.

[0035] Another method for measuring the actual size of displacement involves capturing an image of a specific portion of the object whose displacement is to be measured, whose length is known. Based on the number of pixels in the image corresponding to the specific portion and the known length, a correction value is calculated to convert a single pixel into the actual size. However, in this method, the specific portion is preferably a flat surface. This means that this method is limited by the shape of the specific portion of the object, making it difficult to measure the actual size of the displacement of a measurement point on an object with a complex shape.

[0036] Furthermore, while this method can accurately measure the actual dimensional value of displacement at the specific portion, it is difficult to accurately measure the actual dimensional value of displacement at portions other than the specific portion. This is because, if the distance from the imaging device to the specific portion differs from the distance from the imaging device to portions other than the specific portion, the appropriate correction value for each portion will differ. Furthermore, this method cannot measure the actual dimensional value of displacement unless the length of the specific portion is known.

[0037] Therefore, the inventors diligently studied the correction parameter calculation method, etc., which has fewer constraints on the installation of the camera and distance measuring device and the constraints caused by the shape of the object, that is, for easily and accurately measuring the actual size value of the displacement, and proposed the correction parameter calculation method described below.

[0038] A correction parameter calculation method according to one embodiment of the present disclosure calculates correction parameters for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing the displacement of a motion of an object, the correction parameter calculation method comprising: a first acquisition step of acquiring, from a first camera device, first image data obtained by photographing the object; a second acquisition step of acquiring, from the second camera device, second distance data from a second camera device disposed at a position different from that of the first camera device to the object, and second image data obtained by photographing the object; and a corresponding step of mapping the position of the object in the first image data. a step of establishing a correspondence with the position of the object in the second image data; an estimating step of estimating the position of the first camera device relative to the second camera device based on the correspondence result and the second distance data; a distance calculating step of calculating first distance data from the first camera device to the object based on the position of the first camera device and the second distance data; and a parameter calculating step of calculating the correction parameter for converting a pixel displacement amount of a measurement point of the object into an actual size displacement amount using the first distance data, the pixel displacement amount of the measurement point of the object being based on two or more third image data captured by the first camera device at different timings.

[0039] Thus, the correction parameter calculation method allows calculation of the first distance from the first imaging device to the object based on the estimated position of the first imaging device, eliminating the need to position the first and second imaging devices so that the optical axes of the first and second imaging devices are parallel. Furthermore, the correction parameter calculation method allows calculation of the correction parameter based on the first distance, making it possible to calculate the correction parameter used to convert pixel displacement into actual size displacement even when the length of a specific portion of the object is unknown.

[0040] Furthermore, according to the correction parameter calculation method, by estimating the first distance, the distance from the first imaging device to the object can be accurately obtained. Therefore, according to the correction parameter calculation method, by calculating the correction parameters using the first distance, correction parameters can be obtained that can more accurately calculate the actual size displacement. For example, compared to using the second distance, correction parameters can be obtained for calculating the actual size displacement with higher accuracy. Thus, according to the correction parameter calculation method of the present disclosure, by calculating the displacement using the correction parameters calculated using this correction parameter calculation method, it is possible to easily and accurately convert the displacement into an actual size value when measuring displacement using images.

[0041] A correction parameter calculation method according to one embodiment of the present disclosure calculates correction parameters for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation method comprising: a first acquisition step of acquiring, from a first imaging device, first image data obtained by imaging the object; a second acquisition step of acquiring, from the second imaging device, second distance data from a second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; a step of detecting the shape of the first imaging device in the second image data; an estimating step of estimating the position of the first imaging device based on the detection result; a distance calculating step of calculating first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; and a parameter calculating step of calculating, using the first distance data, the correction parameter for converting a pixel displacement amount of a measurement point of the object into an actual size displacement amount, the pixel displacement amount of the measurement point of the object being based on two or more third image data captured by the first imaging device at different timings.

[0042] This allows estimation of the position information of the first imaging device using only the second image data and the distance data, making it possible to estimate the position information of the first imaging device without being affected by the position or number of feature points obtained from the image. In other words, the position information of the first imaging device can be easily estimated.

[0043] Furthermore, according to the correction parameter calculation method, by estimating the first distance, the distance from the first imaging device to the object can be accurately obtained. Therefore, according to the correction parameter calculation method, by calculating the correction parameters using the first distance, correction parameters can be obtained that can more accurately calculate the actual size displacement. For example, compared to using the second distance, correction parameters can be obtained for calculating the actual size displacement with higher accuracy. Thus, according to the correction parameter calculation method of the present disclosure, by calculating the displacement using the correction parameters calculated using this correction parameter calculation method, it is possible to easily and accurately convert the displacement into an actual size value when measuring displacement using images.

[0044] Furthermore, for example, if there are two or more measurement points in the object at which the displacement is to be measured, in the distance calculation step, two or more first distance data from each of the two or more measurement points to the first imaging device are calculated, and in the parameter calculation step, based on the two or more first distance data, a correction value corresponding to each of the two or more measurement points is calculated as the correction parameter.

[0045] Thus, according to the correction parameter calculation method, the correction value is calculated for each of the plurality of measurement points, and therefore the actual dimensional displacement amount of each of the plurality of measurement points can be calculated with high accuracy.

[0046] Furthermore, for example, in the distance calculation step, a plurality of first distance data from each of a plurality of surface points of the object, including the two or more measurement points, to the first imaging device is calculated. In the parameter calculation step, based on the plurality of first distance data, a correction value corresponding to each of the plurality of surface points is calculated, and a correction value map is generated as the correction parameter based on the calculated plurality of correction values.

[0047] Thus, according to the correction parameter calculation method, when it is desired to measure the actual dimensional displacement at a position other than the measurement point, the actual dimensional displacement can be easily measured by using the correction value map.

[0048] Furthermore, for example, the method further includes a third acquisition step of acquiring displacement direction information indicating a displacement direction of the object, and in the parameter calculation step, the correction parameter is calculated using the displacement direction information.

[0049] Therefore, according to the correction parameter calculation method, when the object is displaced in a direction intersecting the imaging surface of the first imaging device when observed from above, the correction value can be calculated using the displacement direction information, thereby enabling the actual size value of the displacement to be measured with higher accuracy.

[0050] Furthermore, for example, the first imaging device includes a first camera, the second imaging device includes a second camera for capturing the second image data and a depth sensor for measuring the second distance data, and the first camera has a higher resolution or a higher frame rate than the second camera.

[0051] This method can determine the pixel displacement of an object based on black-and-white image data captured by a black-and-white camera. Generally speaking, black-and-white cameras can capture finer images than color cameras. Therefore, by measuring displacement based on the black-and-white image captured by a black-and-white camera, it is possible to measure this displacement with high precision. Furthermore, depth sensors are readily available, providing a highly versatile correction parameter calculation method.

[0052] The displacement calculation method involved in one embodiment of the present invention includes: a fourth acquisition step of acquiring the correction parameter calculated by using the above-mentioned correction parameter calculation method; a fifth acquisition step of acquiring the two or more third image data; and a conversion step of converting the pixel displacement in the two or more third image data into the actual size displacement based on the correction parameter.

[0053] Thus, the actual dimensional displacement amount can be measured using the correction parameters calculated by the correction parameter calculation method described above, and thus the actual dimensional displacement amount of the object can be measured easily and with high accuracy.

[0054] A correction parameter calculation device according to one embodiment of the present disclosure calculates a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation device comprising: a first acquisition unit for acquiring, from a first camera device, first image data obtained by photographing the object; a second acquisition unit for acquiring, from the second camera device, second distance data from a second camera device disposed at a position different from that of the first camera device to the object, and second image data obtained by photographing the object; a matching unit for matching the position of the object in the first image data with the second image data obtained by photographing the object; a position estimating unit that estimates the position of the first imaging device relative to the second imaging device based on the result of the association and the second distance data; a distance calculating unit that calculates first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; and a parameter calculating unit that calculates the correction parameter for converting a pixel displacement amount of a measurement point of the object into an actual size displacement amount using the first distance data, the pixel displacement amount of the measurement point of the object being based on two or more third image data captured by the first imaging device at different timings.

[0055] This achieves the same effects as the aforementioned correction parameter calculation method. Specifically, the correction parameter calculation device can calculate the first distance from the first imaging device to the object based on the estimated position of the first imaging device, eliminating the need to position the first and second imaging devices so that the optical axes of the first and second imaging devices are parallel. Furthermore, the correction parameter calculation device can calculate the correction parameters based on the first distance, making it possible to calculate the correction parameters used to convert pixel displacement to actual size displacement even when the length of a specific portion of the object is unknown.

[0056] Furthermore, the correction parameter calculation device estimates the first distance and can accurately determine the distance from the first imaging device to the object. Therefore, by using the first distance to calculate the correction parameter, the correction parameter calculation device can obtain correction parameters that can more accurately calculate the actual size displacement. For example, compared to using the second distance, correction parameters can be obtained for calculating the actual size displacement with higher accuracy. Thus, by using the correction parameter calculated by the correction parameter calculation device to calculate the displacement, the correction parameter calculation device can easily and accurately convert the displacement into an actual size value when measuring displacement using images.

[0057] A correction parameter calculation device according to one embodiment of the present disclosure calculates a correction parameter for measuring an actual size displacement amount using an image, the actual size displacement amount being an actual size value representing a displacement of a motion of an object, the correction parameter calculation device comprising: a first acquisition unit that acquires, from a first imaging device, first image data obtained by imaging the object; a second acquisition unit that acquires, from the second imaging device, second distance data from a second imaging device disposed at a different position from the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; a position estimating unit that detects the shape of the first imaging device in the second image data and estimates the position of the first imaging device based on the detection result; a distance calculating unit that calculates first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; and a parameter calculating unit that calculates the correction parameter for converting a pixel displacement amount of a measurement point of the object into an actual size displacement amount using the first distance data, the pixel displacement amount of the measurement point of the object being based on two or more third image data captured by the first imaging device at different timings.

[0058] This achieves the same effects as the above-described correction parameter calculation method. Specifically, the correction parameter calculation device can estimate the position information of the first imaging device using only the second image data and the distance data, making it possible to easily estimate the position information of the first imaging device.

[0059] Further, according to the correction parameter calculation device, by estimating the first distance, the distance from the first imaging device to the object can be accurately obtained. Therefore, according to the correction parameter calculation device, by calculating the correction parameter using the first distance, the correction parameter that can calculate a more accurate actual size displacement amount can be obtained. For example, compared to the case where the second distance is used, the correction parameter for calculating an actual size displacement amount with higher accuracy can be obtained. Thus, according to the above-described correction parameter calculation device, by calculating the displacement amount using the correction parameter calculated by the correction parameter calculation device, the displacement amount can be converted into an actual size value with high accuracy and easily when the displacement is measured using images.

[0060] The displacement amount calculation device according to one embodiment of the present disclosure includes: a third acquisition unit that acquires the correction parameter calculated using the above-described correction parameter calculation device; a fourth acquisition unit that acquires the two or more third image data; and a conversion unit that converts the pixel displacement amount in the two or more third image data into the actual size displacement amount based on the correction parameter.

[0061] Thus, the same effects as the above-described displacement amount calculation method can be obtained.

[0062] Further, these general or specific aspects can also be implemented by a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a CD-ROM readable by a computer, and can also be implemented by any combination of the system, the method, the integrated circuit, the computer program, or the recording medium. The program can be stored in the recording medium in advance or supplied to the recording medium via a wide area communication network including the Internet.

[0063] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0064] Further, the embodiments and modified examples described below each represent general or specific examples. The numerical values, shapes, materials, component configurations, arrangement positions and connection modes of components, steps, orders of steps, and the like shown in the following embodiments and modified examples are merely examples, and the present disclosure is not intended to be limited thereto. Further, regarding components among the components in the following embodiments for which independent claims are not recited, the components are described as arbitrary components.

[0065] Further, each drawing is a schematic view, and is not necessarily strictly illustrated. In addition, in each drawing, the same reference signs are added to substantially the same components, and repeated description is sometimes omitted or simplified.

[0066] In addition, in the present specification, the relational terms not strictly used in the sense of technical or relative numerical values, and the numerical values are not only expressed as a strict meaning, but also mean a range substantially equivalent, for example, a difference of several %.

[0067] In addition, in the following description, the image is a still image, but can also be a dynamic image.

[0068] (Embodiment 1)

[0069] Hereinafter, with reference to Figures 1 to 5 The correction parameter calculation method and the like related to Embodiment 1 will be described.

[0070] [1-1. Configuration of displacement measurement system]

[0071] First, with reference to Figure 1 and Figure 2 The displacement measurement system 1 provided with the correction parameter calculation unit 110 for executing the correction parameter calculation method related to Embodiment 1 will be described. Figure 1 is a diagram showing the schematic configuration of the displacement measurement system 1 related to Embodiment 1. Figure 2 is a block diagram showing the functional configuration of the displacement measurement system 1 related to Embodiment 1.

[0072] As shown in Figure 1 , the displacement measurement system 1 related to Embodiment 1 is an information processing system that measures the actual size value of the displacement of the object 60 using two imaging devices arranged at mutually different positions. The first imaging device 10 and the second imaging device 20 image the object 60 from mutually different viewpoints. The object 60 is an object to be measured for displacement, for example, a device that displaces (vibrates, etc.) when a motor or the like moves, but is not limited thereto, and can be, for example, a structure such as an infrastructure structure that displaces (deforms, etc.) due to stress from the outside. The infrastructure structure is, for example, a bridge on which a vehicle such as an automobile or a railway vehicle travels. Furthermore, the displacement in the present specification includes not only the vibration of the object 60, but also the deformation and the change in position of the object 60. The displacement can be considered to represent the movement of the object.

[0073] As shown in Figure 1 and Figure 2 , the displacement measurement system 1 is provided with the first imaging device 10, the second imaging device 20, the displacement measurement device 30, the output device 40, and the input device 50.

[0074] The first imaging device 10 images an image for measuring the displacement of the object 60. The image imaged by the first imaging device 10 is used to detect the amount of displacement (the number of pixels) on the image corresponding to the displacement of the object 60. Further, the image imaged by the first imaging device 10 will be also referred to as the first image, and the amount of displacement on the image corresponding to the displacement of the object 60 will be also referred to as the pixel displacement amount. The pixel displacement amount is an example of a physical amount (for example, the number of pixels) on a pixel. In addition, the first image can be used to calculate a correction value described later.

[0075] The first imaging device 10 is, for example, a monochrome camera. In other words, the first image is, for example, a monochrome image. Further, the first imaging device 10 is not limited to a monochrome camera, and can be a color camera. The first imaging device 10 is, for example, a digital camera or a digital video camera having an image sensor. Further, the first imaging device 10 is an example of an imaging device. The monochrome camera or the color camera possessed by the first imaging device 10 is an example of a first camera. The first camera has a higher resolution or a higher frame rate than a second camera described later. The first camera is, for example, a camera capable of imaging an image having a higher resolution than the second camera, or a camera capable of imaging an image at a higher frame rate than the second camera.

[0076] The second imaging device 20 images an image for measuring an actual size value of the displacement of the object 60. The second imaging device 20 has a configuration capable of acquiring an image obtained by imaging the object 60 and a distance from the second imaging device 20 to the object 60. The image imaged by the second imaging device 20 and the distance measured by the second imaging device 20 are used to estimate the position of the first imaging device 10. Further, the image imaged by the second imaging device 20 will be also referred to as the second image, and the actual size value of the displacement of the object 60 will be also referred to as the actual size displacement amount. The actual size displacement amount is an example of a physical amount (for example, a distance corresponding to the number of pixels) in an actual space. In addition, the distance from the second imaging device 20 to the object 60 is an example of a second distance.

[0077] The second camera device 20 has an imaging unit for capturing a second image and a distance measuring unit for measuring the distance. The imaging unit is, for example, a color camera, but may also be a black and white camera. The imaging unit only needs to be able to obtain a grayscale image of the object 60. The distance measuring unit, for example, has a depth sensor. The depth sensor can measure the distance to one or more points of the object 60, for example, based on the time it takes for a laser to reach the object 60 and return, to obtain the distance (for example, the positional relationship) between the second camera device 20 and the object 60. The depth sensor may also be, for example, a laser radar. In addition, the distance measuring unit is not limited to having a depth sensor, and may also have, for example, a TOF sensor. The second camera device 20 may, for example, be an imaging unit and a distance measuring unit integrally formed, or the imaging unit and the distance measuring unit may be detachably formed. In addition, the second camera device 20 is an example of a measuring device. The black and white camera or color camera included in the second camera device 20 is an example of a second camera.

[0078] When the displacement of the object 60 is less than or equal to a predetermined value, the first imaging device 10 and the second imaging device 20 may, for example, capture images of the same object 60 at different timings. Furthermore, when the displacement of the object 60 is greater than the predetermined value, the first imaging device 10 and the second imaging device 20 may, for example, synchronize and capture images of the object 60. The predetermined value is preferably set based on, for example, an allowable value for displacement error.

[0079] Furthermore, the second imaging device 20 only needs to perform imaging and measurement at least once after being installed. The imaging unit of the second imaging device 20 and the distance measurement unit can perform imaging and measurement at different timings when the displacement of the object 60 is below a predetermined value. However, when the displacement of the object 60 is greater than the predetermined value, they are preferably performed simultaneously.

[0080] The first imaging device 10 and the second imaging device 20 may each be fixed so as not to change position during imaging. Furthermore, the second imaging device 20 is preferably positioned near the first imaging device 10. The first imaging device 10 and the second imaging device 20 are each preferably positioned so as to capture images including each measurement point of the object 60.

[0081] The first image may have a higher resolution than the second image or may be captured at a higher frame rate than the second image. In the first embodiment, the first image is a black and white image and the second image is a color image, but the present invention is not limited thereto.

[0082] The displacement measuring device 30 is an information processing device that calculates a correction value for converting pixel displacement into actual dimensional displacement based on image data and distance data acquired from the first imaging device 10 and the second imaging device 20, and uses the calculated correction value to output the actual dimensional displacement of the object 60. The displacement measuring device 30 may also be, for example, a server device. The displacement measuring device 30 includes a correction parameter calculation unit 110, a displacement detection unit 120, and a displacement calculation unit 130.

[0083] The correction parameter calculation unit 110 calculates a correction value for converting pixel displacement into actual size displacement based on the image data and distance data acquired from the first imaging device 10 and the second imaging device 20. The correction parameter calculation unit 110 includes a first acquisition unit 111, a second acquisition unit 112, a matching unit 113, a position estimation unit 114, a distance calculation unit 115, and a correction value calculation unit 116.

[0084] The first acquisition unit 111 acquires first image data representing a first image from the first imaging device 10 .

[0085] The second acquisition unit 112 acquires the distance image data from the second imaging device 20. The second acquisition unit 112 acquires, for example, the second image data representing the second image and the distance data.

[0086] The first acquisition unit 111 and the second acquisition unit 112 are communication interfaces (communication circuits) connected to the first imaging device 10 and the second imaging device 20 respectively via wireless communication or wired communication so as to be communicable therewith.

[0087] The matching unit 113 matches the position of the object 60 in the first image data with the position of the object 60 in the second image data by matching the feature points of the first and second image data. For example, the matching unit 113 matches the positions (e.g., pixel positions) of the object 60 in the first and second image data. This allows the matching unit 113 to obtain correspondence information indicating the correspondence between the position of the object 60 in the first image data and the position of the object 60 in the second image data. Any existing technology, such as SIFT (Scale Invariant Feature Transform), can be used for feature point detection.

[0088] The position estimation unit 114 estimates the position of the first imaging device 10 based on the correspondence information, distance data, and internal parameters of the first imaging device 10. Specifically, the position estimation unit 114 estimates the position of the first imaging device 10 in the coordinate system of the second imaging device 20. The internal parameters of the first imaging device 10 are assumed to be known. Examples of these internal parameters include at least one of a 3-row x 3-column parameter matrix (cameraMatrix), lens distortion coefficients (distCoeffs), and focal length.

[0089] The distance calculation unit 115 calculates distance information indicating the distance from the first imaging device 10 to the object 60 based on the position information indicating the position of the first imaging device 10 and the distance data. For example, the distance calculation unit 115 calculates the distance from the first imaging device 10 to each of one or more measurement points on the object 60. Alternatively, the distance calculation unit 115 may calculate the distances to a plurality of surface points on the object 60, including the measurement point at which the displacement is to be measured. The distance calculation unit 115 may then generate a distance map that lists the distances from the first imaging device 10 to the object 60. The number of measurement points on the object 60 is not particularly limited and may be two or more.

[0090] The correction value calculation unit 116 calculates a correction value for converting pixel displacement into actual size displacement based on the position of the first imaging device 10, the distance from the first imaging device 10 to the object 60, and internal parameters of the first imaging device 10. The calculation of correction values ​​by the correction value calculation unit 116 will be described later. Alternatively, the correction value calculation unit 116 may calculate a correction value for each of a plurality of surface points and generate a correction value map in which the calculated correction values ​​are arranged. The correction value and the correction value map are examples of correction parameters. Furthermore, the correction value calculation unit 116 is an example of a parameter calculation unit.

[0091] Furthermore, although the example in which the correction parameter calculation unit 110 is built into the displacement measuring device 30 has been described, the present invention is not limited thereto. The correction parameter calculation unit 110 may also be implemented as a separate device. In this case, the correction parameter calculation unit 110 functions as a correction parameter calculation device.

[0092] Furthermore, the displacement measuring device 30 is an example of a displacement amount calculating device.

[0093] The displacement detection unit 120 detects the pixel displacement amount (number of pixels) on the image corresponding to the displacement of the object 60 based on two or more first image data captured by the first imaging device 10. The displacement detection unit 120 detects the pixel displacement amount for each measurement point, for example.

[0094] The displacement amount calculator 130 converts the pixel displacement amount into an actual size displacement amount based on the pixel displacement amount and the correction information indicating the correction parameters, thereby calculating the actual size displacement amount of the object 60. For example, the displacement amount calculator 130 calculates the actual size displacement amount of each of the plurality of measurement points of the object 60 based on the correction value of each of the plurality of measurement points and the pixel displacement amount of each of the plurality of measurement points.

[0095] The output device 40 obtains presentation information including the actual dimensional displacement amount from the displacement measuring device 30 and outputs the obtained presentation information. The output device 40 is, for example, a display device that displays the presentation information as an image. For example, the output device 40 is a liquid crystal display. The image output by the output device 40 is visually recognized by the operator.

[0096] The output device 40 can be a fixed device or a portable terminal carried by the operator. As long as the portable terminal has the output device 40 and can communicate with the displacement measuring device 30, there are no particular limitations; for example, it can be a smartphone or tablet. If the portable terminal has the output device 40, the operator can determine the actual dimensional displacement by checking the output device 40 on the portable terminal near the object 60. The operator is an example of a user.

[0097] The displacement measuring system 1 may include not only a display device but also a device that outputs sound as the output device 40, or may include a device that outputs sound instead of the display device as the output device 40. Furthermore, the displacement measuring system 1 may include a device that displays presentation information on an object (e.g., a screen), such as a projector, as the output device 40. Furthermore, when the displacement measuring device 30 is located remotely, the displacement measuring device 30 and the output device 40 may be connected via a network.

[0098] The input device 50 is a user interface for receiving predetermined information from the operator when measuring displacement. For example, the input device 50 can receive information from the operator for calculating a correction value, information regarding the displacement direction of the object 60, or information regarding the approximate position of the first imaging device 10. Furthermore, the input device 50 can also receive input of predetermined information via voice. Furthermore, the approximate position is an estimate of the installation position of the first imaging device 10.

[0099] The input device 50 is realized by hardware keys (hardware buttons), a slide switch, a touch panel, etc. The input device 50 may be a fixed device or a device included in a portable terminal carried by an operator.

[0100] [1-2. Operation of the displacement measurement system]

[0101] Next, refer to Figures 3 to 5 The operation of the displacement measuring system 1 will be described. Figure 3 : is a flowchart showing the operation of the displacement measuring system 1 according to the first embodiment. Specifically, Figure 3 : is a flowchart showing the operation of the displacement measuring device 30. Figure 3 Steps S11 to S16 shown in the figure represent the operation of the correction parameter calculation unit 110. Figure 4 is a diagram for explaining the feature point matching in the matching unit 113 according to the first embodiment. Figure 4 In the example shown, object 60a is a piece of furniture. Furthermore, to make the description of the characteristic points easier to understand, QR (Quick Response) codes are attached to some of the characteristic points. Each QR code can also be set corresponding to a measurement point on object 60a.

[0102] like Figure 3 As shown in FIG. 1 , the first acquisition unit 111 acquires first image data representing a first image from the first imaging device 10 ( S11 ). The first acquisition unit 111 acquires, for example, Figure 4 The first image data I1 shown in (a) is shown in FIG. Figure 4 In (a), the position (x, y) indicated by "0" represents, for example, the pixel position of the feature point on the first image data I1. Figure 4 In (a), the positions of five characteristic points are shown. In addition, a QR code is reflected in the first image data I1.

[0103] In order to calculate the correction value, the first acquisition unit 111 only needs to acquire at least one piece of first image data in step S11. In order to measure the displacement, the first acquisition unit 111 only needs to acquire two or more pieces of first image data captured at different timings in step S11.

[0104] When the first acquisition unit 111 acquires a plurality of first image data, it outputs at least one piece of the first image data to the matching unit 113 and outputs two or more pieces of the first image data to the displacement detection unit 120. The first acquisition unit 111 may output different first image data to the matching unit 113 and the displacement detection unit 120, or may output first image data that is at least partially identical to the matching unit 113 and the displacement detection unit 120. The two or more first image data output to the displacement detection unit 120 are image data captured at different timings and are an example of third image data. Furthermore, step S11 is an example of the first acquisition step. Alternatively, step S11 may be an example of the fifth acquisition step in the displacement amount calculation method described later. Furthermore, the first acquisition unit 111 may function as a fourth acquisition unit that acquires two or more pieces of third image data.

[0105] Next, the second acquisition unit 112 acquires the second image data representing the second image and the distance data representing the distance from the second imaging device 20 (S12). The second acquisition unit 112 acquires, for example, Figure 4 The distance image data I2 shown in (b) of FIG. 1 shows the second image data and the distance data. The distance image data I2 includes: a position of the object 60 relative to the second imaging device 20 (for example, Figure 4 The coordinates (X, Y, Z) of the second image sensor 20 (indicated by "0" in (b)) and the pixel value (R, G, B) representing the color at that position. The coordinates (X, Y, Z) are relative to the position of the second image sensor 20 (e.g., the coordinates (0, 0, 0)). The distance data is an example of the second distance data.

[0106] Furthermore, the distance image data I2 does not need to include color information. The distance image data I2 only needs to be an image that can be feature-point matched with the first image data I1 and includes the coordinates (X, Y, Z) representing the position of the object 60 relative to the second imaging device 20. For example, the distance image data I2 may be an image with a known grayscale of the object 60 and include the coordinates (X, Y, Z) representing the position of the object 60 relative to the second imaging device 20. The distance image data I2 may also include a black-and-white image or a grayscale image obtained by imaging the object 60.

[0107] The second acquisition unit 112 outputs the distance image data I2 to the matching unit 113 and the position estimation unit 114. The second image data is data including pixel values ​​(R, G, B) representing colors, and the distance data is data including coordinates (X, Y, Z) representing the position of the object 60 relative to the second imaging device 20. Step S12 is an example of the second acquisition step.

[0108] Next, the matching unit 113 associates the first image data I1 with the position of the object 60 in the second image data of the range image data I2 (S13). Figure 4 The first image data I1 and the distance image data I2 shown in FIG are used to illustrate that, for example, 5 feature point matching candidates are extracted (for example, at Figure 4 ("0" in (a) and (b) of FIG), the first image data I1 is associated with the position of the range image data I2. The matching unit 113 associates the pixel position (x, y) in the first image data I1 with the coordinates (X, Y, Z) in the range image data I2. The matching unit 113 associates the pixel position (x, y) in the first image data I1 with the coordinates (X, Y, Z) in the range image data I2. For example, the matching unit 113 determines where the portion of the object 60 at the pixel position (x, y) in the first image data I1 is located on the range image data I2, thereby making the association.

[0109] Then, the matching section 113 establishes correspondence between the pixel position (x, y) in the first image data I1 and the coordinates (X, Y, Z) in the distance data corresponding to the pixel position (x, y) based on the correspondence between the pixel position (x, y) in the first image data I1 and the pixel position (x1, y1) in the second image data, for example. The matching section 113 relates the distance data corresponding to the pixel position to the pixel position (x, y) on the first image data I1.

[0110] The matching section 113 determines groups of 5 feature points, for example, but the groups of feature points are not limited to 5 groups and are appropriately determined depending on the method of acquiring the correspondence, and the like. The matching section 113 can determine groups of 4 feature points, for example, and can determine groups of 6 feature points. Further, the method of calculating the correspondence and the number of feature points are examples and are not limited to the above. The number of groups of feature points will be referred to as n groups hereinafter.

[0111] The matching section 113 outputs correspondence information indicating the correspondence between the first image data I1 and the position of the second image data to the position estimation section 114. The matching section 113 outputs the correspondence information indicating the 5 groups of feature points determined above, for example. Further, the step S13 is an example of a correspondence establishing step.

[0112] Next, the position estimation section 114 estimates the position of the first imaging device 10 with respect to the second imaging device 20 based on the correspondence information, the distance data, and the internal parameters of the first imaging device 10 (S14). If the correspondence between the n groups of specific points of the first image data I1 and the distance image data I2 is known, the position estimation section 114 can estimate the position of the first imaging device 10 in the coordinate system of the second imaging device 20 by solving the PnP problem (Perspective-n-Point Problem). Further, the internal parameters of the first imaging device 10 are assumed to be known.

[0113] The position estimation section 114 calculates R (rotation) and T (position) using each group of feature points having a reprojection error of a reference value or less, for example, and thereby estimates the position and the attitude of the first imaging device 10 with respect to the second imaging device 20. The position estimation section 114 calculates R (rotation matrix) and T (translation vector) that minimize the reprojection error, for example. R (rotation matrix) is an external parameter indicating the attitude of the first imaging device 10. T (translation vector) is an external parameter indicating the position of the first imaging device 10. Further, the position estimation section 114 can estimate at least the position.

[0114] The matching unit 113 and the position estimation unit 114 may also perform processing to estimate the position of the first camera 10 using, for example, robust estimation, which reduces the influence of outliers to eliminate erroneous feature point correspondence. The matching unit 113 and the position estimation unit 114 may also estimate the position of the first camera 10 using, for example, RANSAC (Random Sampling Consensus) estimation, median estimation, or M-estimation, which are examples of robust estimation. RANSAC estimation may also be an estimation method based on reprojection error. The reprojection error is represented by the variance of the second position (xi, yi) on the second image data, obtained by projecting the first position (xi, yi, zi) on the distance data using a function that converts two coordinate systems, and the variance of the third position (xi0, yi0) on the first image data corresponding to the second position. An example of a function that converts two coordinate systems is the projectPoints function.

[0115] The matching unit 113 randomly selects, for example, n sets of feature points and calculates the position of the first imaging device 10. The matching unit 113 then calculates the number of sets of specific points within the n sets whose reprojection errors are below a reference value. The matching unit 113 repeatedly performs the above process on all feature points in the image, determining the n sets of feature points with the largest number of sets of specific points whose reprojection errors are below the reference value. The position estimation unit 114 can then use these n sets of feature points to estimate the position of the first imaging device 10, for example. Alternatively, the n sets can be six. In other words, the position estimation unit 114 can estimate the position of the first imaging device 10 by solving the P6P problem.

[0116] In this manner, the matching unit 113 and the position estimation unit 114 obtain a correspondence between the feature points of the object 60 in the first image data I1 and the second image data, and use this correspondence to estimate the position of the first imaging device 10. In other words, the displacement measuring device 30 according to the first embodiment can estimate the position of the first imaging device 10 even if the first imaging device 10 and the second imaging device 20 are installed at different positions and in different postures.

[0117] The position estimation unit 114 outputs position information indicating the estimated position of the first imaging device 10 to the distance calculation unit 115. Note that step S14 is an example of an estimation step.

[0118] The distance calculation unit 115 calculates the distance from the first imaging device 10 to the object 60 based on the position information and the distance data (S15). The distance calculation unit 115 calculates the distance from the first imaging device 10 to the object 60 based on the position (coordinates) of the first imaging device 10 included in the position information and the coordinates (X, Y, Z) of the object 60 based on the distance data. Alternatively, the distance calculation unit 115 converts the distance to the object 60 measured by the distance measurement unit of the second imaging device 20 into the distance to the object 60 as viewed from the first imaging device 10. This obtains the position of the object 60 as viewed from the first imaging device 10. For example, the distance to each measurement point of the object 60 as viewed from the first imaging device 10 is determined. Data including the distance calculated by the distance calculation unit 115 (the distance from the first imaging device 10 to the object 60) is an example of first distance data. The distance calculated by the distance calculation unit 115 is an example of the first distance. The distance calculation unit 115 outputs distance information indicating the distance between the first imaging device 10 and the object 60 to the correction value calculation unit 116. Step S15 is an example of a distance calculation step.

[0119] The correction value calculation unit 116 calculates a correction value for converting the pixel displacement amount into the actual size displacement amount based on the position and distance information of the first imaging device 10 and the internal parameters of the first imaging device 10 (S16). The correction value calculation unit 116 calculates the correction value for each of the multiple measurement points of the object 60, for example. The correction value includes, for example, information indicating the actual size value corresponding to one pixel at the measurement point. Step S16 is an example of a parameter calculation step. Alternatively, step S16 may be an example of a fourth acquisition step.

[0120] Here, refer to Figure 5 The processing of the correction value calculation unit 116 will be described. Figure 5 This is a diagram for explaining the actual size conversion method of the displacement involved in the first embodiment. Figure 5 , an example in which the object 60 is displaced in a direction parallel to the imaging plane (projection plane) of the first imaging device 10 is shown. Figure 5 The optical center O shown represents the center of the lens 11 of the first imaging device 10. Position P1 (x, y) represents the position on the imaging plane corresponding to the measurement point position M1 (X, Y, Z) of the object 60 at a first moment. The displacement (Δx1, Δy1) on the image represents the difference between position P2 on the imaging plane corresponding to the measurement point position M2 at a second moment different from the first moment and position P1. The displacement (Δx1, Δy1) represents the number of pixels on the image.

[0121] like Figure 5As shown, the triangle ΔOP1P2 and the triangle ΔOM1M2 are in a similar relationship. The correction value calculation unit 116 uses this similar relationship to calculate the correction value for converting the pixel displacement amount into the actual size displacement amount. If the distance between the image center (Cx, Cy) on the imaging plane of the first imaging device 10 and the optical center O is defined as the focal length f, the distance L1 from the optical center O to the position P1 is calculated as follows:

[0122]

[0123] The first time is, for example, an initial time when the object 60 has not yet moved.

[0124] In addition, if the actual size displacement is set to displacement (Dx1, Dy1), since the triangle △OP1P2 and the triangle △OM1M2 are in a similar relationship, the following equation holds true for the distance L2 between the optical center O and the position M1:

[0125] L2:L1=Dy1:Δy1=Dx1:Δx1 (Equation 2)

[0126] As can be seen from (Equation 2), if the distance L2, i.e., the distance from the optical center O to the position M1, is known, the displacement (Dx1, Dy1) as the actual dimensional displacement can be calculated. Because the distance L2 may vary at each measurement point, the distance calculation unit 115 calculates the distance L2 at each measurement point.

[0127] The correction value calculation unit 116 calculates the correction value based on the above-mentioned (Equation 2). The correction value used to calculate the displacement Dx1 is the correction value based on (Equation 2) and is L2 / L1. The correction value used to calculate the displacement Dy1 is also the correction value based on (Equation 2) and is L2 / L1.

[0128] In addition, the image center (Cx, Cy) and the focal length f are acquired as internal parameters of the first imaging device 10. The correction value calculation unit 116 may calculate the internal parameters of the first imaging device 10 using a graph image.

[0129] The correction value calculator 116 outputs correction information indicating the calculated correction value to the displacement calculator 130. The displacement calculator 130 acquires the correction information from the correction value calculator 116. The displacement calculator 130 functions as a third acquisition unit that acquires correction parameters.

[0130] Next, the displacement detection unit 120 calculates the pixel displacement amount of the object 60 on the first image data I1 based on the first image data I1 (S17). The displacement detection unit 120 calculates the pixel displacement amount based on, for example, the displacement of the object 60 projected onto the imaging plane of the first imaging device 10. For example, the displacement detection unit 120 calculates the number of pixels on the image corresponding to the displacement of each of the plurality of measurement points on the object 60. The displacement detection unit 120 outputs the calculated pixel displacement amount to the displacement amount calculation unit 130.

[0131] The displacement calculation unit 130 calculates the actual size displacement based on the pixel displacement and the correction value (S18). For each of the multiple measurement points of the object 60, the displacement calculation unit 130 calculates the actual size displacement at that measurement point by calculating the pixel displacement and the correction value. Thus, the displacement calculation unit 130 functions as a conversion unit that converts the pixel displacement in two or more third image data sets into actual size displacement based on the correction information.

[0132] The displacement amount calculation unit 130 outputs the presentation information including the calculated actual size displacement amount to the output device 40 (S19). The output device 40 then displays the presentation information obtained from the displacement amount calculation unit 130 as an image. Step S19 is an example of a conversion step.

[0133] also, Figure 3 The actions shown in steps S11 to S16 can be performed each time the process for calculating the actual size displacement of the object 60 is executed, or each time at least one of the installation position, posture, and camera parameters (including internal parameters) of at least one of the first camera device 10 and the second camera device 20 is changed. In addition, the correction value calculation unit 116 can also store the calculated correction value in a storage unit (not shown). Then, when calculating the actual size displacement of the object 60, the correction value calculation unit 116 can read the correction value from the storage unit and output the read correction value to the displacement calculation unit 130. In other words, the correction value calculation unit 116 can also use the correction value calculated in the past as the correction value for calculating the current actual size displacement. In this way, the processing load of the correction parameter calculation unit 110 can be reduced.

[0134] Steps S11 to S16 are performed in the correction parameter calculation method for calculating the correction parameter. Furthermore, steps S17 to S19 are performed in the displacement calculation method for calculating the actual size displacement. Alternatively, the displacement calculation method may include steps S11 to S16.

[0135] As described above, the displacement measuring system 1 according to the first embodiment includes the first imaging device 10 and the second imaging device 20, which are arranged at different positions, and the displacement measuring device 30 having a correction parameter calculation unit 110. The correction parameter calculation unit 110 estimates the position of the first imaging device 10 based on the first image data I1 obtained from the first imaging device 10 and the distance image data I2 obtained from the second imaging device 20. Since the position of the first imaging device 10 is estimated by the correction parameter calculation unit 110, the displacement measuring device 30 can calculate the actual dimensional value of the displacement with high accuracy, even when the positions and postures of the first imaging device 10 and the second imaging device 20 differ in their initial state.

[0136] As described above, according to the displacement measurement system 1, there are fewer constraints on the position and posture of the first imaging device 10 and the second imaging device 20, eliminating the need for fine-tuning the position and posture of the first imaging device 10 and the second imaging device 20. This reduces the time required for setup at the measurement site. Furthermore, compared to measuring the actual dimensional displacement of the object 60 using a multi-viewpoint system, the number of imaging devices required is reduced, making setup and removal of the imaging devices easier at the measurement site.

[0137] Furthermore, as described above, the correction parameter calculation unit 110 calculates a correction value for converting pixel displacement into actual size displacement using, for example, the estimated position of the first imaging device 10. This allows the correction parameter calculation unit 110 to measure the actual size displacement of the object 60 even when the object 60 lacks scale markings or when the actual size of a specific portion of the object 60 is unknown.

[0138] Furthermore, as described above, the correction parameter calculation unit 110 calculates a correction value for each of the multiple measurement points on the object 60, for example. This allows the operator to select a measurement point without having to consider the difference in distance between the first imaging device 10 and each of the multiple measurement points. In other words, the correction parameter calculation unit 110 increases the degree of freedom in selecting measurement points.

[0139] Furthermore, the displacement measuring device 30 measures the actual dimensional displacement of the object 60 using data from two or more first images captured by the first imaging device 10. Thus, even when the displacement of the object 60 is caused by periodic vibrations that are difficult to measure using a laser displacement meter or the like, the displacement measuring device 30 can still measure the actual dimensional displacement.

[0140] In addition, in this embodiment, an example in which the physical quantity in the actual space is the actual size displacement is described, but it is not limited to this. The physical quantity in the actual space is not particularly limited as long as it is a physical quantity that can be obtained based on image data, and it can also be, for example, length, area, etc. In addition, the physical quantity in the actual space can also be the actual size displacement based on multiple frames, the speed of displacement, etc. In this case, in the parameter calculation step (equivalent to S16), the first distance data is used to calculate the correction parameter for converting the physical quantity of the measurement point of the object 60 on the pixel into the physical quantity in the actual space, and the physical quantity of the measurement point of the object 60 on the pixel is based on two or more third image data captured by the first camera device 10 at different timings. In addition, in this case, the displacement measuring device 30 described in this embodiment can also be said to be a physical quantity measuring device.

[0141] The correction parameter calculation method for calculating the correction parameter for calculating the physical quantity in such an actual space is as follows. For example, it can also be a correction parameter calculation method for calculating the correction parameter for measuring the actual size physical quantity using an image, and the actual size physical quantity is the actual size value of the physical quantity of the object 60 in the actual space. The correction parameter calculation method includes: a first acquisition step (equivalent to S11), acquiring the first image data I1 obtained by photographing the object 60 from the first camera device 10; a second acquisition step (equivalent to S12), acquiring the second distance data from the second camera device 20 configured at a position different from the first camera device 10 to the object 60, and the second image data obtained by photographing the object 60 from the second camera device 20; an establishment correspondence step (equivalent to S13), 1 and the position of the object 60 in the second image data are associated with each other; an estimating step (S14) of estimating the position of the first imaging device 10 relative to the second imaging device 20 based on the association result and the second distance data; a distance calculating step (S15) of calculating distance data from the first imaging device 10 to the object 60 based on the position of the first imaging device 10 and the second distance data; and a parameter calculating step (equivalent to S16) of calculating, using the first distance data, a correction value (an example of a correction parameter) for converting a physical quantity at a measurement point of the object 60 on a pixel based on the third image data captured by the first imaging device 10 into a physical quantity in real space.

[0142] Further, for example, the correction parameter calculation device 110 that calculates a correction parameter for measuring an actual size physical quantity of the object 60, which is an actual size value of a physical quantity of the object 60 in an actual space, using the image can also be implemented. For example, the correction parameter calculation device 110 can also include: a first acquisition unit 111 that acquires first image data obtained by imaging the object 60 from the first imaging device 10; a second acquisition unit 112 that acquires second distance data from the object 60 from the second imaging device 20 disposed at a position different from the first imaging device 10 and second image data obtained by imaging the object 60 from the second imaging device 20; a correspondence establishing unit 113 that establishes correspondence between the position of the object 60 in the first image data and the position of the object 60 in the second image data; a position estimation unit 114 that estimates the position of the first imaging device 10 with respect to the second imaging device 20 based on the correspondence establishment result and the second distance data; a distance calculation unit 115 that calculates first distance data from the object 60 from the first imaging device 10 based on the position of the first imaging device 10 and the second distance data; and a correction value calculation unit 116 that calculates a correction parameter for converting a physical quantity of a measurement point of the object 60 on a pixel based on the third image data imaged by the first imaging device 10 into a physical quantity in an actual space using the first distance data.

[0143] (Modified example of Embodiment 1)

[0144] Hereinafter, the correction parameter calculation method and the like related to the present modified example will be described with reference to Figures 6 to 8 The correction parameter calculation method and the like related to the present modified example will be described. In the following description, the same reference numerals are assigned to the same components as those of Embodiment 1, and the description will be omitted or simplified in some cases. The configuration of the displacement measurement system related to the present modified example is the same as that of the displacement measurement system 1 related to Embodiment 1, and the description will be omitted.

[0145] The displacement measurement system 1 related to the present modified example is characterized in that an actual size displacement amount can be calculated with high accuracy even when the displacement direction of the object 60 is different from the direction horizontal with respect to the imaging surface (projection surface) of the first imaging device 10. Figure 6 is a flowchart showing the operation of the displacement measurement system 1 related to the present modified example.

[0146] As Figure 6As shown, the correction value calculation unit 116 obtains the displacement direction information of the object 60 (S21). The correction value calculation unit 116 may also obtain the displacement direction information via the input device 50, for example. For example, when the object 60 is a motor, the displacement direction information may also be information indicating the displacement direction based on the driving direction of the driving unit of the motor. In addition, for example, when the displacement direction of the object 60 can be predicted in design, the displacement direction information may also be information indicating the predicted displacement direction. In addition, for example, when the object 60 is a bridge or the like and is subjected to stress from a vehicle or the like, the displacement direction information may also be information indicating the direction in which the stress is applied (for example, a vertical direction). In addition, the displacement direction information may also be information indicating the direction relative to the imaging surface of the first imaging device 10, for example. In addition, the displacement direction indicated by the displacement direction information is not limited to one, and may be two or more. Step S21 is an example of the third acquisition step.

[0147] Furthermore, the displacement direction information is not limited to being obtained via the input device 50. The displacement direction information may also be determined based on, for example, two or more distance image data I2 (e.g., second image data). In this case, the second acquisition unit 112 acquires two or more distance image data I2 captured at different timings from the second imaging device 20, and outputs the acquired at least two or more distance image data I2 to the correction value calculation unit 116 (not shown). The correction value calculation unit 116 may also determine the displacement direction based on the position (X, Y, Z) of a predetermined measurement point contained in each of the two or more distance image data I2, thereby acquiring the displacement direction information. By using two or more distance image data I2, when the displacement is a short-period vibration, etc., although it is difficult to accurately measure the actual size displacement, the displacement direction can be acquired.

[0148] Alternatively, the orientation of the surface of the displacement measurement point on the object 60 may be acquired from the distance image data I2, and the displacement direction may be defined based on the orientation of the surface (based on the surface normal direction or a direction parallel to the surface).

[0149] In addition, the displacement direction information can also be obtained based on the first image data I1. Figure 4 As shown in (a), a QR code is attached. Furthermore, information related to the displacement direction of the object 60 may be stored for the QR code. The correction value calculation unit 116 may also obtain the displacement direction of the object 60 based on the QR code included in the first image data I1. In this case, the displacement direction is, for example, a direction relative to the surface of the object 60 to which the QR code is attached.

[0150] Further, as long as the displacement direction information can be acquired on the basis of the first image data I1, it is not limited to the case where the QR code is pasted on the object 60. For example, the AR marker can be pasted on the object 60. In addition, the correction value calculation section 116 can acquire the displacement direction information as information corresponding to the AR marker on the basis of the AR marker included in the first image data I1, for example. The displacement direction information can be displayed superimposed on the first image data I1, for example.

[0151] Then, the correction value calculation section 116 calculates a correction value for converting the pixel displacement amount into the actual size displacement amount on the basis of the position of the first imaging device 10, the distance information, the internal parameter of the first imaging device 10, and the displacement direction information (S22). That is, the correction value calculation section 116 calculates the correction value using the displacement direction information in addition to the embodiment 1. Further, the step S22 is an example of the parameter calculation step.

[0152] Here, the displacement correction using the displacement direction information will be described with reference to Figure 7A and Figure 7B Figure 7A is a first diagram for explaining the displacement correction considering the displacement direction according to the present modified example. In addition, the object 60b is a square for convenience. Figure 7B is a second diagram for explaining the displacement correction considering the displacement direction according to the present modified example.

[0153] Figure 7A and Figure 7B are plan views of the object 60b viewed from above. In addition, the shape of the object 60b is a square for convenience.

[0154] As shown in Figure 7A and Figure 7B , in the case where the object 60b is viewed from above, when the displacement direction of the object 60b is not parallel with respect to the imaging surface of the first imaging device 10, only the displacement in the direction parallel with the imaging surface among the displacements of the object 60b is projected on the imaging surface. In Figure 7A , the displacement direction is the direction connecting the upper left and the lower right of the paper surface. In this case, for example, if the displacement of the measurement point of the object 60b is set to Al, the displacement of the measurement point of the object 60b projected on the imaging surface becomes A2. The displacement A2 is the component in the direction parallel with the imaging surface among the displacement Al.

[0155] In addition, in Figure 7B ​In this case, the displacement direction is a direction connecting the lower left and the upper right of the paper surface. In this case, for example, if the displacement of the measurement point of the object 60b is set to Bl, the displacement of the measurement point of the object 60b projected onto the imaging surface becomes B2. The displacement B2 is a component in a direction parallel to the imaging surface in the displacement Bl. For example, in a case where the direction of the displacement Al and the direction of the displacement Bl are mutually orthogonal directions and the displacement amounts are equal, the direction and the size of the displacements A2 and B2 on the imaging surface are equal. Further, the displacements Al and Bl are actual size displacement amounts, for example, actual size values. In addition, the displacements A2 and B2 are pixel displacement amounts, for example, pixel numbers.

[0156] As such, the displacements Al and Bl having different actual displacement directions are sometimes detected as displacements having the same direction on the imaging surface. This can measure the displacement in the direction horizontal to the imaging surface, but cannot measure the displacement in the direction vertical to the imaging surface.

[0157] Therefore, in the present modification example, the correction value calculation section 116 acquires the displacement direction information of the object 60 in step S21, and calculates the correction value for converting the pixel displacement amount of the object 60 on the first image data II into the actual size displacement amount using the displacement direction information in step S22, as also explained above. Here, the correction value includes an actual size value corresponding to one pixel and information indicating the displacement direction. The correction value can also include information indicating the actual size value (Xa, Yb, Zc) corresponding to one pixel, for example. The actual size value Xa indicates the actual size value of the displacement in the X-axis direction in the 3-dimensional orthogonal coordinate system in the actual space when deviating by one pixel on the imaging surface. The actual size value Xb indicates the actual size value of the displacement in the Y-axis direction in the 3-dimensional orthogonal coordinate system in the actual space when deviating by one pixel on the imaging surface. The actual size value Xc indicates the actual size value of the displacement in the Z-axis direction in the 3-dimensional orthogonal coordinate system in the actual space when deviating by one pixel on the imaging surface. The actual size values (Xa, Yb, Zc) are an example of the correction parameter. Further, the calculation method of the correction value is not limited to the above case.

[0158] Here, with reference to Figure 8 The processing of the correction value calculation section 116 will be described. Figure 8 is a view for explaining the actual size conversion method of the displacement involved in the present modification example.

[0159] As Figure 8As shown, if the position P1 on the imaging plane corresponding to the measurement point position M1 (X, Y, Z) of the object 60 at the first moment is represented by (x, y), the distance L1 from the optical center O to position P1 can be calculated using (Equation 1). The first moment is, for example, the initial moment when the object 60 has not moved. Furthermore, if the difference (displacement) between position P3 on the imaging plane corresponding to the measurement point position M3 at a second moment different from the first moment and position P1 is represented by (Δx2, Δy2), the first correction value can be calculated using (Equation 2). Here, the first correction value is a value that allows calculation of the actual dimensional displacement in a direction parallel to the imaging plane of the first imaging device 10. For example, the first correction value is the displacement (Dx2, Dy2) among the displacements (Dx2, Dy2, Dz2) that can be calculated as the actual dimensional value based on the displacement (Δx2, Δy2) on the image.

[0160] The correction value calculation unit 116 converts the calculated first correction value and the displacement direction information into a second correction value, which is three-dimensional information. The correction value calculation unit 116 calculates the second correction value for converting the pixel displacement amount, which is two-dimensional information representing the displacement on the imaging plane, into the actual size displacement amount, which is three-dimensional information. The correction value calculation unit 116 may also calculate the second correction value based on the first correction value and the angle between the imaging plane of the first imaging device 10 and the displacement direction. The second correction value is, for example, a value that allows calculation of the displacement (Dx2, Dy2, Dz2) of the actual size value from the displacement (Δx2, Δy2) on the image. The method by which the correction value calculation unit 116 calculates the second correction value is not limited to the above.

[0161] As described above, the correction parameter calculation unit 110 of the displacement measurement system 1 according to this modification further calculates a correction value for converting the pixel displacement amount into the actual size displacement amount using the displacement direction information indicating the displacement direction of the object 60 .

[0162] Thus, when the object 60 is displaced in a direction intersecting the imaging plane of the first imaging device 10 when viewed from above, the displacement measuring system 1 can calculate a correction value using displacement direction information, thereby accurately measuring the actual dimensional value of the displacement.

[0163] (Implementation Method 2)

[0164] [2-1. Configuration of the displacement measurement system]

[0165] Reference Figures 9 to 11 The configuration of the displacement measuring system 1 a according to the second embodiment will be described. Figure 9 This is a diagram showing a schematic configuration of a displacement measuring system 1 a according to the second embodiment. Figure 10is a block diagram showing functional configuration of the displacement measurement system la according to Embodiment 2. The displacement measurement device 30a according to Embodiment 2 differs from the displacement measurement device 30 according to Embodiment 1 mainly in that the 2nd acquisition unit 112 acquires the 2nd image data and the distance data including the 1st image data of the 1st image device 10, and the position estimation unit 114 estimates the position information of the 1st image device 10 using only the 2nd image data and the distance data. In the following description, the difference from Embodiment 1 will be mainly described, the same reference numerals are given to the same configuration as Embodiment 1, and the description is sometimes omitted or simplified.

[0166] As shown in Figure 9 , the displacement measurement system la according to Embodiment 2 is an information processing system that measures the actual size value of the displacement of the object 60 using two image devices arranged at different positions from each other. The 1st image device 10 and the 2nd image device 20 image the object 60 from different viewpoints from each other.

[0167] As shown in Figure 9 and Figure 10 , the displacement measurement system la includes the 1st image device 10, the 2nd image device 20, the displacement measurement device 30a, the output device 40, and the input device 50.

[0168] The 2nd image device 20 images an image including the 1st image device 10 and the object 60. The 2nd image device 20 can image an image including the 1st image device 10 to the extent that the shape of the 1st image device 10 can be detected, or can image in a manner that a mark attached to the housing of the 1st image device 10 is reflected in the case where the mark is attached to the housing of the 1st image device 10. The 2nd image device 20, for example, images an image including at least one of the plurality of surfaces constituting the 1st image device 10.

[0169] In addition, the 2nd image device 20 preferably images in a position or a posture in which the 1st image device 10 and the object 60 do not overlap. The 2nd image device 20, for example, can also be arranged behind the 1st image device 10.

[0170] The displacement measurement device 30a differs from the displacement measurement device 30 according to Embodiment 1 in that the matching unit 113 is not provided.

[0171] The 1st acquisition unit 111 acquires the 1st image data indicating the 1st image including the object 60 from the 1st image device 10. Then, the 1st acquisition unit 111 outputs the acquired 1st image data to the displacement detection unit 120.

[0172] The second acquisition unit 112 acquires the second image data indicating the second image including the first imaging device 10 and the object 60, and the distance data from the second imaging device 20.

[0173] The position estimation unit 114 detects the first imaging device 10 from the distance data or the second image data. At least one of the appearance, size, and shape of the first imaging device 10 is known in advance, and thus the position estimation unit 114 can detect the first imaging device 10 by 2-dimensional or 3-dimensional pattern matching. Alternatively, the position estimation unit 114 can detect the first imaging device 10 using the housing or lens of the first imaging device 10, or a marker attached to the housing of the first imaging device 10 as shown in Figure 11

[0174] Figure 11 is a diagram indicating an example of the marker 200 to which the first imaging device 10 according to Embodiment 2 is attached. Figure 11 (a) of Figure 11 (b) indicates a state in which the marker 200 is attached to the side surface of the first imaging device 10. At least one marker is attached to the first imaging device 10.

[0175] Further, the position estimation unit 114 determines the 3-dimensional position coordinates of the optical center based on the detected position of the first imaging device 10. The position of the optical center with respect to the housing or the like of the first imaging device 10 can be determined according to design values with respect to the combination of the second camera and the lens, or can be found in advance by optical calibration. The position estimation unit 114 outputs position information including the position coordinates of the first imaging device 10 when the found optical center is the reference (origin) to the distance calculation unit 115. Further, other configurations and operations of the displacement measurement system la are the same as those of Embodiment 1. In Embodiment 2, since the position estimation unit 114 estimates the position information of the first imaging device 10 using only the second image data and the distance data, the position information of the first imaging device 10 can be estimated without being affected by the position or number of feature points obtained from the image.

[0176] [2-2. Action of displacement measurement system]

[0177] Next, the action of the displacement measurement system la will be described with reference to Figure 12 Figure 12 is a flowchart indicating the action of the displacement measurement system la according to Embodiment 2. Specifically, Figure 12 is a flowchart indicating the action of the displacement measurement device 30a.

[0178] As shown in Figure 12 ​​As shown in FIG. 1 , the first acquisition unit 111 acquires first image data representing a first image from the first imaging device 10 ( S11 ). The first acquisition unit 111 acquires, for example, Figure 4 The first image data I1 shown in (a) of FIG. 1 is an example of a first acquisition step.

[0179] Next, the second acquisition unit 112 acquires second image data representing a second image and distance data representing a distance from the second imaging device 20 (S31). The second acquisition unit 112 acquires second image data representing both the object 60 and the first imaging device 10, and distance data representing a distance from the second imaging device 20. The second acquisition unit 112 outputs the acquired second image data and distance data to the position estimation unit 114. Step S31 is an example of the second acquisition step.

[0180] The position estimation unit 114 detects the first imaging device 10 based on the distance data or the second image data. For example, the position estimation unit 114 detects the first imaging device 10 in the second image data based on the distance data or the second image data (S32). The position estimation unit 114 detects the shape of the first imaging device 10 in the second image data using the method described above, for example.

[0181] The position estimation unit 114 then estimates the three-dimensional position coordinates of the optical center based on the detected position of the first imaging device 10. It can also be said that the position estimation unit 114 estimates the position of the first imaging device 10 relative to the second imaging device 20 based on the detected position of the first imaging device 10 (S33). In other words, the position estimation unit 114 estimates the position of the first imaging device 10 without using the correspondence information between the two images. The position estimation unit 114 outputs position information, including the position coordinates of the first imaging device 10 with the determined optical center as the reference (origin), to the distance calculation unit 115. This position information also includes the distance data acquired by the second acquisition unit 112.

[0182] The processing after step S15 is the same as that in the first embodiment, and therefore the description thereof is omitted.

[0183] As described above, according to the displacement measuring system 1a, the position estimation unit 114 estimates the position information of the first camera device 10 using the second image data including the first camera device 10 and the distance data. Therefore, the position information of the first camera device 10 can be estimated without being affected by the position or number of feature points in the two images.

[0184] Further, in the present embodiment, an example in which the physical quantity in the actual space is the actual dimensional displacement quantity is described, but the present application is not limited to this. The physical quantity in the actual space is not particularly limited as long as it is a physical quantity that can be acquired from the image data, and can be, for example, a length, an area, or the like. In addition, the physical quantity in the actual space can also be a speed of displacement or the like based on the actual dimensional displacement quantity of a plurality of frames. In this case, in the parameter calculation step (corresponding to S16), using the first distance data, a correction parameter for converting a pixel physical quantity of a measurement point of the object 60 based on two or more pieces of third image data imaged by the first imaging device 10 at mutually different timings into an actual dimensional physical quantity in the actual space is calculated. In addition, in this case, the displacement measurement device 30a described in the present embodiment can also be said to be a physical quantity measurement device.

[0185] In addition, for example, it can be a correction parameter calculation method of calculating a correction parameter for measuring a physical quantity of an object in an actual space using an image, including: a first acquisition step (corresponding to S11) of acquiring first image data I1 obtained by imaging an object 60 from a first imaging device 10; a second acquisition step (corresponding to S31) of acquiring second distance data from the object from a second imaging device 20 disposed at a position different from the first imaging device 10 and second image data obtained by imaging the object 60 and the first imaging device 10 from the second imaging device 20; a step of detecting the shape of the first imaging device 10 in the second image data (corresponding to S32); an estimation step (corresponding to S33) of estimating the position of the first imaging device 10 based on the detection result; a distance calculation step (corresponding to S15) of calculating first distance data from the first imaging device 10 to the object 60 based on the position of the first imaging device 10 and the second distance data; and a parameter calculation step (corresponding to S16) of calculating a correction value (an example of a correction parameter) for converting a pixel physical quantity of a measurement point of the object 60 based on third image data imaged by the first imaging device 10 into an actual dimensional physical quantity in the actual space using the first distance data.

[0186] Furthermore, for example, the present invention may be implemented as a correction parameter calculation device 110 that calculates correction parameters for measuring physical quantities of an object in real space using an image. For example, the correction parameter calculation device 110 may include: a first acquisition unit 111 that acquires first image data obtained by photographing the object 60 from the first camera device 10; a second acquisition unit 112 that acquires second distance data from the second camera device 20, which is arranged at a position different from the first camera device 10, to the object 60, and second image data obtained by photographing the object 60 and the first camera device 10 from the second camera device 20; a position estimation unit 114 that estimates the position of the first camera device 10 in the second image data; The shape of the object 60 is detected and the position of the first camera device 10 is estimated based on the detection result; the distance calculation unit 115 calculates first distance data from the first camera device 10 to the object 60 based on the position of the first camera device 10 and the second distance data; and the correction value calculation unit 116 uses the first distance data to calculate a correction parameter for converting the pixel physical quantity of the measurement point of the object 60 into the actual size physical quantity in the actual space, wherein the pixel physical quantity of the measurement point of the object 60 is based on the third image data captured by the first camera device 10.

[0187] (Other embodiments)

[0188] While the correction parameter calculation method and the like according to one or more aspects of the present disclosure have been described above based on Embodiment 1, Embodiment 2, and their variations (hereinafter also referred to as embodiments, etc.), the present disclosure is not limited to these embodiments, etc. As long as they do not depart from the spirit of the present disclosure, various variations that occur to those skilled in the art, as well as combinations of components from different embodiments, may also fall within the scope of one or more aspects of the present disclosure.

[0189] For example, in the above embodiments, the first and second imaging devices are described as being ground-mounted imaging devices, but the present invention is not limited thereto. At least one of the first and second imaging devices may be an imaging device mounted on an aerial vehicle such as a drone.

[0190] In addition, in the above embodiments, the first acquisition unit and the second acquisition unit are described as examples of communication units, but the present invention is not limited to this. At least one of the first acquisition unit and the second acquisition unit may be, for example, a connection unit for connecting a recording medium. The connection unit may be, for example, a USB terminal for connecting a USB (Universal Serial Bus), an SD card slot for inserting an SD card, an optical drive for inserting an optical disc, etc.

[0191] In addition, the order in which each step in the flowchart is executed is an example order for specifically explaining the present disclosure, and may be an order other than the above-mentioned order. In addition, part of the above-mentioned steps may be executed simultaneously (in parallel) with other steps.

[0192] The division of functional modules in the block diagram is merely an example. It is also possible to implement multiple functional modules as a single functional module, to divide a single functional module into multiple modules, or to transfer some functions to other functional modules. Furthermore, it is also possible to process the functions of multiple functional modules with similar functions in parallel or in a time-sharing manner by a single hardware or software.

[0193] Furthermore, the correction parameter calculation device in the above-described embodiments may be implemented by a single device or by a plurality of devices connected to each other. If the correction parameter calculation device is implemented by a plurality of devices, the components of the correction parameter calculation device may be distributed among the plurality of devices in any manner.

[0194] Furthermore, in the above embodiments, examples have been described in which the displacement measuring device does not include an output device, that is, the displacement measuring device and the output device are separate components. However, the displacement measuring device may also include an output device. In this case, the output device functions as an output unit (e.g., a display unit) that is part of the displacement measuring device.

[0195] Furthermore, the communication method between devices in the displacement measurement system in the above-described embodiments is not particularly limited. Both wireless and wired communication can be performed between devices. Furthermore, a combination of wireless and wired communication can also be used between devices.

[0196] Furthermore, part or all of the components of the calculation device in the above-described embodiments and the like may be constituted by a single system LSI (Large Scale Integration).

[0197] A system LSI is a highly multifunctional LSI that integrates multiple processing units on a single chip. Specifically, it is a computer system consisting of a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), and other components. The ROM stores computer programs. The system LSI achieves its functions by operating the microprocessor according to the computer program.

[0198] Although referred to here as a system LSI, it is sometimes referred to as an IC, LSI, super LSI, or ultra LSI, depending on the degree of integration. Furthermore, integrated circuitry is not limited to LSIs; dedicated circuits or general-purpose processors can also be used. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI fabrication, or reconfigurable processors, which allow the connections and settings of circuit cells within the LSI to be reconfigured, can also be used.

[0199] Furthermore, if semiconductor technology advances or other derivative technologies lead to the emergence of integrated circuit technology that replaces LSIs, this technology can naturally be used to integrate functional modules. There is also the possibility of applying biotechnology, etc.

[0200] Furthermore, all or part of the various processes described above may be implemented by hardware such as electronic circuits or by software. Furthermore, software-based processes are implemented by a processor included in the displacement measuring device executing a program stored in a memory.

[0201] Another aspect of the present disclosure may also be a computer program for causing a computer to execute the characteristic steps included in the correction parameter calculation method. Another aspect of the present disclosure may also be a computer-readable non-volatile recording medium containing such a program. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, by installing the distributed program on another device having a processor and causing the processor to execute the program, the device can be caused to perform the aforementioned processes.

[0202] Industrial Applicability

[0203] The present disclosure can be widely utilized in devices and the like that measure the actual size value of the displacement when measuring the displacement of an object using an image.

[0204] Description of reference numerals:

[0205] 1.1a Displacement measurement system

[0206] 10. First Camera Device

[0207] 11 Lens

[0208] 20 Second camera device

[0209] 30, 30a Displacement measuring device

[0210] 40 Output device

[0211] 50 Input device

[0212] 60, 60a, 60b Objects

[0213] 110 correction parameter calculation section (correction parameter calculation means)

[0214] 111 first acquisition section

[0215] 112 second acquisition section

[0216] 113 matching section

[0217] 114 position estimation section

[0218] 115 distance calculation section

[0219] 116 correction value calculation section (parameter calculation section)

[0220] 120 displacement detection section

[0221] 130 displacement amount calculation section

[0222] 200 mark

[0223] A1, A2, B1, B2 displacement

[0224] f focal length

[0225] I1 first image data

[0226] I2 distance image data

[0227] L1, L2 distance

[0228] M1-M3, P1-P3, T position

[0229] O optical center

Claims

1. A method for calculating a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the method comprising: a first acquisition step of acquiring, from a first imaging device, first image data obtained by imaging the object; a second acquisition step of acquiring, from a second imaging device, second distance data from the second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object; a third acquisition step of acquiring displacement direction information indicating a direction of displacement of the object in three dimensions; a step of establishing a correspondence between the position of the object in the first image data and the position of the object in the second image data; an estimating step of estimating a position of the first camera device relative to the second camera device based on the correspondence establishment result and the second distance data; a distance calculation step of calculating first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; as well as A parameter calculation step of calculating the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information, wherein the pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first camera device at different timings.

2. A correction parameter calculation method for calculating a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation method comprising: a first acquisition step of acquiring, from a first imaging device, first image data obtained by imaging the object; a second acquisition step of acquiring, from a second imaging device, second distance data from the second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; a third acquisition step of acquiring displacement direction information indicating a direction of displacement of the object in three dimensions; a step of detecting a shape of the first imaging device in the second image data; an estimating step of estimating the position of the first camera device based on the detection result; a distance calculation step of calculating first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; as well as A parameter calculation step of calculating the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information, wherein the pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first camera device at different timings.

3. The correction parameter calculation method according to claim 1 or 2, There are two or more measurement points in the object where the displacement is to be measured, In the distance calculation step, two or more first distance data from each of two or more measurement points to the first imaging device are calculated. In the parameter calculation step, based on the two or more first distance data, a correction value corresponding to each of the two or more measurement points is calculated as the correction parameter.

4. The correction parameter calculation method according to claim 3, In the distance calculation step, a plurality of first distance data from each of a plurality of surface points of the object including the two or more measurement points to the first imaging device are calculated. In the parameter calculation step, a correction value corresponding to each of the plurality of surface points is calculated based on the plurality of first distance data, and a correction value map is generated as the correction parameter based on the calculated plurality of correction values.

5. The correction parameter calculation method according to any one of claims 1, 2 and 4, The displacement direction information is obtained via an input device.

6. The correction parameter calculation method according to any one of claims 1, 2, and 4, The first imaging device includes a first camera. The second imaging device includes a second camera for capturing the second image data and a depth sensor for measuring the second distance data. The first camera has a higher resolution or a higher frame rate than the second camera.

7. The correction parameter calculation method according to any one of claims 1, 2, and 4, The object is assigned object displacement information, wherein the object displacement information indicates information related to a displacement direction of the object. The displacement direction information is acquired based on the first image data obtained by capturing an image of the object including the object displacement information.

8. A correction parameter calculation method for calculating a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation method comprising: a first acquisition step of acquiring, from a first imaging device, first image data obtained by imaging the object; a second acquisition step of acquiring, from a second imaging device, second distance data from the second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; a third acquisition step of acquiring displacement direction information indicating a direction of displacement of the object in three dimensions; a step of detecting a shape of the first imaging device in the second image data; an estimating step of estimating the position and posture of the first camera device based on the detection result; a distance calculation step of calculating first distance data from the first imaging device to the object based on the position and posture of the first imaging device and the second distance data; as well as A parameter calculation step of calculating the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information, wherein the pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first camera device at different timings.

9. A method for calculating displacement, comprising: a fourth obtaining step of obtaining the correction parameter calculated using the correction parameter calculation method according to any one of claims 1, 2, 4, and 8; a fifth obtaining step of obtaining the two or more third image data; and The conversion step converts the pixel shift amounts in the two or more third image data into the actual size shift amounts based on the correction parameters.

10. A correction parameter calculation device for calculating a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation device comprising: a first acquisition unit that acquires first image data obtained by capturing the object from a first imaging device; a second acquiring unit that acquires, from a second imaging device, second distance data from the second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object; a third acquisition unit that acquires displacement direction information indicating a direction of displacement of the object in three dimensions; a matching unit that establishes a correspondence between the position of the object in the first image data and the position of the object in the second image data; a position estimating unit configured to estimate a position of the first imaging device relative to the second imaging device based on the association result and the second distance data; a distance calculation unit that calculates first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; as well as A parameter calculation unit calculates the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information, wherein the pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first camera device at different timings.

11. A correction parameter calculation device for calculating a correction parameter for measuring an actual size displacement using an image, the actual size displacement being an actual size value representing a displacement of a motion of an object, the correction parameter calculation device comprising: a first acquisition unit that acquires first image data obtained by capturing the object from a first imaging device; a second acquiring unit that acquires, from a second imaging device, second distance data from the second imaging device disposed at a position different from that of the first imaging device to the object, and second image data obtained by imaging the object and the first imaging device; a third acquisition unit that acquires displacement direction information indicating a direction of displacement of the object in three dimensions; a position estimating unit that detects the shape of the first imaging device in the second image data and estimates the position of the first imaging device based on the detection result; a distance calculation unit that calculates first distance data from the first imaging device to the object based on the position of the first imaging device and the second distance data; as well as A parameter calculation unit calculates the correction parameter for converting the pixel displacement amount of the measurement point of the object into the actual size displacement amount using the first distance data and the displacement direction information, wherein the pixel displacement amount of the measurement point of the object is based on two or more third image data captured by the first camera device at different timings.

12. A displacement calculation device comprising: a fourth acquiring unit for acquiring the correction parameter calculated using the correction parameter calculating apparatus according to claim 10 or 11; a fifth acquiring unit that acquires the two or more third image data; and The conversion unit converts the pixel shift amounts in the two or more third image data into the actual size shift amounts based on the correction parameters.

Images