Three-dimensional displacement measuring method and three-dimensional displacement measuring device

Abstract

To provide a method of effectively measuring the displacement of a subject over time in a three-dimensional space based on a plurality of images obtained by capturing images of the subject from a plurality of viewpoints.SOLUTION: A three-dimensional displacement measuring method includes: a camera calibration step (S4100) of calculating camera parameters of N (N is an integer greater than 1) cameras based on M (M is an integer greater than N) calibration images imaged at M mutually different viewpoints by multiple cameras including N cameras; and a displacement measurement step (S4200) of measuring displacement of a subject over time in a three-dimensional space based on (1) N images generated by imaging the subject by means of each of the N cameras at each of different time points and (2) the camera parameters. Therein, N is determined based on the camera parameters and accuracy of the displacement measurement.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a three-dimensional displacement measuring method and a three-dimensional displacement measuring device. [Background technology]

[0002] In the field of computer vision, 3D reconstruction techniques match feature points between multiple 2D images to estimate the position and orientation of the camera and the 3D position of the subject. In addition, 3D reconstruction techniques involve camera calibration and 3D point cloud reconstruction.

[0003] For example, the device described in Patent Document 1 performs calibration among three or more cameras, and converts each camera coordinate system into a virtual camera coordinate system of an arbitrary viewpoint based on the acquired camera parameters. In the virtual camera coordinate system, the device performs correspondence between images after the coordinate conversion by block matching, and estimates distance information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-250452 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-described conventional techniques, it is not possible to effectively measure the displacement of a subject over time in three-dimensional space based on a plurality of images obtained by capturing images of the subject from a plurality of viewpoints.

[0006] Therefore, the present disclosure provides a technique that can effectively measure the displacement of a subject over time in three-dimensional space based on a plurality of images obtained by capturing images of the subject from a plurality of viewpoints. [Means for solving the problem]

[0007] A three-dimensional displacement measurement method according to one embodiment of the present disclosure includes a camera calibration step of calculating camera parameters of N cameras based on M calibration images captured from M (M is an integer greater than N) different viewpoints by a plurality of cameras including N cameras (N is an integer greater than 1), and a displacement measurement step of measuring the displacement of the subject over time in three-dimensional space based on (1) N images generated by capturing images of the subject by each of the N cameras at different times, and (2) the camera parameters, where N is determined based on the camera parameters and the accuracy of the displacement measurement.

[0008] A three-dimensional displacement measurement method according to one embodiment of the present disclosure includes a camera calibration step of calculating camera parameters of a plurality of cameras including N (N is an integer greater than 1) cameras based on M calibration images captured from M (M is an integer greater than N) different viewpoints; and a displacement measurement step of measuring the displacement of the subject over time in three-dimensional space based on (1) N images generated by capturing images of a subject by each of the N cameras at different times and (2) the camera parameters. Effect of the Invention

[0009] According to the present disclosure, it is possible to effectively measure the displacement of a subject over time in three-dimensional space based on a plurality of images obtained by capturing images of the subject from a plurality of viewpoints. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing how three-dimensional displacement measurement is performed in the embodiment. [Diagram 2] FIG. 2 is a diagram showing the principle of three-dimensional modeling in the embodiment. [Diagram 3] FIG. 3 is a diagram showing the principle of three-dimensional displacement measurement in the embodiment. [Figure 4A]FIG. 4A is a diagram showing an example of a shooting cycle and an exposure time of a camera according to an embodiment. [Figure 4B] FIG. 4B is a diagram showing another example of the shooting cycle and exposure time of the camera in the embodiment. [Diagram 5] FIG. 5 is a block diagram of the three-dimensional displacement measuring device according to the embodiment. [Figure 6] FIG. 6 is a flowchart of the three-dimensional reconstruction system according to the embodiment. [Figure 7] FIG. 7 is a block diagram of a control unit according to the embodiment. [Figure 8] FIG. 8 is a flowchart showing the operation of the control unit in the embodiment. [Figure 9] FIG. 9 is a diagram showing the base line length, the distance between the imaging target and the camera, and the convergence angle used in three-dimensional displacement measurement in the embodiment. [Figure 10] FIG. 10 is a diagram showing a method of determining a viewpoint position used for camera calibration in the embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a multi-view frame set according to the embodiment. [Figure 12] FIG. 12 is a block diagram of a three-dimensional image processing unit according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing the operation of the three-dimensional image processing unit in the embodiment. [Figure 14] FIG. 14 is a flowchart showing the operation of the three-dimensional displacement measuring device according to the embodiment when generating a free viewpoint video. [Figure 15] FIG. 15 is a flowchart showing the operation of the 3D image processing unit when generating a free viewpoint video. [Figure 16] FIG. 16 is a block diagram of a modified example of the three-dimensional image processing unit. [Figure 17] FIG. 17 is a flowchart showing the operation of the modified example of the three-dimensional image processing unit. [Figure 18] FIG. 18 is a diagram showing an overview of three-dimensional space recognition. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (Findings on which this disclosure is based) In conventional technology, three-dimensional space recognition is realized by reconstructing (modeling) the three-dimensional shape of the subject or by using the reconstruction result. A three-dimensional space reconstruction device that reconstructs the three-dimensional shape of the subject performs modeling using video data provided by an imaging system including multiple cameras that capture images of the same scene, and camera parameters indicating the position, orientation, etc. of each camera (hereinafter referred to as "position and orientation") obtained by calibration. The position of the camera corresponds to the viewpoint.

[0012] There is also a demand for grasping the displacement of measurement objects such as structures (e.g., infrastructure structures), machine parts, etc., in three-dimensional space. However, conventional techniques have not been able to effectively measure the displacement of the measurement object over time in three-dimensional space. For example, conventional techniques have been insufficient in reducing the processing load required to measure the displacement of the object over time in three-dimensional space.

[0013] In general, in camera calibration processing and three-dimensional displacement measurement processing, there is a trade-off relationship in which the more the number of viewpoints, that is, the more images, the greater the processing load, but the higher the accuracy. In response to this trade-off relationship, the inventors have discovered that by prioritizing the accuracy of camera calibration over the accuracy of three-dimensional displacement measurement, the processing load can be reduced and the accuracy of the two processes as a whole can be improved.

[0014] Therefore, a three-dimensional displacement measurement method according to one embodiment of the present disclosure includes a camera calibration step of calculating camera parameters of N cameras based on M calibration images captured from M (M is an integer greater than N) different viewpoints by a plurality of cameras including N cameras (N is an integer greater than 1), and a displacement measurement step of measuring the displacement of the subject over time in three-dimensional space based on (1) N images generated by capturing images of the subject by each of the N cameras at different times, and (2) the camera parameters.

[0015] As a result, in the three-dimensional displacement measurement method, a number of viewpoints M, which is greater than the number of viewpoints N in the three-dimensional displacement measurement process, is used for the camera calibration process so as to improve the accuracy of the camera parameters, thereby improving the accuracy of the three-dimensional displacement measurement process.

[0016] Furthermore, for example, in a three-dimensional displacement measuring method according to one embodiment of the present disclosure, the displacement measuring step may include a corresponding point calculation step of determining a first corresponding point of the subject included in a first image included in the N images and a second corresponding point included in a second image included in the N images and corresponding to the first corresponding point, a three-dimensional point calculation step of determining a three-dimensional point of the subject in the three-dimensional space based on the camera parameters, the first corresponding point, and the second corresponding point, and a displacement calculation step of determining the displacement of the three-dimensional point over time from a plurality of the three-dimensional points corresponding to the different times.

[0017] As a result, the three-dimensional displacement measurement method can determine the displacement of the subject in three-dimensional space by using, for example, an image matching technique and the principle of triangulation to determine the displacement of a three-dimensional point over time based on camera parameters, a first corresponding point, and a second corresponding point.

[0018] Furthermore, for example, in the three-dimensional displacement measuring method according to one aspect of the present disclosure, N may be determined based on the camera parameters and the accuracy of the displacement measurement.

[0019] As a result, in the three-dimensional displacement measuring method, the number of camera viewpoints N can be determined so as to satisfy the accuracy of the measurement performed in the displacement measuring step. Therefore, the three-dimensional displacement measuring method can accurately obtain the displacement of the subject in three-dimensional space.

[0020] Furthermore, the inventors of the present application have found that the accuracy of camera calibration is improved by arranging the viewpoints used for camera calibration as follows, rather than arranging them evenly. For example, in a three-dimensional displacement measuring method according to one aspect of the present disclosure, the N cameras include a first camera and a second camera adjacent to each other, the M viewpoints include a first midpoint between a first position where the first camera is arranged and a second position where the second camera is arranged, and the midpoint between the first position and the first midpoint is defined as a second midpoint, the number of viewpoints included in the M viewpoints and provided between the second midpoint and the first position may be greater than the number of viewpoints included in the M viewpoints and provided between the first midpoint and the second midpoint.

[0021] That is, in this embodiment, the viewpoints used for camera calibration are set more in the range closer to the second midpoint than in the range farther from the second midpoint based on the position of the viewpoint used for three-dimensional displacement measurement. Such an uneven arrangement improves the accuracy of camera calibration.

[0022] For example, in a non-uniform arrangement of viewpoints for camera calibration, the camera calibration step does not require providing a viewpoint between the first midpoint and the second midpoint.

[0023] Also, for example, a three-dimensional displacement measuring device according to one embodiment of the present disclosure includes a camera calibration unit that calculates camera parameters of the N cameras based on M calibration images captured by a plurality of cameras including N cameras (N is an integer greater than 1) from M different viewpoints (M is an integer greater than N), and a displacement measuring unit that measures the displacement of the subject over time in three-dimensional space based on (1) N images generated by capturing images of the subject by each of the N cameras at different times, and (2) the camera parameters.

[0024] As a result, the three-dimensional displacement measuring device can achieve the same effects as the three-dimensional displacement measuring method according to one aspect of the present disclosure.

[0025] Hereinafter, the embodiment will be described in detail with reference to the drawings.

[0026] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the scope of the claims. Furthermore, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept are described as optional components. Furthermore, each figure is not necessarily a strict illustration. In each figure, substantially the same configuration is given the same reference numeral, and duplicated explanations are omitted or simplified.

[0027] (Embodiment) The three-dimensional displacement measuring device according to this embodiment can reconstruct a time-series three-dimensional model in which coordinate axes coincide between times. Here, three-dimensional reconstruction in this disclosure is defined. Video or images of a subject existing in real space captured from different viewpoints by multiple cameras are called multi-view video or multi-view image. In other words, a multi-view image includes multiple two-dimensional images of the same subject captured from different viewpoints. In addition, multi-view images captured in time series are called multi-view video. Reconstructing a subject in three-dimensional space using these multi-view images is called three-dimensional reconstruction.

[0028] Specifically, the 3D displacement measuring device first obtains a 3D model at each time by performing 3D reconstruction independently for each time. Next, the 3D displacement measuring device detects a stationary camera and a stationary object (stationary 3D point), and uses the detected stationary camera and stationary object to align the coordinates of the 3D model between times, thereby generating a time-series 3D model with aligned coordinate axes.

[0029] As a result, the 3D displacement measuring device can generate a time-series 3D model that has high accuracy in the relative positional relationship between the subject and camera at each time, regardless of whether the camera is fixed or not, or whether the subject is moving or stationary, and that can use transition information in the time direction.

[0030] FIG. 1 is a schematic diagram of a three-dimensional displacement measurement according to the embodiment. In the three-dimensional displacement measurement method according to the embodiment, the subject 1001 is photographed by the camera 100-1 and the camera 100-2. Here, the subject 1001 may be a compressor or the like fixed at a predetermined position. The camera 100-1 and the camera 100-2 may be fixed cameras fixed at different positions in a direction that includes the subject 1001 in the photographing range. For example, the same space or the same subject 1001 is photographed from multiple viewpoints by one or more calibrated cameras (for example, fixed cameras), so that the photographed space or the subject 1001 can be three-dimensionally reconstructed. That is, by associating feature points and the like between multiple two-dimensional images photographed from multiple viewpoints, the position of the camera, the orientation of the camera, and the three-dimensional displacement or three-dimensional position of the subject 1001 can be estimated. That is, in the three-dimensional displacement measurement device, the camera calibration and the three-dimensional displacement measurement of the subject 1001 are performed using multiple two-dimensional images photographed from multiple viewpoints. The three-dimensional displacement measuring apparatus may further perform three-dimensional reconstruction of the subject 1001 .

[0031] Three-dimensional displacement measurement is useful, for example, for infrastructure inspection systems. For example, there is a need for a measuring device to measure the vibration of a wall, such as a soundproof wall of a bullet train, in the depth direction of the wall. There is also a need for a measuring device to measure the displacement of equipment that generates vibration during operation, such as a compressor.

[0032] FIG. 2 is a diagram showing the principle of three-dimensional modeling in the embodiment. First, three-dimensional modeling in the embodiment is defined. In the present embodiment, three-dimensional modeling refers to a process in which a three-dimensional displacement measuring device reconstructs a subject in a three-dimensional space using two-dimensional multi-viewpoint images obtained by photographing a subject existing in real space from different viewpoints using a plurality of cameras. In the following, three-dimensional modeling may also be referred to as three-dimensional reconstruction.

[0033] As shown in FIG. 2, a triangle indicated by a dotted line is formed by using camera coordinates and camera parameters of at least two viewpoints. In the three-dimensional displacement measuring method in the embodiment, each point on the image plane represented by the coordinate system on the image plane of the image captured by each camera is reconstructed as a three-dimensional model using a world coordinate system based on the principle of triangulation using this triangle. The coordinate system on the image plane is, for example, image plane coordinate 1 or image plane coordinate 2 shown in FIG. 2. The three-dimensional model is a reconstructed (i.e., three-dimensional reconstruction) of the subject in a three-dimensional space represented by the world coordinate system. The three-dimensional model represents the three-dimensional position of each of a plurality of points on the subject reflected in a plurality of two-dimensional images captured from multiple viewpoints. Here, the three-dimensional position is represented by, for example, three-value information consisting of an X component, a Y component, and a Z component represented by an orthogonal coordinate system (XYZ axes) in a three-dimensional space. Note that the information corresponding to each point on the three-dimensional model may include not only information representing the three-dimensional position of each point, but also information representing the color of each point and the surface shape of each point and its surroundings.

[0034] In the three-dimensional modeling, the camera parameters of each camera may be acquired in advance or may be estimated simultaneously with the creation of the three-dimensional model. Here, the camera parameters consist of internal parameters and external parameters. The internal parameters include the focal length of the camera, the image center, etc., and the external parameters include the three-dimensional position of the camera, the three-dimensional orientation of the camera, etc.

[0035] In Figure 2, a typical pinhole camera model is used as an example. In this example, the lens distortion of the camera is not taken into consideration. If lens distortion is taken into consideration during 3D modeling, the position of each point used in the 3D model is the corrected position obtained by normalizing the position of the point in the coordinate system on the image plane of the image captured by each camera using a distortion model.

[0036] FIG. 3 is a diagram showing the principle of three-dimensional displacement measurement in an embodiment. The three-dimensional displacement measurement method of the present disclosure is defined. Calculating the displacement of an object 1001 in three-dimensional space using two-dimensional images captured from multiple viewpoints is called three-dimensional displacement measurement. FIG. 3 shows the mechanism of three-dimensional displacement measurement. By using the respective camera coordinates and camera parameters in at least two viewpoints, a triangle as shown by the dotted line in FIG. 3 is formed. Based on the principle of triangulation using this triangle, points on an image captured from multiple viewpoints and expressed in a coordinate system on an image plane are reconstructed in a world coordinate system. The coordinate system on the image plane is, for example, image plane coordinates 1 or image plane coordinates 2 shown in FIG. 3.

[0037] The three-dimensional model of the object 1001 is represented by the three-dimensional positions of multiple points on the object 1001 captured in multiple two-dimensional images taken from multiple viewpoints. When the object 1001 moves or vibrates, the positions (X t ,Y t ,Z t ) and (X t+i ,Y t+i ,Z t+i ) is reconstructed to obtain the displacement (X t+i -X t ,Y t+i -Y t ,Z t+i -Z t) is calculated. In the three-dimensional displacement measurement, for example, only the depth direction (Z component) may be measured. That is, in the three-dimensional displacement measurement, a so-called depth map may be created. In addition, the three-dimensional displacement measurement of the present disclosure may measure the displacement in any two components of the X component, the Y component, and the Z component.

[0038] For example, when the subject 1001 is a vibrating body such as a compressor, the time difference i is the reciprocal of the frequency at which the resonant frequency of the vibrating body can be sampled. However, the time difference i is not limited to this. For example, when the subject 1001 is a structure and the growth of a crack in the structure is to be measured, the time difference i is determined according to the durability years required for the structure.

[0039] In the above example, the positions of the three-dimensional points are reconstructed at time t and time t+i. However, the number of times the three-dimensional point positions are reconstructed is not limited to two, and may be three or more times for every time difference i. Furthermore, when the number of times the three-dimensional point positions are reconstructed is three or more times for every time difference i, the three-dimensional displacement measurement of the present disclosure may calculate not only the displacement of the three-dimensional point, but also the velocity and acceleration of the displacement.

[0040] The specific calculations in the 3D displacement measurement are as follows. First, in camera coordinate system 1, the matrix representing the internal parameters of the camera is K1, the translation vector representing the parallel translation among the external parameters is T1, and the rotation matrix representing the rotation is R1. In addition, in camera coordinate system 2, the matrix representing the internal parameters of the camera is K2, the translation vector representing the parallel translation among the external parameters is T2, and the rotation matrix representing the rotation is R2.

[0041] Next, by image matching, a corresponding point between image plane coordinate 1 and image plane coordinate 2 is calculated for a point on the object 1001. Here, in the image matching, block matching may be performed to search for a position where a difference value of a block, which is a rectangular area in an image, is minimum. Specifically, in the block matching, the degree of match of the corresponding rectangular areas included in each image is evaluated by a match evaluation function such as the sum of luminance differences (SAD: Sum of Absolute Difference) or the sum of squares of luminance differences (SSD: Sum of Squared Difference), and a position where the match evaluation value is maximum or minimum is searched for. As the match evaluation function, cross correlation (CC) or normalized cross correlation (NCC) may be used. In addition, in the image matching, feature matching may be performed to calculate a feature vector from a small area of ​​an image using SIFT (Scale-Invariant Feature Transform), SURF (Speed ​​Up Robust Features), etc., and search for a position where the difference value is minimum.

[0042] In image matching, the corresponding point at time t in the image plane coordinate 1 is a 1,t =(x 1,t ,y 1,t ), and the corresponding point on the image plane coordinate 2 is b 2,t =(x 2,t ,y 2,t ) The corresponding point a in image plane coordinate 1 1,t is a specific example of the first corresponding point, and the corresponding point b 2,t is a specific example of the second corresponding point. Also, the corresponding point at the image plane coordinate 1 at time t+i is a 1,t+i =(x 1,t+i ,y 1,t+i ), and the corresponding point on the image plane coordinate 2 is b 2,t+i =(x 2,t+i ,y 2,t+i ) The process described here corresponds to the corresponding point calculation step.

[0043] Then, using the intrinsic and extrinsic parameters of the camera, 1,t , b 2,t , a 1,t+i , b 2,t+i Calculate the 3D points that represent the 3D positions corresponding to each of the points. 1,t The three-dimensional point corresponding to A 1,t , b at time t 2,t The three-dimensional point corresponding to B 2,t , a at time t+i 1,t+i The three-dimensional point corresponding to A 1,t+i , b at time t+i 2,t+i The three-dimensional point corresponding to B 2,t+i Let us assume that.

[0044] The conversion between each corresponding point on the two-dimensional image and the three-dimensional point corresponding to each corresponding point is calculated by the following equations (1) to (4).

[0045] a 1,t =K1[R1|T1]A 1,t Formula (1) b 2,t =K2[R2|T2]B 2,t Formula (2) a 1,t+i =K1[R1|T1]A 1,t+i Formula (3) b 2,t+i =K2[R2|T2]B 2,t+i Formula (4)

[0046] At this time, the 3D point A that corresponds to the corresponding point on the 2D image is 1,t and B. 2,t are the same point in the world coordinate system, but the three-dimensional point A calculated using equations (1) and (2) 1,t and B. 2,t do not necessarily coincide. Similarly, 3D point A 1,t+i and B. 2,t+i is the same point in the world coordinate system, but the 3D point A calculated using equations (3) and (4) 1,t+i and B. 2,t+i In that case, as shown in equations (5) and (6), the three-dimensional point A 1,t and B. 2,tor 3D point A 1,t+i and B. 2,t+i A three-dimensional point that corresponds to the corresponding point on the two-dimensional image may be calculated by calculating the midpoint between the two-dimensional image and the three-dimensional image.

[0047] (X t ,Y t ,Z t )=(A 1,t +B 2,t ) / 2 Equation (5) (X t+i ,Y t+i ,Z t+i )=(A 1,t+i +B 2,t+i ) / 2 Equation (6)

[0048] In addition, when the 3D point corresponding to the corresponding point on the 2D image does not match, the method of determining the 3D point is not limited to the above. 1,t and B. 2,t or 3D point A 1,t+i and B. 2,t+i Alternatively, only one of the three-dimensional points A and B may be used as a three-dimensional point corresponding to the corresponding point on the two-dimensional image at each time. In addition, although the formulas (5) and (6) show a method for calculating the midpoint between two three-dimensional points, other calculation methods may be used. For example, when a three-dimensional point A 1,t and B. 2,t A 3D point at time t obtained by weighting the 1,t+i and B. 2,t+i The 3D point at time t+i obtained by weighting the 3D points may be used as the 3D points corresponding to each time. The process described here corresponds to the 3D point calculation step.

[0049] The calculated three-dimensional points are not limited to specific points on the subject, but may be any points on the subject as long as they can be determined from corresponding points.

[0050] From the above, the three-dimensional displacement V between time t and time t+i t,t+i is calculated using equation (7).

[0051] V t,t+i =(X t+i -X t ,Y t+i -Y t , Z t+i -Z t ) Formula (7)

[0052] The process described here corresponds to the displacement calculation step.

[0053] Furthermore, the above-mentioned method of measuring three-dimensional displacement is just one example, and the present invention is not limited to this.

[0054] Fig. 4A is a diagram showing an example of the shooting cycle and exposure time of the camera in the embodiment. Fig. 4B is a diagram showing another example of the shooting cycle and exposure time of the camera in the embodiment. The horizontal direction in Fig. 4A and Fig. 4B indicates time, and the time when a rectangular signal is on indicates that the camera 100-1 or the camera 100-2 is exposed. When an image is captured by the camera 100-1 or the camera 100-2, the time that the shutter is open is called the exposure time.

[0055] During the exposure time, the scene exposed to the image sensor through the lens is captured as an image. In Fig. 4A, the exposure times of the frames captured by the two cameras 100-1 and 100-2 with different viewpoints overlap. As a result, the frames captured by the two cameras 100-1 and 100-2 are determined to be synchronized frames containing the scene at the same time.

[0056] On the other hand, in Fig. 4B, since there is no overlap in the exposure times of the two cameras 100-1 and 100-2, the frames captured by the two cameras 100-1 and 100-2 are determined to be asynchronous frames that do not contain scenes from the same time. Shooting synchronous frames with multiple cameras, as in Fig. 4A, is called synchronous shooting.

[0057] Next, the configuration of a three-dimensional displacement measuring device according to this embodiment will be described. Fig. 5 is a block diagram of a three-dimensional displacement measuring device according to the embodiment. Three-dimensional reconstruction system 3 shown in Fig. 5 includes a plurality of cameras 100-1 to 100-n, a three-dimensional displacement measuring device 2000, and a control unit 300. Furthermore, three-dimensional displacement measuring device 2000 includes a receiving unit 210, a storage unit 220, an acquiring unit 230, a three-dimensional image processing unit 240, and a transmitting unit 250.

[0058] The multiple cameras 100-1 to 100-n capture an object 1001 in a shooting space 1000, and transmit a multi-viewpoint video, which is a plurality of captured images, to the receiving unit 210. The multi-viewpoint video may be transmitted via a public communication network such as the Internet, or a private communication network. Alternatively, the multi-viewpoint video may be temporarily stored in an external storage device such as a hard disk drive (HDD) or a solid-state drive (SSD), and input to the three-dimensional displacement measuring device 2000 when necessary. Alternatively, the multi-viewpoint video may be temporarily transmitted to an external storage device such as a cloud server via a network and stored therein. Then, the multi-viewpoint video may be transmitted to the three-dimensional displacement measuring device 2000 when necessary.

[0059] Each of the n cameras 100-1 to 100-n may be a fixed camera such as a surveillance camera, or a non-fixed camera. That is, the n cameras 100-1 to 100-n may be fixed cameras fixed in different positions and with different attitudes, or may be mobile cameras such as video cameras, smartphones, or wearable cameras, or may be mobile cameras such as drones with a shooting function. Here, n is an integer of 2 or more.

[0060] Furthermore, camera identification information such as a camera ID that identifies the camera that captured the images may be added to the multi-viewpoint images as header information of the images or frames.

[0061] A plurality of cameras 100-1 to 100-n may be used for synchronous shooting, in which the subject is shot at the same time in each frame. Alternatively, the times of the built-in clocks of the plurality of cameras 100-1 to 100-n may be synchronized, and shooting time information may be added to the shot images for each video or frame, without synchronous shooting, or an index number indicating the shooting order may be added.

[0062] Information indicating whether the images were shot synchronously or asynchronously may be added as header information to each image set, each image, or each frame of the multi-view images.

[0063] The control unit 300 is realized, for example, by using a CPU (Central Processing Unit), and reads and executes a program stored in the storage unit 220. The control unit 300 determines the number of viewpoints and viewpoint positions used by the three-dimensional reconstruction system 3 to perform one or more processes among the camera calibration, three-dimensional modeling, and three-dimensional displacement measurement in the three-dimensional image processing unit 240. Here, the number of viewpoints refers to the number of positions at which two-dimensional images used in one or more processes among the camera calibration, three-dimensional modeling, and three-dimensional displacement measurement are captured, and specifically, may refer to the number of cameras, or may refer to the number of points at which images are captured by fixed cameras and moving cameras. The viewpoint position refers to the position at which two-dimensional images used in one or more processes among the camera calibration, three-dimensional modeling, and three-dimensional displacement measurement are captured, and specifically, refers to the position of the camera.

[0064] Then, receiving unit 210 receives the multi-view video captured by cameras 100-1 to 100-n, and inputs the received multi-view video to storage unit 220.

[0065] The storage unit 220 stores the input multi-view video. The storage unit 220 also temporarily stores video data of the shooting space 1000, meta information such as a timestamp added to the video data, camera parameters of the cameras 100-1 to 100-n, and shooting settings such as the currently applied frame rate or resolution. The storage unit 220 also stores a program that is read and executed by the control unit 300. The storage unit 220 may be a volatile memory such as a RAM, a DRAM, or an SRAM. The storage unit 220 may also be realized by a non-volatile memory such as a ROM or a flash memory.

[0066] The acquisition unit 230 selects frames from the multiple multi-viewpoint videos stored in the storage unit 220, and inputs the selected frames to the 3D image processing unit 240 as a multi-viewpoint frame set.

[0067] The three-dimensional image processing unit 240 performs camera calibration using the input multi-view frame set, and performs three-dimensional modeling of the subject or three-dimensional displacement measurement of the subject using the result of the camera calibration. The three-dimensional image processing unit 240 calculates camera parameters as a result of the camera calibration, and outputs them to the transmission unit 250.

[0068] The transmitting unit 250 transmits at least one of the camera parameters, the three-dimensional model of the subject, and the three-dimensional displacement of the subject calculated by the three-dimensional image processing unit 240 to the outside of the three-dimensional displacement measuring device 2000.

[0069] The receiving unit 210, the acquiring unit 230, the three-dimensional image processing unit 240, and the transmitting unit 250 may each be realized by a processor and a memory. Also, the receiving unit 210, the acquiring unit 230, the three-dimensional image processing unit 240, and the transmitting unit 250 may each be realized by a dedicated circuit.

[0070] FIG. 6 is a flowchart of the three-dimensional reconstruction system according to the embodiment.

[0071] First, the control unit 300 determines the number of viewpoints and viewpoint positions used by the 3D reconstruction system 3 to perform one or more processes of camera calibration, 3D modeling, or 3D displacement measurement (step S2100). The details of determining the number of viewpoints and viewpoint positions will be described later.

[0072] Next, cameras 100-1 to 100-n capture multi-viewpoint videos of a subject based on the number of viewpoints and viewpoint positions determined by control unit 300 (step S2200).

[0073] Then, receiving unit 210 receives the multi-viewpoint videos captured by a plurality of cameras among cameras 100-1 to 100-n, and inputs them to storage unit 220 (Step S2300).

[0074] Next, storage unit 220 stores the input multi-view video (step S2400).

[0075] Next, acquisition unit 230 selects frames from the stored multi-view video based on the number of viewpoints and viewpoint positions determined by control unit 300, and inputs the selected frames to three-dimensional image processing unit 240 as a multi-view frame set (step S2500).

[0076] Here, the multi-view frame set may be composed of a plurality of frames, one frame selected from each of all the videos captured by the cameras 100-1 to 100-n. The multi-view frame set may also be composed of a plurality of frames, at least one frame selected from all the videos captured by the cameras 100-1 to 100-n. The multi-view frame set may also be composed of a plurality of frames, two or more videos selected from the multi-view videos, and one frame selected from each of the selected videos. The multi-view frame set may also be composed of a plurality of frames, two or more videos selected from the multi-view videos, and at least one frame selected from each of the selected videos.

[0077] In addition, if camera-specific information is not added to each frame of a multi-view frame set, the camera-specific information may be added individually to the header information of each frame, or the camera-specific information may be added collectively to the header information of the multi-view frame set.

[0078] Furthermore, if an index number indicating the shooting time or shooting order is not added to each frame of a multi-view frame set, an index number indicating the shooting time or shooting order may be added individually to the header information of each frame. In a similar case, an index number indicating the shooting time or shooting order may be added collectively to the header information of the multi-view frame set.

[0079] Then, the three-dimensional image processing unit 240 performs camera calibration using the input multi-view frame set, and performs three-dimensional modeling or three-dimensional displacement measurement using the result of the camera calibration (step S2600). The details of the three-dimensional modeling or three-dimensional displacement measurement will be described later.

[0080] Furthermore, the processes of steps S2500 and S2600 are repeated for each multi-view frame set. Here, in the repeatedly performed steps S2500 and S2600, camera calibration may be performed only once. In this case, the result of the camera calibration performed initially may be repeatedly used in the repeatedly performed steps S2500 and S2600.

[0081] Next, transmission unit 250 transmits the camera parameters, the three-dimensional model of the subject, or the three-dimensional displacement of the subject to the outside of three-dimensional displacement measuring apparatus 2000 (step S2700).

[0082] Here, the three-dimensional reconstruction system 3 ends the processing.

[0083] The camera parameters, the three-dimensional model of the subject, or the three-dimensional displacement of the subject calculated in step S2600 may be used in processes subsequent to step S2600.

[0084] The camera parameters, the three-dimensional model of the subject, or the three-dimensional displacement of the subject output in step S2700 may be stored in storage unit 220.

[0085] 7 is a block diagram of a control unit in the embodiment. The control unit 300 includes a measurement use determination unit 310 and a camera calibration use determination unit 320. The measurement use determination unit 310 and the camera calibration use determination unit 320 may be realized by a processor and a memory, or may be realized by a dedicated logic circuit.

[0086] The measurement determination unit 310 determines the number of viewpoints, viewpoint positions, etc. for capturing two-dimensional images used when the three-dimensional reconstruction system 3 performs three-dimensional displacement measurement of the subject 1001 .

[0087] Furthermore, the camera calibration determination unit 320 determines the number of viewpoints, viewpoint positions, etc., for capturing two-dimensional images used when the three-dimensional reconstruction system 3 performs camera calibration.

[0088] The number of viewpoints used by the three-dimensional reconstruction system 3 to perform camera calibration, three-dimensional modeling, and three-dimensional displacement measurement may be different. The control unit 300 may determine only the viewpoint position. That is, the measurement viewpoint determination unit 310 may determine only the viewpoint position of the viewpoint for capturing a two-dimensional image used when the three-dimensional reconstruction system 3 performs three-dimensional displacement measurement of the subject 1001. The camera calibration viewpoint determination unit 320 may determine only the viewpoint position of the viewpoint for capturing a two-dimensional image used when the three-dimensional reconstruction system 3 performs camera calibration.

[0089] Note that the control unit 300 may determine the number of viewpoints and viewpoint positions of the cameras before the cameras are introduced into the three-dimensional displacement measurement device 2000. Furthermore, the cameras may be introduced according to the number of viewpoints and viewpoint positions of the cameras determined by the control unit 300.

[0090] The control unit 300 determines the number of viewpoints N used for three-dimensional displacement measurement and the number of viewpoints M used for camera calibration so as to satisfy the relationship M > N. Furthermore, the control unit 300 determines the viewpoints used for three-dimensional displacement measurement and the viewpoints used for camera calibration so that the viewpoints used for three-dimensional displacement measurement are included in the viewpoints used for camera calibration (i.e., the two viewpoints match).

[0091] The cameras 100-1 to 100-n or the acquisition unit 230 use N fixed cameras as N viewpoints used for three-dimensional displacement measurement. In other words, for three-dimensional displacement measurement, the cameras do not capture images at the MN viewpoints. Furthermore, of the M viewpoints used for camera calibration, (MN) fixed cameras may be used as (MN) viewpoints other than the N viewpoints used for three-dimensional displacement measurement, or (MN) or less moving cameras may be used. When (MN) or less moving cameras are used, the (MN) or less moving cameras capture images while moving, thereby realizing capture from (MN) or more viewpoints. Furthermore, of the M viewpoints used for camera calibration, (MN) viewpoints other than the N viewpoints used for three-dimensional displacement measurement may be realized by combining fixed cameras and moving cameras.

[0092] Next, a description will be given of the operation of the control unit 300. Fig. 8 is a flow chart showing the operation of the control unit in this embodiment.

[0093] First, the measurement determination unit 310 determines the number of viewpoints and the viewpoint positions for three-dimensional displacement measurement (step S3100).

[0094] Next, the camera calibration determination unit 320 determines the number of viewpoints and viewpoint positions for camera calibration (step S3200). At this time, the camera calibration determination unit 320 sets the number of viewpoints M for camera calibration to be greater than the number of viewpoints N for three-dimensional displacement measurement, and sets the viewpoints for three-dimensional displacement measurement to be included in the viewpoints for camera calibration.

[0095] Here, the control unit 300 ends the operation. In step S3100, the measurement determination unit 310 may determine the number of viewpoints and the viewpoint positions for three-dimensional modeling.

[0096] Next, an example of a method for determining the viewpoint in control unit 300 will be described. First, the determination of the number of viewpoints N for three-dimensional displacement measurement will be described with reference to Fig. 9. Fig. 9 is a diagram showing the base length, the distance between the imaging target and the camera, and the convergence angle used in three-dimensional displacement measurement in the embodiment. Fig. 9 shows the positional relationship between subject 1001, which is the imaging target, and two cameras 100-1 and 100-2 out of n cameras 100-1 to 100-n.

[0097] The control unit 300 sets the number of viewpoints used in the three-dimensional displacement measurement as N, the distance between the object 1001 and the cameras 100-1 and 100-2 as d, and the accuracy of the three-dimensional modeling or three-dimensional displacement measurement as Δz. These values ​​are arbitrarily determined. The number of viewpoints N may be the number of cameras that can be used in the three-dimensional displacement measurement, or may be the maximum number within a settable range. In the following description, it is assumed that the number of viewpoints N is equal to the number n of the cameras 100. However, the number of viewpoints N does not have to be equal to the number n of the cameras 100. In addition, the distance d between the object 1001 and the cameras 100-1 and 100-2 may be determined from at least one of the size of the shooting space 1000, how the object 1001 appears in the two-dimensional image captured by the cameras 100-1 and 100-2, and the like. Here, the distance d is the shortest distance from the line passing through the positions of the two cameras 100-1 and 100-2 to the object 1001, as shown in FIG. 9. However, the distance d is not limited to the example shown in Fig. 9. For example, the distance d may be the average value of the distances from the subject 1001 to the two cameras 100-1 and 100-2. The control unit 300 calculates the base length L, which is the distance between the two cameras 100-1 and 100-2, by using the accuracy Δz of the three-dimensional modeling or three-dimensional displacement measurement, the focal length f of the camera, the element spacing ΔH of the camera, and the distance d between the subject 1001 and the two cameras 100-1 and 100-2, according to Equation (8).

[0098] L=(d×d×ΔH) / f×Δz Equation (8)

[0099] For example, when the number of viewpoints N used in the three-dimensional displacement measurement is 2, the number of cameras used in the three-dimensional displacement measurement is 2. In this case, the control unit 300 uses the base line length L and the distance d between the subject and the cameras as information for determining the viewpoint positions of the two cameras.

[0100] When the number of viewpoints N used in the three-dimensional displacement measurement is greater than two, the control unit 300 performs the following calculation. First, the control unit 300 determines the convergence angle of the optical axes of the two cameras as θ, and calculates θ using the base line length L and distance d according to equation (9).

[0101] θ=2arctan(L / 2d) Equation (9)

[0102] Moreover, the control unit 300 calculates the value P by the formula (10). Here, the value P is a value indicating the number of cameras used in the three-dimensional reconstruction system 3, which is determined corresponding to the convergence angle θ.

[0103] P=floor(2π / θ) Equation (10)

[0104] If P is larger than N, the control unit 300 calculates N by equation (11).

[0105] N=P formula (11)

[0106] As described above, the control unit 300 determines the number of viewpoints N based on the camera parameters and the required accuracy Δz.

[0107] Furthermore, the base line length L, the distance d, and the convergence angle θ may be used as information for determining the viewpoint position of the camera.

[0108] The positional relationship between the subject 1001 and the two cameras 100-1, 100-2 described above is not limited to the two cameras 100-1, 100-2, but is similar to any other combination of two cameras among the n cameras 100-1 to 100-n.

[0109] Fig. 10 is a diagram showing a method for determining the viewpoint positions used for camera calibration in the embodiment. Fig. 10 is a diagram showing a plurality of cameras viewed along the vertical direction. First, the control unit 300 arbitrarily sets the convergence angle φ between the viewpoints used for camera calibration. Here, if θ>2π-(N-1)×θ, the control unit 300 calculates the number of viewpoints M used for camera calibration by equation (12).

[0110] M=N+floor(θ / φ)×(N-1) Equation (12)

[0111] Here, floor(θ / φ) viewpoints are added between each of the N viewpoints used in the three-dimensional displacement measurement. The floor(θ / φ) viewpoints to be added are added between each of the N viewpoints used in the three-dimensional displacement measurement, as shown in FIG. 10. For example, the control unit 300 adds one or two viewpoints to the intermediate position c between the viewpoint position p1 and the viewpoint position p2. Here, the control unit 300 sets the position p3 to a position obtained by dividing the area between the viewpoint position p1 and the intermediate position c in half, and sets the position p4 to a position obtained by dividing the area between the viewpoint position p2 and the intermediate position c in half. Then, the control unit 300 places the number of viewpoints between the viewpoint position p1 and the position p3, which is the number of floor(θ / φ) viewpoints minus the number of viewpoints placed at the intermediate position c. Similarly, the control unit 300 places the number of viewpoints between the viewpoint position p2 and the position p4, which is the number of floor(θ / φ) viewpoints minus the number of viewpoints placed at the intermediate position c. According to such an arrangement method, for example, the majority of the added viewpoints are arranged at positions closer to the camera arranged at the position corresponding to the viewpoint used in the three-dimensional displacement measurement than the midpoint between the cameras arranged at the positions corresponding to the viewpoint used in the three-dimensional displacement measurement. That is, there are more calibration viewpoints arranged between position p3 (p4) and viewpoint position p1 (p2) than calibration viewpoints arranged between intermediate position c and position p3 (p4). In other arrangement examples, no calibration viewpoint is provided between position p3 (p4) and intermediate position c.

[0112] For this reason, in the three-dimensional displacement measurement method, the viewpoints used for camera calibration are set in a range closer to the position p3 (p4) based on the position of the viewpoint used for three-dimensional displacement measurement, and more in a range farther from the position p3 (p4). Such an uneven arrangement improves the accuracy of camera calibration.

[0113] Next, the details of the multi-view frame set will be described. Fig. 11 is a diagram showing an example of a multi-view frame set in the embodiment. Here, an example will be described in which the acquisition unit 230 determines the multi-view frame set by selecting one frame each from the five cameras 100-1 to 100-5.

[0114] It is also assumed that multiple cameras capture images synchronously. The header information of each frame is assigned a camera ID, 100-1 to 100-5, that identifies the camera that captured the image. The header information of each frame is also assigned a frame number, 001 to Q, that indicates the order of capture within each camera, and frames with the same frame number between cameras indicate that the subject 1001 was captured at the same time.

[0115] The acquisition unit 230 sequentially outputs the multi-view frame sets 200-1 to 200-Q to the three-dimensional image processing unit 240. The three-dimensional image processing unit 240 sequentially performs three-dimensional modeling or three-dimensional displacement measurement using the multi-view frame sets 200-1 to 200-Q by repeated processing.

[0116] Multi-view frame set 200-1 is composed of five frames: frame number 001 of camera 100-1, frame number 001 of camera 100-2, frame number 001 of camera 100-3, frame number 001 of camera 100-4, and frame number 001 of camera 100-5. These five frames are used in iterative process 1 of 3D image processing unit 240 as the first set of frames of the multi-view video. That is, in iterative process 1, a three-dimensional model at the time when frame number 001 was captured is reconstructed, or a three-dimensional displacement is measured.

[0117] 11, in the multi-view frame set 200-2, the frame numbers are updated for all cameras. The multi-view frame set 200-2 is composed of five frames: frame number 002 of camera 100-1, frame number 002 of camera 100-2, frame number 002 of camera 100-3, frame number 002 of camera 100-4, and frame number 002 of camera 100-5. These five frames are used in the repetitive process 2 of the three-dimensional image processing unit 240. That is, in the repetitive process 2, a three-dimensional model at the time when frame number 002 was captured is reconstructed, or a three-dimensional displacement is measured.

[0118] From this point on, even in repetitive process 3 and onwards, the frame numbers are updated for all cameras in the same manner as in repetitive process 2. This enables 3D reconstruction system 3 to reconstruct a 3D model of subject 1001 at each time and to measure the 3D displacement of subject 1001 at each time.

[0119] However, since 3D reconstruction is performed independently at each time, the coordinate axes and scales of the reconstructed 3D models do not necessarily match. In other words, in order to obtain a 3D model of the moving subject 1001, it is necessary to match the coordinate axes and scales at each time.

[0120] In this case, each frame is assigned a shooting time, and based on the shooting time, the acquisition unit 230 creates a multi-view frame set that combines synchronous frames and asynchronous frames. Below, we will explain how to determine whether a frame is synchronous or asynchronous using the shooting times between two cameras.

[0121] The shooting time of the frame selected from camera 100-1 is T1, the shooting time of the frame selected from camera 100-2 is T2, the exposure time of camera 100-1 is TE1, and the exposure time of camera 100-2 is TE2. Here, the shooting times T1 and T2 refer to the times when exposure starts in the examples of Figures 4A and 4B, that is, the times of the rise of the rectangular signal.

[0122] In this case, the exposure end time of the camera 100-1 is T1+TE1. At this time, if formula (13) or formula (14) is satisfied, the two cameras 100-1 and 100-2 are photographing the subject 1001 at the same time, and the two frames are determined to be synchronized frames.

[0123] T1≦T2≦T1+TE1 Formula (13) T1≦T2+TE2≦T1+TE1 Formula (14)

[0124] When cameras 100-1 to 100-n capture multi-viewpoint videos based on the number of viewpoints and viewpoint positions determined by control unit 300, acquisition unit 230 may create a multi-viewpoint frame set of frames equal in number to the number of input multi-viewpoint videos. On the other hand, when cameras 100-1 to 100-n capture multi-viewpoint videos regardless of the number of viewpoints or viewpoint positions determined by control unit 300, acquisition unit 230 may select videos corresponding to the number of viewpoints or viewpoint positions determined by control unit 300, and create a multi-viewpoint frame set.

[0125] That is, N images captured at different times are regarded as one set in the three-dimensional displacement measuring device 2000. For each of the multiple sets, the three-dimensional displacement measuring device 2000 measures the three-dimensional position of the subject 1001 at the time of the set using the N images and camera parameters included in the set.

[0126] 12 is a block diagram of a three-dimensional image processing unit in the embodiment. Three-dimensional image processing unit 240 includes a camera calibration unit 410 and a three-dimensional processing unit 420. Note that three-dimensional processing unit 420 is a specific example of a displacement measuring unit.

[0127] The camera calibration unit 410 performs camera calibration. Specifically, the camera calibration unit 410 calculates the internal parameters, external parameters, and lens distortion coefficients of each of the multiple cameras 100-1 to 100-n as camera parameters. The internal parameters indicate characteristics of the optical system such as the focal length, aberration, and image center of the camera. The external parameters indicate the position and orientation of the camera in three-dimensional space.

[0128] The camera calibration unit 410 may calculate the internal parameters, external parameters, and lens distortion coefficients separately using M calibration images, which are M frames obtained by multiple cameras 100-1 to 100-n capturing the black and white intersections of a checkerboard, or may calculate the internal parameters, external parameters, and lens distortion coefficients collectively using corresponding points between M frames as in Structure from Motion, and perform overall optimization. In the latter case, the M frames do not need to be images of a checkerboard.

[0129] The three-dimensional processing unit 420 performs three-dimensional processing such as three-dimensional modeling of the object 1001 and three-dimensional displacement measurement of the object 1001 using the multi-view frame set and camera parameters. Specifically, the three-dimensional processing unit 420 generates a model of the object 1001 from three-dimensional points as three-dimensional modeling using the multi-view frame set and the result of camera calibration. Instead of three-dimensional modeling, the three-dimensional processing unit 420 may measure the three-dimensional displacement of the object 1001 using the multi-view frame set and the result of camera calibration as three-dimensional displacement measurement. When measuring the three-dimensional displacement, the three-dimensional processing unit 420 may use data of three-dimensional points of the three-dimensional model of the object 1001 generated by the three-dimensional modeling. Here, the three-dimensional points used in the three-dimensional displacement measurement may be any points of the object 1001.

[0130] The three-dimensional image processing unit 240, and the camera calibration unit 410 and the three-dimensional processing unit 420 constituting the three-dimensional image processing unit 240, are realized by a processor and a memory. These may also be realized by a dedicated logic circuit.

[0131] Fig. 13 is a flowchart showing the operation of the three-dimensional image processing unit in the embodiment. In the process shown in Fig. 13, a multi-view frame set with the number of viewpoints determined by the control unit 300 is used.

[0132] First, the camera calibration unit 410 calculates the camera parameters of the cameras present at each viewpoint where shooting was performed, using the multi-view frame set corresponding to the number of viewpoints M and viewpoint positions determined by the control unit 300 (step S4100). Here, the control unit 300 may calculate the camera parameters only for at least N cameras. Note that step S4100 is an example of a camera calibration step.

[0133] Next, the three-dimensional processing unit 420 generates a three-dimensional model of the object 1001 or measures the three-dimensional displacement of the object 1001 using the multi-view frame set corresponding to the number of viewpoints N and the viewpoint positions determined by the control unit 300 and the camera parameters calculated by the camera calibration unit 410 (step S4200). Here, the three-dimensional processing unit 420 is a specific example of a displacement measurement unit. Note that step S4200 is an example of a displacement measurement step.

[0134] In this embodiment, the number of viewpoints used as a reference is the number of viewpoints N used in three-dimensional modeling or three-dimensional displacement measurement. For example, the number of viewpoints N is equal to the number of cameras that are always installed in the imaging space 1000.

[0135] Furthermore, the accuracy of the camera parameters calculated in step S4100 significantly affects the accuracy of generating a three-dimensional model of the subject 1001 or measuring the three-dimensional displacement of the subject 1001, which is performed in step S4200. Therefore, in order to improve the accuracy of the camera parameters, in step S4100, a number of viewpoints M, which is greater than the number of viewpoints N in step S4200, is used. In this case, the number of cameras does not necessarily have to be M. It is also possible to perform camera calibration by using still areas in the captured images, which are captured using less than M moving cameras, and capturing images from multiple viewpoints. A still area is an area in an image in which a still object is captured, excluding a moving area in which a moving object is captured.

[0136] As described above, the three-dimensional displacement measuring device 2000 may perform three-dimensional modeling and may further perform free viewpoint video generation in addition to three-dimensional displacement measurement in the three-dimensional image processing unit 240.

[0137] The free viewpoint video generation performed by the three-dimensional displacement measuring device 2000 according to the embodiment will be described below.

[0138] The three-dimensional displacement measuring device 2000 can reconstruct a time-series three-dimensional model with coordinate axes aligned between times. Specifically, the three-dimensional displacement measuring device 2000 first performs three-dimensional reconstruction independently for each time to obtain a three-dimensional model for each time. Next, the three-dimensional displacement measuring device 2000 detects a stationary camera and a stationary object (stationary three-dimensional point), and uses the detected stationary camera and stationary object to align the coordinates of the three-dimensional model between times to generate a time-series three-dimensional model with coordinate axes aligned.

[0139] Next, a description will be given of the operation of the three-dimensional displacement measurement apparatus 2000 when generating a free viewpoint video. Fig. 14 is a flowchart showing the operation of the three-dimensional displacement measurement apparatus 2000 according to this embodiment when generating a free viewpoint video.

[0140] First, the receiving unit 210 receives the multi-view video captured by the multiple cameras 100-1 to 100-n (S101). The storage unit 220 stores the received multi-view video (S102).

[0141] Next, the acquisition unit 230 selects frames from the multi-view video and outputs them as a multi-view frame set to the 3D image processing unit 240 (S103). Note that the multi-view frame set selected here may have the same configuration as the multi-view frame set selected in step S2500 of FIG. 6.

[0142] Next, the 3D image processing unit 240 generates a free viewpoint video by performing a camera calibration process, a 3D modeling process, and a free viewpoint video synthesis process using the multi-view frame set (S104).

[0143] Furthermore, the processes of steps S103 and S104 are repeated for each multi-view frame set.

[0144] Finally, the transmitting unit 250 transmits at least one of the camera parameters, the 3D model of the subject, and the free viewpoint video to an external device (S105).

[0145] Next, the free viewpoint video generation by the 3D image processing unit 240 will be described in detail.

[0146] 7 to 10, control unit 300 determines the optimum number of viewpoints for each process in camera calibration unit 410 and three-dimensional processing unit 420. The number of viewpoints determined here refers to the number of mutually different viewpoints.

[0147] The accuracy of the camera parameters calculated in the camera calibration process has a large effect on the accuracy in the three-dimensional modeling process and the free viewpoint video synthesis process. Therefore, in order to prevent the accuracy in the three-dimensional modeling process and the free viewpoint video synthesis process from decreasing, the control unit 300 determines the number of viewpoints M of the multi-view frame set used in the camera calibration process to be greater than the number of viewpoints N in the three-dimensional modeling process so as to improve the accuracy of the camera parameters. That is, the control unit 300 causes the camera calibration unit 410 to execute the camera calibration process using M frames obtained by adding k (k is an integer equal to or greater than a) frames captured by a non-fixed cameras to N frames captured by n cameras 100-1 to 100-n. Note that the a non-fixed cameras do not necessarily have to be k, and may be k frames (images) obtained as a result of capturing images at k viewpoints by moving the a non-fixed cameras.

[0148] In addition, in the free viewpoint video synthesis process, the calculation of the corresponding positions between the images obtained by the real cameras and the images of the virtual viewpoint requires a large processing time because the processing load increases as the number of real cameras increases. On the other hand, the texture information obtained from the images obtained by the cameras that are located close to each other among the n cameras 100-1 to 100-n is similar to each other. Therefore, whether all the images are used in the free viewpoint video synthesis process or one of the images is used, the accuracy of the result obtained by the free viewpoint video synthesis process does not change much. Therefore, the control unit 300 determines the number of viewpoints l, which is smaller than the number of viewpoints N in the three-dimensional modeling process, as the number of viewpoints of the multi-view frame set to be used in the free viewpoint video synthesis process.

[0149] FIG. 15 is a flowchart showing the operation of the 3D image processing unit 240 when generating a free viewpoint video.

[0150] First, similar to step S4100, camera calibration unit 410 calculates camera parameters of multiple cameras 100-1 to 100-n using M calibration images captured at M different viewpoints by n cameras 100-1 to 100-n arranged at different positions (S310). Note that the M viewpoints here are based on the number of viewpoints determined by control unit 300.

[0151] The camera calibration unit 410 performs camera calibration processing using M calibration images obtained by n cameras 100-1 to 100-n including fixed and non-fixed cameras. In camera calibration processing, the greater the number of cameras, the closer the distance between the cameras becomes, and the closer the fields of view of the multiple cameras that are close to each other become, making it easier to match multiple images obtained from multiple cameras that are close to each other. Therefore, when performing camera calibration, the camera calibration unit 410 increases the number of viewpoints by using non-fixed cameras in addition to the fixed cameras that are always installed in the shooting space 1000.

[0152] The non-fixed camera may be at least one moving camera, and when a moving camera is used as the non-fixed camera, images captured at different times are included. That is, the M calibration images used in the camera calibration process include images captured at different times. In other words, the M viewpoints of the multi-view frame set constituted by the M calibration images include frames obtained by asynchronous shooting. For this reason, the camera calibration unit 410 performs the camera calibration process using corresponding points between images of feature points obtained from a still area, which is an area in which a still object is captured, among the M calibration images. Therefore, the camera calibration unit 410 calculates camera parameters corresponding to the still area. The still area is an area excluding a moving area in which a moving object is captured among the M calibration images. The moving area captured in the frame is detected, for example, by calculating the difference from a past frame, calculating the difference from a background image, or automatically detecting the area of ​​a moving object by machine learning.

[0153] It should be noted that the camera calibration unit 410 does not need to always perform the camera calibration process in step S310 in the free viewpoint video generation process in the 3D image processing unit 240, and may perform it once every predetermined number of times.

[0154] Next, the three-dimensional processing unit 420 reconstructs a three-dimensional model using the N three-dimensional processing images captured by the n cameras 100-1 to 100-n and the camera parameters obtained in the camera calibration process (S320). That is, the three-dimensional processing unit 420 reconstructs a three-dimensional model using the N three-dimensional processing images captured at the N viewpoints based on the number of viewpoints N determined by the control unit 300. As a result, the three-dimensional processing unit 420 reconstructs the subjects in the N three-dimensional processing images as three-dimensional points. The N three-dimensional processing images used in the three-dimensional modeling process are images captured by the n cameras 100-1 to 100-n at any timing. That is, the multi-view frame set of N viewpoints composed of the N three-dimensional processing images is a multi-view frame set obtained by synchronous shooting. For this reason, the three-dimensional processing unit 420 performs the three-dimensional modeling process using the regions including stationary objects and moving objects (that is, all regions) among the N three-dimensional processing images. In addition, the three-dimensional processing unit 420 may use the measurement results of the position of the subject in three-dimensional space using a laser scan, or may calculate the position of the subject in three-dimensional space using corresponding points in multiple stereo images, as in a multi-view stereo method.

[0155] Next, the three-dimensional processing unit 420 synthesizes a free viewpoint video using the l third images captured by each of the n cameras 100-1 to 100-n, the camera parameters calculated in the camera calibration process, and the three-dimensional model reconstructed in the three-dimensional modeling process (S330). That is, the three-dimensional processing unit 420 synthesizes a free viewpoint video using the l third images captured at the l viewpoints based on the number of viewpoints l determined by the control unit 300. Specifically, the three-dimensional processing unit 420 synthesizes a free viewpoint video by calculating texture information of the virtual viewpoint using texture information of the real camera based on the corresponding positions between the image of the real camera and the image of the virtual viewpoint obtained by the camera parameters and the three-dimensional model.

[0156] (Effects, etc.) According to the three-dimensional displacement measurement method of this embodiment, a number of viewpoints M, which is greater than the number of viewpoints N in the three-dimensional displacement measurement process, is used for the camera calibration process so as to improve the accuracy of the camera parameters, thereby improving the accuracy of the three-dimensional displacement measurement process.

[0157] Furthermore, according to the three-dimensional displacement measuring method of the present embodiment, the displacement of the three-dimensional points over time is calculated based on the highly accurate camera parameters, the first corresponding points, and the second corresponding points by using an image matching technique and the principle of triangulation, so that the displacement of the subject in the three-dimensional space can be calculated effectively.

[0158] Moreover, according to the three-dimensional displacement measuring method of the present embodiment, the number of camera viewpoints N can be determined so as to satisfy the accuracy of the measurement performed in the displacement measuring step. Therefore, the three-dimensional displacement measuring method can accurately obtain the displacement of the subject in three-dimensional space.

[0159] According to the three-dimensional displacement measuring device 2000 of this embodiment, in consideration of the fact that the accuracy of the camera parameters calculated in the camera calibration process has a large effect on the accuracy in the three-dimensional modeling process and the free viewpoint video synthesis process, the number of viewpoints M of the multi-view frame set used in the camera calibration process is determined to be greater than the number of viewpoints N in the three-dimensional modeling process so as to improve the accuracy of the camera parameters. Therefore, the accuracy in the three-dimensional modeling process and the free viewpoint video synthesis process can be improved.

[0160] Furthermore, according to the three-dimensional displacement measuring device 2000 of this embodiment, the number of viewpoints l, which is smaller than the number of viewpoints N in the three-dimensional modeling process, is determined as the number of viewpoints of the multi-viewpoint frame set used in the free viewpoint video synthesis process, thereby reducing the processing load required to generate a free viewpoint video.

[0161] (Modification) 16 is a block diagram of a modified example of the three-dimensional image processing unit. In this modified example of the present embodiment, three-dimensional image processing unit 240A is made up of camera calibration unit 510 and three-dimensional processing unit 520.

[0162] The camera calibration unit 510 has the same function, configuration, etc. as the camera calibration unit 410, but differs in the following points. The camera calibration unit 510 accepts the output of the camera calibration unit 510 as a fed-back input and performs processing again based on the input. The three-dimensional processing unit 520 has the same function and configuration as the three-dimensional processing unit 420. The three-dimensional processing unit 520 performs processing based on the data finally output from the camera calibration unit 510. Note that the final output from the camera calibration unit is, for example, the output excluding the initial output from the camera calibration unit.

[0163] FIG. 17 is a flowchart showing the operation of the modified example of the three-dimensional image processing unit.

[0164] First, the camera calibration unit 510 calculates the camera parameters of the cameras present at each viewpoint where shooting was performed, using a multi-view frame set corresponding to the number of viewpoints and viewpoint positions determined by the control unit 300 (step S5100).

[0165] Next, the camera calibration unit 510 calculates the camera parameters of the cameras located at each viewpoint where shooting was performed using a multi-view frame set corresponding to the number of viewpoints and viewpoint positions determined by the control unit 300, using the values ​​resulting from the camera calibration performed in step S5100 as initial values ​​(step S5200).

[0166] Then, the three-dimensional processing unit 520 performs three-dimensional modeling of the subject 1001 or three-dimensional displacement measurement of the subject using a multi-view frame set corresponding to the number of viewpoints and viewpoint positions determined by the control unit 300 and the camera parameters calculated by the camera calibration unit 510 (step S5300).

[0167] In the camera calibration process, the more cameras used for camera calibration, the closer the distance between the cameras. When the distance between the cameras is short, the areas of the shooting space 1000 that are in the field of view of the cameras are close to each other, so that it is easy to associate the images captured by each camera. Therefore, when performing camera calibration, in addition to the fixed cameras that are always installed in the shooting space 1000, non-fixed cameras are used to increase the number of viewpoints used for camera calibration beyond the number of viewpoints realized by the fixed cameras. Here, the non-fixed camera may be at least one mobile camera. However, when a mobile camera is used as the non-fixed camera, the shooting times of the images captured at different viewpoints are different, so that camera parameters corresponding only to the still areas of the captured images are calculated.

[0168] At this time, the calculated camera parameters do not necessarily correspond to the moving area in the image captured by the fixed camera. Here, the moving area is an area in the image in which a moving object is captured. In addition, in a method such as Structure from Motion, the camera parameters are optimized overall. Therefore, when this method is used, the camera parameters are not necessarily optimized if only the fixed camera is considered.

[0169] Therefore, in this modification, two-stage camera calibration is performed over two steps, step S5100 and step S5200.

[0170] First, in step S5100, the camera calibration unit 510 performs rough camera calibration using a multi-view frame set using B viewpoints, which includes images captured by at least one camera other than the A cameras that are permanently installed in the shooting space 1000.

[0171] Next, in step S5200, the camera calibration unit 510 optimizes the camera parameters calculated by the camera calibration unit 510 in the environment of the A cameras, using only images captured by the A cameras that are constantly installed in the imaging space 1000. Here, optimization may refer to performing a process of reprojecting three-dimensional points obtained secondarily during the calculation of the camera parameters onto the captured image, and minimizing the evaluation value, using the error (reprojection error) between the reprojected points and the feature points detected on the image as the evaluation value.

[0172] The number of viewpoints used in the second-stage camera calibration may be greater than the number A of cameras that are constantly installed in the imaging space 1000. The number of viewpoints used in the second-stage camera calibration may be greater than the number of viewpoints used in three-dimensional image processing (e.g., three-dimensional displacement measurement). However, the number of viewpoints used in the second-stage camera calibration may be equal to or less than the number of viewpoints used in three-dimensional image processing (e.g., three-dimensional displacement measurement).

[0173] In the above three-dimensional displacement measuring device, the n cameras 100-1 to 100-n may include a stereo camera having two cameras. The stereo camera may have two cameras that capture images in approximately the same direction, that is, a first camera and a second camera, and the distance between the two cameras may be a predetermined distance or less. In this way, when the n cameras 100-1 to 100-n are configured as stereo cameras, the stereo camera is configured with n / 2 first cameras and n / 2 second cameras. The two cameras of the stereo camera may be integrated or may be separate.

[0174] Furthermore, the first and second cameras constituting the stereo camera may capture images at different sensitivities. The first camera is a camera that captures images at a first sensitivity. The second camera is a camera that captures images at a second sensitivity different from the first sensitivity. The first and second cameras are cameras with different color sensitivities.

[0175] The three-dimensional processing unit reconstructs a three-dimensional model using n images for three-dimensional processing obtained by capturing images using all of the n cameras 100-1 to 100-n. Since the three-dimensional processing unit uses luminance information in the three-dimensional modeling process, it is possible to calculate the three-dimensional model with high accuracy using all of the n cameras regardless of differences in color sensitivity.

[0176] The three-dimensional processing unit synthesizes a free viewpoint video using n / 2 third images, which are multiple images obtained by capturing images using n / 2 first cameras or n / 2 second cameras, the camera parameters calculated by the camera calibration unit, and the three-dimensional model reconstructed by the three-dimensional processing unit. In the free viewpoint video generation process, the three-dimensional processing unit has little effect on accuracy even if n / 2 images captured by either one of the n / 2 first cameras or the n / 2 second cameras are used. Therefore, the three-dimensional processing unit performs free viewpoint synthesis using n / 2 images captured by either the first camera or the second camera according to the situation of the shooting space 1000. For example, the n / 2 first cameras are cameras with high color sensitivity to red, and the n / 2 second cameras are cameras with high color sensitivity to blue. In this case, the three-dimensional processing unit switches the images to be used so that if the subject is a reddish color, an image captured by the first camera with high color sensitivity to red is used, and if the subject is a blued color, an image captured by the second camera with high color sensitivity to blue is used to execute the free viewpoint video synthesis process.

[0177] According to the 3D displacement measurement device, free viewpoint video synthesis is performed using one of two types of images obtained from two types of cameras with different sensitivities depending on the situation of the shooting space. Therefore, it is possible to generate a free viewpoint video with high accuracy.

[0178] The first and second cameras are not limited to cameras with different color sensitivities, and may be cameras with different luminance sensitivities. In this case, the three-dimensional processing unit can switch between the cameras depending on the situation, such as daytime and nighttime, sunny and cloudy, etc.

[0179] Although a stereo camera is used in the above embodiment, it is not necessary to use a stereo camera. Therefore, the n cameras are not limited to being composed of n / 2 first cameras and n / 2 second cameras, and may be composed of i first cameras and j second cameras.

[0180] Although the multiple cameras 100-1 to 100-n are described as being composed of fixed cameras and non-fixed cameras, this is not limiting, and all of the multiple cameras may be composed of fixed cameras. Also, although the n images used in the three-dimensional modeling are described as images captured by fixed cameras, they may include images captured by non-fixed cameras.

[0181] FIG. 18 is a diagram showing an overview of three-dimensional space recognition. As described above, the same space can be photographed from multiple viewpoints using a calibrated camera (for example, a fixed camera), thereby enabling the space to be three-dimensionally reconstructed (three-dimensional space reconstruction). Using this three-dimensionally reconstructed data, three-dimensional displacement measurement and the like can be performed. This makes it possible to realize a next-generation wide-area monitoring system and a three-dimensional displacement measurement system. Specifically, at an intersection on a roadway or the like, a passing vehicle 10 or the like can be photographed using a fixed camera 100-a or a non-fixed camera 100-b, and the three-dimensional displacement of the vehicle 10 or the like can be measured.

[0182] According to the above-mentioned three-dimensional displacement measuring device, three-dimensional displacement measurement of the subject is performed and further, three-dimensional modeling of the subject may be performed, however, without being limited to this, three-dimensional modeling of the subject may be performed without performing three-dimensional displacement measurement of the subject. [Industrial Applicability]

[0183] The present disclosure is applicable to three-dimensional displacement measuring methods and three-dimensional displacement measuring devices, and is applicable to, for example, infrastructure inspection systems, industrial machinery monitoring systems, three-dimensional spatial recognition systems, and next-generation monitoring systems. [Explanation of symbols]

[0184] 3. 3D Reconstruction System 10 Vehicles 100-1~100-n Camera 100-a Fixed Camera 100-b Non-fixed camera 200-1~200-Q Multi-view frame set 210 Receiving unit 220 Storage section 230 Acquisition Department 240, 240A 3D image processing unit 250 Transmitter 300 Control section 310 Measurement Decision Unit 320 Camera Calibration Decision Unit 410, 510 Camera Calibration Unit 420, 520 3D processing section 1000 Shooting Space 1001 Subject 2000 3D Displacement Measuring Device

Claims

1. a camera calibration step of calculating camera parameters of the N cameras based on M calibration images captured at M (M is an integer greater than N) different viewpoints by a plurality of cameras including N cameras (N is an integer greater than 1); (1) N images are generated by capturing an image of a subject by each of the N cameras at different times, and (2) a displacement measurement step is included in which a displacement of the subject over time in a three-dimensional space is measured based on the camera parameters, The N is determined based on the camera parameters and the accuracy of the measurement of the displacement. Three-dimensional displacement measurement method.

2. The displacement measuring step includes: a corresponding point calculation step of calculating a first corresponding point of the subject included in a first image included in the N images and a second corresponding point included in a second image included in the N images and corresponding to the first corresponding point; a three-dimensional point calculation step of calculating a three-dimensional point of the subject in the three-dimensional space based on the camera parameters, the first corresponding point, and the second corresponding point; A displacement calculation step of calculating a displacement of the three-dimensional point over time from the plurality of three-dimensional points corresponding to the different times. The three-dimensional displacement measuring method according to claim 1 .

3. The N cameras include a first camera and a second camera adjacent to each other, the M viewpoints include a first midpoint between a first position where the first camera is disposed and a second position where the second camera is disposed; When a midpoint between the first position and the first midpoint is defined as a second midpoint, the number of viewpoints included in the M number of viewpoints and provided between the second midpoint and the first position is greater than the number of viewpoints included in the M number of viewpoints and provided between the first midpoint and the second midpoint. The three-dimensional displacement measuring method according to claim 1 or 2.

4. In the camera calibration step, a viewpoint is not provided between the first midpoint and the second midpoint. The three-dimensional displacement measuring method according to claim 3 .

5. a camera calibration unit that calculates camera parameters of the N cameras based on M calibration images captured at M (M is an integer greater than N) different viewpoints by a plurality of cameras including N cameras (N is an integer greater than 1); (1) N images generated by capturing an image of a subject by each of the N cameras at different times; and (2) a displacement measurement unit configured to measure a displacement of the subject over time in a three-dimensional space based on the camera parameters, The N is determined based on the camera parameters and the accuracy of the measurement of the displacement. Three-dimensional displacement measurement device.

Images