Information processing device, information processing system, calibration method and program

Abstract

To provide an information processing device.SOLUTION: An information processing device 110 is a device for calibrating three-dimensional measurement using a plurality of imaging means 160. The information processing device 110 comprises object determination means 272 for, on the basis of information obtained from images captured by the imaging means 160, determining imaging means to be a re-correction object among the plurality of imaging means 160. The information processing device 110 comprises calculation means 274 for, on the basis of three-dimensional position information obtained by using the combination of imaging means except the imaging means to be a re-correction object and of corresponding positional information in the images captured by the imaging means to be a re-correction object, calculating a parameter of three-dimensional measurement using the imaging means to be a re-correction object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, a calibration method, and a program. [Background technology]

[0002] Three-dimensional measurements of moving objects have been performed in factories for parts assembly and high-precision alignment adjustment. For three-dimensional measurements, imaging systems including stereo cameras are known.

[0003] In the above-described three-dimensional measurement, if the camera position is displaced due to a sudden collision or a change over time, it is necessary to perform calibration correction in relation to the camera that has been displaced.

[0004] In relation to calibration correction, for example, a conventional technique disclosed in Japanese Patent Laid-Open No. 2005-233639 (Patent Document 1) is known. Patent Document 1 discloses a three-dimensional position calculation process for calculating the three-dimensional position of a person from images captured by two stereo cameras, and a projected person area calculation process for correlating a projected image of the person projected onto a two-dimensional plane based on the three-dimensional positions, for the purpose of automatically calibrating the stereo cameras. In the conventional technique disclosed in Patent Document 1, the positions of the projected person areas are compared, and calibration correction between the stereo cameras is performed based on the relative positional relationship.

[0005] Patent Document 1 discloses a method for calibrating stereo cameras when the stereo cameras move intentionally or accidentally while the stereo camera system is in operation. However, this method assumes that the left and right cameras in the stereo camera are accurately calibrated, and does not address cases where an individual camera is misaligned. Furthermore, in the conventional technology of Patent Document 1, the calibration correction unit determines that an error has occurred in the calibration of the stereo cameras when a positional deviation occurs in the projected person area, and performs calibration correction. In other words, it is not possible to determine which stereo camera has moved. Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been made in consideration of the above points, and aims to provide an information processing device that is capable of determining which imaging means require re-correction in three-dimensional measurement using multiple imaging means, and correcting parameters in three-dimensional measurement using those imaging means. [Means for solving the problem]

[0007] In order to solve the above problems, the present disclosure provides an information processing device having the following characteristics for calibrating three-dimensional measurement using multiple imaging means: The information processing device includes: a target determination means that detects an imaging means among the multiple imaging means that is to be recorrected based on information obtained from images captured by the imaging means, and a calculation means that calculates parameters for three-dimensional measurement using the imaging means to be recorrected based on three-dimensional position information acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in the images captured by the imaging means to be recorrected. [Effects of the Invention]

[0008] With the above configuration, in three-dimensional measurement using a plurality of imaging means, it is possible to determine which imaging means require re-correction and correct the parameters in the stereo measurement including that imaging means. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating the overall configuration of a three-dimensional measurement system according to one or more embodiments. [Figure 2] FIG. 1 is a hardware configuration diagram of a personal computer that can be used as a three-dimensional measurement apparatus according to one or more embodiments. [Figure 3] FIG. 1 is a functional block diagram of a three-dimensional measuring apparatus according to one or more embodiments. [Figure 4] 1 illustrates a calibration correction of extrinsic parameters of a camera to be recalibrated according to one or more embodiments. [Figure 5] FIG. 2 is a diagram illustrating a computational method for implementing calibration correction of external parameters, in accordance with one or more embodiments. [Figure 6] 1 is a flowchart showing a three-dimensional measurement process executed by a three-dimensional measurement apparatus 110 according to one or more embodiments. [Figure 7] 10 is a flowchart showing control of determining a camera to be recalibrated and performing calibration correction, executed by the three-dimensional measuring apparatus 110 according to one or more embodiments. [Figure 8] 1A to 1C are diagrams illustrating a method for determining a misaligned camera, performed by the three-dimensional measurement apparatus 110 according to one or more embodiments. [Figure 9] 10 is a flowchart showing control of determining a camera to be recalibrated and performing calibration correction in addition to three-dimensional measurement, which is executed by a three-dimensional measuring apparatus 110 according to another embodiment. [Figure 10] 10A and 10B are diagrams illustrating a method for determining a camera that is misaligned using the velocity of the center of gravity, which is executed by the three-dimensional measuring apparatus 110 according to another embodiment. [Figure 11] 10A and 10B are diagrams illustrating a method for determining a camera that is misaligned using center-of-gravity acceleration, which is executed by the three-dimensional measuring apparatus 110 according to still another embodiment. [Figure 12]10 is a flowchart showing control of determining a camera to be recalibrated and performing calibration correction in addition to three-dimensional measurement, executed by a three-dimensional measuring apparatus 110 according to yet another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described, but the embodiment of the present invention is not limited to the embodiment described below. Note that the embodiment described below will be described with reference to a three-dimensional measurement system 100 including a plurality of high-speed vision cameras 160 and a three-dimensional measurement device 110 that performs three-dimensional measurement of an object, and the three-dimensional measurement device 110, as an example of an information processing system and an information processing device, respectively.

[0011] Fig. 1 shows the overall configuration of a three-dimensional measurement system 100 according to one or more embodiments. As shown in Fig. 1, the three-dimensional measurement system 100 includes a three-dimensional measurement device 110, a camera synchronizer 150, and a plurality of high-speed vision cameras 160a to 160z connected to the three-dimensional measurement device 110 and the camera synchronizer 150, respectively.

[0012] The multiple high-speed vision cameras 160a to 160z are arranged at different positions so as to face a predetermined direction, and are configured to capture images of the object 102 as a subject from different viewpoints. Therefore, parallax occurs between the multiple images captured by the multiple high-speed vision cameras 160a to 160z. Preferably, the field of view of each of the multiple high-speed vision cameras 160a to 160z is set so as to cover the range in which the object 102 can move. Each high-speed vision camera 160 captures an image of the object 102 within its field of view based on a timing signal input from the camera synchronizer 150, and inputs the captured image to the three-dimensional measuring device 110. The high-speed vision camera 160 constitutes the imaging means in this embodiment.

[0013] The high-speed vision camera 160 is an imaging device that captures images faster than a normal camera, which captures images at about 30 fps to 60 fps, and typically, one that achieves a frame rate of about 1000 fps is preferably used. Due to this high frame rate, the high-speed vision camera 160 can also capture fast-moving moving objects. The shutters of each high-speed vision camera 160 are synchronized by an external camera synchronizer 150, so that it is possible to capture the same frame at the same time. Note that, although the embodiment described below will be described assuming that a high-speed vision camera 160 is used, this is not particularly limited, and a normal camera may also be used.

[0014] The camera synchronizer 150 externally controls the shutter synchronization of each high-speed vision camera 160. By connecting the external camera synchronizer 150 to multiple high-speed vision cameras 160 and setting them as master-slave by program, shutter synchronization becomes possible by inputting and outputting external triggers.

[0015] The three-dimensional measuring device 110 performs three-dimensional measurement of the object 102, including the depth, based on multiple images obtained by capturing the object 102 with multiple high-speed vision cameras 160a to 160z. For convenience of explanation, the object 102 will be described as a marker made of retroreflective material, but the marker may be attached to another object (for example, a component in a factory assembly), and the attached marker may be detected. Furthermore, the object 102 to be recognized is not limited to a marker made of retroreflective material, and any object of any shape may be used as the object as long as it can be recognized from captured images.

[0016] The three-dimensional measuring device 110 may be configured to include three or more high-speed vision cameras 160. When performing three-dimensional measurement, if the object 102 is hidden by an obstacle (hereinafter sometimes referred to as occlusion), the three-dimensional measuring device 110 is configured to continue the three-dimensional measurement using a high-speed vision camera 160 other than the one where the occlusion occurred. How the three-dimensional measurement is continued using a high-speed vision camera 160 other than the one where the occlusion occurred will not be explained here. In the embodiment to be explained, the explanation will continue assuming that the three-dimensional measurement is performed using any pair of high-speed vision cameras 160.

[0017] Before describing the details of the functions of the three-dimensional measurement system 100, the hardware configuration of the three-dimensional measurement device 110 will be described below.

[0018] 2 shows the hardware configuration of a personal computer that can be used as the three-dimensional measuring apparatus 110 according to this embodiment. Here, the hardware configuration of a personal computer 500 will be described.

[0019] As shown in FIG. 2, the personal computer 500 is constructed using a general-purpose computer, and as shown in FIG. 2, includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an HDD (Hard Disk Drive) 504, an HDD controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0020] Of these, the CPU 501 controls the overall operation of the personal computer 500. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HDD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HDD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 2.

[0021] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW medium 513, which is an example of a removable recording medium. Note that this is not limited to DVD-RW, and may be DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0022] The functional configuration of the three-dimensional measuring device 110 will be described below with reference to Fig. 3. Fig. 3 shows functional blocks of the three-dimensional measuring device 110 according to this embodiment.

[0023] As shown in Fig. 3, the three-dimensional measurement device 110 includes an image input unit 210 that receives input of images from different viewpoints from multiple high-speed vision cameras (referred to as HSV in Fig. 3) 160. The three-dimensional measurement device 110 also includes an object search unit 220 that searches for a predetermined object 102 in the input images, and a tracking unit 230 that tracks the object 102 in the images. The three-dimensional measurement device 110 further includes a three-dimensional measurement unit 250 that performs three-dimensional measurement based on the multiple images from different viewpoints that have been input to the image input unit 210, a calibration unit 260 that performs calibration for the three-dimensional measurement, and a calibration correction unit 270.

[0024] The image input unit 210 receives input of images captured by each of the multiple high-speed vision cameras 160, and sequentially passes the acquired images captured by the multiple high-speed vision cameras 160 to the target search unit 220. Time-series data consisting of multiple frames of images captured by each high-speed vision camera 160 is input in the same number as the number of high-speed vision cameras 160.

[0025] The object search unit 220 searches each captured image for a predetermined object 102. If the object 102 is a high-brightness area that can be distinguished from the background and other objects, it can be binarized, for example, and the contour of the object can be searched for in the binarized image.

[0026] The tracking unit 230 tracks the position of the object 102 in the captured image. More specifically, the tracking unit 230 includes a center of gravity calculation unit 232 and an area setting unit 234.

[0027] The centroid calculation unit 232 calculates the centroid position of the object 102 as a feature point in each frame based on image information of a predetermined partial region referred to as a region of interest (ROI). The region of interest can be set as a rectangular region of a size according to the outline of the object 102 found by the object search unit 220. By performing image processing only within the region of interest, it is possible to significantly reduce the amount of calculation. The centroid position is measured in the coordinate system of the image captured by the high-speed vision camera 160.

[0028] The region setting unit 234 sets a region of interest (ROI) centered on the centroid position calculated by the centroid calculation unit 232. The region of interest is set for the current frame based on the centroid position calculated for a past frame (typically the previous frame), and is updated as needed.

[0029] The calibration unit 260 performs calibration processing for each high-speed vision camera 160. More specifically, the calibration unit 260 includes a correction transformation matrix calculation unit 262 and a fundamental matrix calculation unit 264. The correction transformation matrix calculation unit 262 calculates a correction transformation matrix. The fundamental matrix calculation unit 264 calculates a fundamental matrix. Once the correction transformation matrix and fundamental matrix are calculated, a projection matrix Pn (n is an index that identifies the camera) required for three-dimensional measurement is calculated.

[0030] The three-dimensional measurement unit 250 uses the above-mentioned parameters to perform three-dimensional measurement using the principle of triangulation. In the embodiment described below, the three-dimensional measurement can be performed using a stereo method. The three-dimensional measurement using the stereo method is performed based on the parallax of feature points obtained from two images captured by the high-speed vision camera 160. Note that with the stereo method, it is only necessary to compare a certain portion of the feature points, and the calculation method is simple, so the processing time can be shortened.

[0031] In the stereo method, the distance at a pixel position is calculated by solving the simultaneous equations (1) and (2) below. ~is the homogeneous form of the image coordinates (u,v) (center of gravity position), P is the projection matrix, and X ~ w is the world coordinate (X w , Y w , Z w ) is a homogeneous form of the following. In the following formulas (1) and (2), the variables accented with a tilde (~) are referred to as m ~ or X ~ w The symbol "'" identifies one camera selected as a stereo pair from the other camera. Since three-dimensional measurement using the stereo method can be performed appropriately using existing technology, no further detailed explanation will be given.

[0032]

number

[0033] The calibration unit 260 calibrates the parameters necessary for three-dimensional measurement, making three-dimensional measurement possible thereafter. However, after the initial calibration, there is a possibility that one of the high-speed vision cameras 160 in the three-dimensional measurement system 100 may become misaligned due to an unintended sudden collision with an object, changes over time, or the like. In such cases, it is necessary to recalibrate the parameters related to the misaligned camera, more specifically, to recalculate the projection matrix.

[0034] The calibration correction unit 270 according to this embodiment deals with positional deviations that occur after the above-mentioned initial calibration. The calibration correction unit 270 attempts to detect positional deviations that have occurred in the high-speed vision camera 160 periodically or irregularly in response to explicit instructions. The calibration correction unit 270 corrects parameters used in three-dimensional measurement for the high-speed vision camera 160 for which a positional deviation has been detected. More specifically, the calibration correction unit 270 includes a re-correction target camera determination unit 272 and a parameter calculation unit 274.

[0035] The recalibration target camera determination unit 272 determines which of the multiple high-speed vision cameras 160 needs recalibration, that is, which high-speed vision camera 160 has a positional deviation (hereinafter, may be referred to as the camera to be recalibrated), based on information obtained from the captured image of the high-speed vision camera 160. Here, the information obtained from the captured image and used to determine the camera to be recalibrated is not particularly limited, but may include the position of one point in a three-dimensional coordinate system, the relative distance between two points in a three-dimensional coordinate system, and the velocity and acceleration of a feature point of an object (for example, the center of gravity of the object) in an image coordinate system. The parameter calculation unit 274 calculates new external parameters for the high-speed vision camera 160 to be recalibrated, and performs calibration correction using these.

[0036] FIG. 4 is a diagram illustrating calibration correction of the extrinsic parameters of a camera to be recalibrated according to one or more embodiments. In the embodiment of the present invention, an object m1 is imaged by three high-speed vision cameras (hereinafter referred to as cameras Cn (n = 1 to 3)), and a pair selected according to the position of the object and the occlusion situation is used as a stereo camera to perform three-dimensional measurement. Here, assuming that the position of any one of the multiple high-speed vision cameras may shift due to fluctuations over time or a sudden collision, three or more high-speed vision cameras 160 are used to detect this position shift.

[0037] The recalibration target camera determination unit 272 determines the high-speed vision camera 160 to be recalibrated based on the three-dimensional measurement position of the object m1 obtained by switching between stereo pairs and measuring, the center of gravity of the image 172 of the object m1 in the captured image 170, and other factors. For example, in FIG. 4, the second camera C2 shown by the solid line is assumed to have shifted to position C2' shown by the dotted line due to a change over time or a sudden collision. In this case, the camera to be recalibrated that has shifted in position can be determined by comparing the measurement values ​​of the stereo pair (C1, C3) of cameras without any positional shift with the measurement values ​​of the stereo pair (C1, C2) including the shifted camera. The recalibration target camera determination unit 272 constitutes the target determination means in this embodiment. The method for determining the camera to be recalibrated will be described in detail later.

[0038] Once the camera to be recalibrated is identified, calibration correction is performed. For calibration correction, the camera captures images of the target object at different positions as shown in Figure 4, and then accurately calculates the position (X t ,Y t ,Z t ) (t=1,...; t is an index that identifies a time point, but different objects may be used.) and the corresponding center of gravity position (x2 t ,y2 t ) (t=1, ...), and derive the extrinsic parameters of camera C2 by using one or more sets of three-dimensional positions and center of gravity positions. The above-mentioned projection matrix Pn is defined as the product of the internal parameters and external parameters of camera n, and by using the newly calculated external parameters of the correction target camera m, a new projection matrix Pm' (m indicates the correction target camera) is derived and incorporated as a parameter, enabling calibration correction of the three-dimensional measurement.

[0039] The parameter calculation unit 274 calculates the three-dimensional position information (X t ,Y t ,Z t) and the corresponding position information (xm t ,ym t ), the extrinsic parameters of the recalibration target camera m are calculated based on the above. The calculated extrinsic parameters are input to the calibration unit 260 and used to derive a new projection matrix Pm'. The parameter calculation unit 274 constitutes the calculation means in this embodiment.

[0040] 5A is a diagram showing an outline of a calculation method for realizing calibration correction of external parameters in this embodiment. As shown in FIG. 5A, the calibration correction involves calculating the center of gravity position (x2 t ,y2 t ) and the three-dimensional measurement value (X t ,Y t ,Z t ) is used.

[0041] In this embodiment, we focus on the positional shift due to camera collision, and assume that the internal parameters of each camera do not change from the initial calibration. To calculate the external parameters, at least four pairs of the three-dimensional positions and center of gravity positions are required, and it is possible to derive a new projection matrix including a new rotation matrix R' and a new translation matrix T' from these four pairs. For a specific calculation method, see Toru Tamaki, "Image Engineering Special Lecture Notes: 3D Reconstruction with Linear Algebra," [online], November 25, 2009, [searched July 14, 2021], Internet.<URL:https: / / ir.lib.hiroshima-u.ac.jp / 00027688> In this way, the translation matrix T and the rotation matrix R are calculated as parameters of the camera m to be recalibrated based on multiple positions in the three-dimensional coordinate system and multiple corresponding positions in the image coordinate system, and a new projection matrix Pm' is derived from these and incorporated as a parameter to achieve calibration correction. Note that it is also possible to calculate the rotation matrix R' and the translation matrix T' by using more than four sets.

[0042] In the above description, the rotation matrix R' and the translation matrix T' are calculated as camera extrinsic parameters using four or more pairs of three-dimensional positions and center of gravity positions. This assumes both translational and rotational deviations of the camera due to a collision or the like. On the other hand, the calculation can be simplified by assuming that a collision or the like mainly causes translational deviation of the camera. FIG. 5(B) is a diagram showing an outline of a calculation method for achieving calibration correction of extrinsic parameters in another embodiment. FIG. 5(B) explains a calculation method for calibrating and correcting only the translation matrix T'.

[0043] In the method shown in FIG. 5(A), both the translation matrix T' and the rotation matrix R' are calculated as extrinsic parameters, whereas in the method shown in FIG. 5(B), only the translation matrix T' is calculated, assuming no deviation in the rotational direction. In FIG. 5(B), we assume that C2 has been translationally shifted to the position indicated by the dashed line C2'. In this case, as in the description of FIG. 5(A), the calibration correction can be performed using the position (x2, y2) of the center of gravity of the object in the image coordinate system captured by camera C2, which is determined to have been shifted, and the accurate values ​​(X, Y, Z) obtained by three-dimensionally measuring a pair (C1, C3) other than camera C2, which is determined to have been shifted, as a stereo pair. However, since the intrinsic parameters of each camera and the rotation matrix R, which is part of the extrinsic parameters, are assumed to remain unchanged from the initial calibration, a new projection matrix P' including the translation matrix T' can be derived with at least one pair of positions (X, Y, Z) and center of gravity positions (x2, y2) in the three-dimensional coordinate system. This allows the translation matrix alone to be calculated as an external parameter using a simpler method. Note that even in this case, the translation matrix T' can be calculated using more than one set.

[0044] Hereinafter, with reference to the flowcharts shown in FIGS. 6 and 7 and the conceptual diagram shown in FIG. 8, the process of detecting the high-speed vision camera 160 to be recalibrated and performing calibration correction, which is executed by the three-dimensional measuring device 110 according to this embodiment, will be described in more detail.

[0045] 6 is a flowchart showing three-dimensional measurement processing executed by the three-dimensional measurement apparatus 110 according to this embodiment. The processing shown in Fig. 6 starts from step S100 in response to an instruction from the operator to start three-dimensional measurement, for example.

[0046] In step S101, the three-dimensional measurement device 110 performs initial setting of each high-speed vision camera 160 using the calibration unit 260, and calculates the projection matrix Pn required for three-dimensional measurement. In step S102, the three-dimensional measurement device 110 sets a region of interest (ROI) of a size that matches the area of ​​the target object in all high-speed vision cameras 160. In step S103, the three-dimensional measurement device 110 calculates the center of gravity position within the ROI set in step S102. In step S104, the three-dimensional measurement device 110 selects an appropriate stereo pair from the multiple high-speed vision cameras 160 depending on the situation. In step S105, the three-dimensional measurement device 110 performs three-dimensional measurement based on the principle of triangulation using the center of gravity positions (xn, yn) calculated from each image of the selected stereo pair and the projection matrix Pn calculated in step S101. In step S106, the three-dimensional measuring device 110 resets the region of interest with the calculated center of gravity of the object as the center, and loops the process back to step S103.

[0047] FIG. 7 is a flowchart showing the control of determining the camera to be recalibrated and performing calibration correction, which is performed by a timer interrupt during the loop of the main routine of three-dimensional measurement shown in FIG. 6. The control shown in FIG. 7 is called, for example, when a timer interrupt process occurs while three-dimensional measurement is being performed in the main routine, and starts from step S200. The timer interrupt can be set to any time at which the camera position deviation is to be determined. In the case of a high-speed vision camera, one main loop can be performed in as little as about 1 ms, so high-speed determination is also possible. The timer interrupt is an example and is not particularly limited. For example, the process shown in FIG. 7 may be performed in response to an explicit recalibration instruction from an operator.

[0048] In step S201, the three-dimensional measuring device 110 performs three-dimensional measurements with all stereo pairs including the reference camera, and obtains three-dimensional position information (X p ,Y p ,Z p) (p=1,...,N-1; p is an index for identifying the pair, and N is the number of cameras). In step S202, the three-dimensional measurement device 110 obtains the three-dimensional position information (X p ,Y p ,Z p ) and calculate the number of pairs that match (if they are within a predetermined error range, they are considered to match).

[0049] FIG. 8 illustrates a method for determining a misaligned camera, performed by the three-dimensional measurement device 110 according to one or more embodiments. FIG. 8(A) shows a method for determining a misaligned camera based on three-dimensional positions. In the example of FIG. 8(A), for simplicity, cameras C1 to C4 are depicted as being placed parallel to one another, but they may be placed in any position and orientation. Here, assuming that only one camera is misaligned, the number of cameras will be four or more; in the example shown in FIG. 8(A), the number of cameras is four (N=4).

[0050] Three-dimensional measurements are performed with each stereo pair (C1, C2), (C1, C3), and (C1, C4) using camera C1 as the reference camera, and each three-dimensional position information (X p ,Y p ,Z p If there is no positional deviation, all the three-dimensional position information (X p ,Y p ,Z p ) should match within a certain error range. In other words, if the number of cameras is four, three stereo pairs including the reference camera will be generated. In this case, all the three-dimensional position information (X p ,Y p ,Z p ) (p=1~3) should match. When the number of cameras is expanded to an arbitrary number N, N-1 stereo pairs are generated by fixing the reference camera, and all 3D measurement positions (X p ,Y p ,Z p ) will match.

[0051] Therefore, in the judgment of step S202, if the number of cameras is 4, there are three sets of three-dimensional measurement positions (X p ,Y p ,Z p If it is determined that the camera IDs match, the process branches to step S203. In step S203, it is determined that all cameras are normal, and the routine ends in step S209.

[0052] On the other hand, if the reference camera is displaced, all the three-dimensional position information (X p ,Y p ,Z p ) do not match. In other words, if the number of cameras is N, all the 3D position information (X p ,Y p ,Z p ) will be inconsistent.

[0053] Therefore, if it is determined in step S202 that zero pairs of three-dimensional measurement positions match, the process branches to step S205. In step S205, it is determined that the reference camera is misaligned and confirmation with a different stereo pair using a different reference camera is necessary, and control proceeds to step S206. In step S206, the process branches depending on whether confirmation of all pairs has been completed. If it is determined in step S206 that there are still unconfirmed pairs (NO), the process branches to step S207, the reference camera is changed, and the process returns to step S201. In this case, three-dimensional measurement is performed with a stereo pair including a different reference camera. For example, three-dimensional measurement is performed with each stereo pair (C2, C1), (C2, C3), and (C2, C4) using camera C2 as the reference camera.

[0054] On the other hand, if it is determined in step S206 that all pairs have been confirmed (YES), the process branches to step S208, where it is determined that positional deviations have occurred in two or more cameras, and the routine ends in step S209.

[0055] Next, let us consider the case where a positional deviation occurs in one camera other than the reference camera. If the number of cameras is four, three stereo pairs including the reference camera will be generated, but only one of the three stereo pairs will have a different 3D measurement position. Therefore, the remaining two stereo pairs will have a different 3D measurement position (X p ,Y p ,Z p In the example shown in FIG. 8(A), if the camera C2 is shifted as shown by the broken line, the three-dimensional measurement position (X1 ’ ,Y1 ’ ,Z1 ’ ) will have an error with other values ​​((X2, Y2, Z2) = (X3, Y3, Z3) = (X, Y, Z). Here, (X, Y, Z) are the correct three-dimensional positions), so if the error is greater than a certain value, it can be determined that camera C2 is misaligned. If the number of cameras is expanded to an arbitrary number N, N-1 sets of stereo pairs including the reference camera will be generated, and the three-dimensional measurement position (X p ,Y p ,Z p ) match, it can be determined that a positional shift has occurred in one camera.

[0056] Therefore, in the judgment of step S202, if the number of cameras is 4, two sets of three-dimensional measurement positions (X p ,Y p ,Z p ) match, the process branches to step S204.

[0057] In step S204, it is determined that a positional shift has occurred in the other camera m that is included in the remaining mismatched stereo pair and is not the reference camera (in the example of Figure 8(A) , C2 of the pair (C1, C2) that is not the reference camera), and new extrinsic parameters are calculated for that camera m, and the projection matrix Pm' is calculated, and this routine ends in step S209.

[0058] Note that when the number of cameras is four, there are three pairs including the reference camera, and the only possible cases are that three pairs match, two pairs match, or zero pairs match. On the other hand, when the number of cameras is an arbitrary N that is five or more, there are N-1 pairs including the reference camera, and in addition to N-1 pairs match, N-2 pairs match, or zero pairs match, other cases also exist. However, since the other cases correspond to cases where two or more cameras are misaligned, in this case (if "otherwise" is selected in step S202), the process branches to step S208, where it is determined that two or more cameras are misaligned, and the routine ends in step S209.

[0059] As described above, in the embodiment described above, the recalibration target camera determination unit 272 can detect the camera to be recalibrated by comparing the three-dimensional positions obtained using each set of multiple cameras.

[0060] In the method for checking whether the three-dimensional position of a point matches or mismatches, which has been described with reference to Fig. 8(A), a reference camera is defined, and whether the three-dimensional position matches or mismatches is checked in a world coordinate system based on the reference camera. In contrast, a method for determining whether a camera has a positional deviation without defining a reference camera will be described below with reference to Fig. 8(B).

[0061] Figure 8(B) shows a method for determining a misaligned camera based on the relative distance between two points in a three-dimensional coordinate system. Here, too, for simplicity, the cameras are depicted as being placed parallel to one another, but they may be placed in any position and orientation. Even in the method shown in Figure 8(B), if we assume that only one camera is misaligned, the number of cameras will be four or more, and in the example shown in Figure 8(A), the number of cameras is four.

[0062] In FIG. 8B, it is assumed that camera C2 has shifted. In this method, each stereo pair measures the relative distance L between two points on a moving object, and the recalibration target camera determination unit 272 detects the captured image by comparing the relative distances between the two points in a three-dimensional coordinate system measured using each pair of cameras. For example, when comparing the relative distances L1, L2, and L3 of the stereo pair (C1, C2), (C1, C3), and (C3, C4), if camera C2 shifts, L2 and L3 will match, but L1 is expected to have a different value from L2 and L3. Here, the world coordinate system (C1, C2) (C1, C3) in which camera C1 is the reference camera and the world coordinate system (C3, C4) in which camera C3 is the reference camera may differ in the position coordinates of one point because the reference cameras are different, but the relative distance between the two points will not change. Therefore, since the relative distances of the stereo pair (C1, C3) and the stereo pair (C3, C4) that are not misaligned match, it can be determined that camera C2, which has a misaligned relative distance and does not match the others, is misaligned.

[0063] In this way, in the method shown in Figure 8(A), a reference camera is defined and the object to be recorrected is determined based on whether or not the coordinate values ​​match, whereas in the method shown in Figure 8(B), it is possible to handle this by comparing any pair without defining a reference camera.

[0064] 8A and 8B, assuming a misalignment of one camera, the number of cameras required is four or more, but it is possible to determine the camera to be recorrected with at least one measurement of the position and relative distance. In the above-described embodiment, it is assumed that only one of the multiple cameras will be misaligned, and the number of cameras is four or more. However, the number of cameras that can be detected as recorrection targets is not limited to one, and the system may be configured to handle misalignment of multiple cameras, provided that a sufficient number of cameras are installed.

[0065] In the embodiment described with reference to Figures 6 to 8, the camera to be re-corrected is determined based on the position of one point and the relative distance between two points in a three-dimensional coordinate system measured three-dimensionally using a stereo pair. Hereinafter, with reference to Figures 9 to 12, another embodiment will be described in which the camera to be re-set is determined using the velocity and acceleration of a feature point of an object (for example, the center of gravity of the object) in the image coordinate system of an image captured by the camera.

[0066] Fig. 9 is a flowchart showing control executed by a three-dimensional measuring apparatus 110 according to another embodiment to perform three-dimensional measurement, as well as to determine a camera to be recalibrated and perform calibration correction. The process shown in Fig. 9 shows processing to determine a camera that has shifted in position due to a change in the center of gravity position in the image coordinate system of the object. The control shown in Fig. 9 starts, for example, from step S300 in response to an instruction from an operator to start three-dimensional measurement.

[0067] In step S301, the three-dimensional measurement device 110 performs initial setting of each high-speed vision camera 160 using the calibration unit 260. In step S302, the three-dimensional measurement device 110 sets a region of interest (ROI) of a size that matches the area of ​​the object in all high-speed vision cameras 160. In step S303, the three-dimensional measurement device 110 calculates the center of gravity position within the ROI set in step S102. In step S304, the three-dimensional measurement device 110 selects an appropriate stereo pair from the multiple high-speed vision cameras 160 depending on the situation. In step S305, the three-dimensional measurement device 110 performs three-dimensional measurement based on the principle of triangulation using the center of gravity positions (xn, yn) and the projection matrix Pn calculated from each image of the selected stereo pair. In step S306, the three-dimensional measurement device 110 resets the region of interest centered on the calculated center of gravity position of the object. Up to this point, the flow is the same as that shown in FIG. 6, but in the process shown in FIG. 8, after step S306, the process proceeds to step S307.

[0068] In step S307, the three-dimensional measuring device 110 calculates the center of gravity position in the region of interest of all cameras, and calculates the center of gravity velocity Vn by combining it with the previous center of gravity position.t In step S308, the three-dimensional measuring device 110 determines whether all the cameras satisfy the tolerance condition. In a specific embodiment, the tolerance condition determined here is the center-of-gravity velocities Vn of all the cameras calculated in step S307. t (t identifies the time) is the previous center of gravity velocity Vn t-1 Compared with the condition that there is no change of more than a certain threshold α (Vn t-1 -α <Vn t <Vn t-1 +α).

[0069] FIG. 10 illustrates a method for determining a camera that is misaligned using the velocity of the center of gravity, which is executed by a three-dimensional measuring device 110 according to another embodiment. FIG. 10(A) is a conceptual diagram showing a method for determining a camera that is misaligned. In FIG. 10(A), similar to FIGS. 8(A) and 8(B), for simplicity, cameras C1 to C3 are depicted as being placed parallel to each other, but they may be placed in any position and orientation. Here, assuming that only one camera is misaligned, the number of cameras will be three or more; in the example shown in FIG. 10(A), the number of cameras is three (N=3).

[0070] In the example shown in Figure 10(A), assume that camera C2 has shifted as indicated by the dashed line. At this time, if the center of gravity position of the region of interest set by camera C2 is viewed over time, it is likely to change suddenly. If the center of gravity position of the region of interest of the other cameras does not change suddenly, but the change in the center of gravity position of only specific camera C2 is large, it can be determined that a position shift has occurred in camera C2 for some reason.

[0071] Figure 10(B) is a graph showing a schematic representation of the time change in center-of-gravity velocity V2 of an object moving at a predetermined velocity V, captured by the high-speed vision camera 160. Each camera constantly tracks the object, obtaining the center-of-gravity position (xn, yn) of the object. From the time change in this center-of-gravity position, the center-of-gravity velocity Vn can be derived for each camera. If the object is moving in the same direction at a constant velocity V, the center-of-gravity velocity observed by each camera should be constant. However, as shown in Figure 10(A), if the second camera C2 becomes misaligned at time t1, the direction of the object relative to that camera changes, resulting in a change in the constant value of the center-of-gravity velocity, as shown in Figure 10(B). Therefore, by monitoring the center-of-gravity velocity Vn, it can be determined that the camera has become misaligned if a change in velocity equals or exceeds a certain threshold α.

[0072] 9 again, if there is a camera in which a change in velocity Vn equal to or greater than a certain threshold α has occurred (NO in step S308), the process proceeds to step S309, where it is determined that the position of that camera has shifted, new extrinsic parameters of the recorrection target camera m are calculated, and a new projection matrix Pm is obtained. On the other hand, if no change equal to or greater than the certain threshold α has occurred in any of the cameras (YES in step S308), the process proceeds to step S310, where the count t for saving the next center-of-gravity velocity is incremented, and the process returns to step S303.

[0073] In this way, the recorrection target camera determination unit 272 detects changes over time in the positions of feature points in images captured using each of the multiple cameras, and can detect a camera whose speed calculated based on the changes over time in the positions of the detected feature points satisfies a predetermined condition as a correction target. Note that in the embodiment described with reference to Fig. 9 and Fig. 10, the recorrection target can be determined based only on the center of gravity position and speed in the image coordinate system observed by each camera, so the number of cameras required can be reduced.

[0074] 9 and 10 are based on the premise that the object is moving at a constant speed. Hereinafter, a further embodiment that can handle cases where the speed of the object changes (i.e., when accelerating or decelerating) will be described with reference to FIG.

[0075] FIG. 11 illustrates a method for determining a camera that is misaligned using center-of-gravity acceleration, executed by a three-dimensional measurement device 110 according to yet another embodiment. FIG. 11(A) illustrates a change in the center-of-gravity velocity of an object when a camera is misaligned. As described above, each camera constantly tracks the object, and the center-of-gravity position (xn, yn) of the object is obtained. From the change in the center-of-gravity position over time, the center-of-gravity acceleration an can be derived for each camera. If the object is moving with a constant velocity change (constant acceleration), the center-of-gravity velocity should increase or decrease at a constant rate, as shown in FIG. 11(A). On the other hand, when converted to acceleration, it should result in a constant acceleration. FIG. 11(B) illustrates a change in the center-of-gravity acceleration of an object when a camera is misaligned.

[0076] Here, as shown in FIG. 10(A), if the second camera C2 is displaced at time t1, the direction of travel of the object relative to that camera changes, causing a change in the constant value of the center-of-gravity velocity, and similarly, as shown in FIG. 11(B), causing a change in the constant value of the center-of-gravity acceleration. Therefore, by monitoring the center-of-gravity acceleration an, it is possible to determine that the camera has been displaced if a change in velocity equals or exceeds a certain threshold value β. In this case, in step S307 shown in FIG. 9, in addition to the velocity Vn, the acceleration an is calculated, and in step S308, the center-of-gravity acceleration an of all cameras calculated in step S307 is calculated. t (t identifies the time) is the previous center of gravity acceleration an t-1 The tolerance condition (an t-1 -β <an t <an t-1 +β) can be determined.

[0077] In this way, the recorrection target camera determination unit 272 detects changes over time in the positions of feature points in images captured using each of multiple cameras, and can detect as a correction target a camera whose acceleration calculated based on the changes over time in the positions of the detected feature points satisfies predetermined conditions.

[0078] 9 to 11, a camera that has been suddenly displaced is detected as a target for re-correction. Hereinafter, a further embodiment that can deal with a case where the camera has been displaced over time will be described with reference to FIG.

[0079] 12 is a flowchart showing control of determining a camera to be recalibrated and performing calibration correction in addition to three-dimensional measurement, executed by a three-dimensional measurement device 110 according to yet another embodiment. Fig. 12 shows the process of determining a camera that has shifted in position over time as a camera to be recalibrated.

[0080] The control shown in Fig. 12 starts from step S400 in response to an instruction from an operator to start three-dimensional measurement, for example. Note that the processing from steps S400 to S407 shown in Fig. 12 is the same as the processing from steps S300 to S307 shown in Fig. 9, and therefore a description thereof will be omitted.

[0081] In step S408, the three-dimensional measuring device 110 divides the time index t by a predetermined time time, and branches the process depending on whether the remainder is 0. Here, the process branches to step S409 only once every time times, and branches to step S410 the remaining time-1 times. This constant time sets the time for accumulating changes in the center of gravity velocity and comparing them with an arbitrarily set velocity threshold γ.

[0082] In step S410, the three-dimensional measuring device 110 determines whether or not the first permissible condition, which is the same as the permissible condition shown in Fig. 9, is satisfied. In a specific embodiment, the first permissible condition determined here is the center-of-gravity velocities Vn t(t identifies the time) is the previous center of gravity velocity vn t-1 Compared with the condition that there is no change of more than a certain threshold α (Vn t-1 -α <Vn t <Vn t-1 +α).

[0083] If there is a camera for which a change in velocity Vn equal to or greater than a certain threshold α has occurred (NO in step S410), the process proceeds to step S411. In step S411, the three-dimensional measuring device 110 determines that the position of the camera for which a change in velocity Vn equal to or greater than the threshold α has occurred is misaligned, calculates new extrinsic parameters for the camera m to be recorrected, and obtains a new projection matrix Pm. On the other hand, if it is determined in step S410 that the first permissible condition is satisfied for all cameras (YES), the process proceeds to step S412. In step S412, the three-dimensional measuring device 110 increments the count t for saving the new velocity, and returns the process to step S403.

[0084] On the other hand, in step S409, the three-dimensional measuring device 110 determines whether all cameras satisfy the second permissible condition. In a specific embodiment, the second permissible condition determined here is the center-of-gravity velocities Vn accumulated for the time period time of all cameras calculated in step S407. t (t identifies the time) is the center of gravity velocity Vn t-time Compared with the condition that there is no change of more than a certain threshold γ (Vn t-time -γ <Vn t <Vn t-time +γ). Note that although the cumulative velocity is used as the condition here, the cumulative acceleration may also be used as the condition.

[0085] If there is a camera for which a change in cumulative velocity Vn equal to or greater than a certain threshold γ has occurred (NO in step S409), the process proceeds to step S411. In step S411, the three-dimensional measuring device 110 determines that the position of the camera for which a change in cumulative velocity Vn equal to or greater than the threshold γ has occurred is misaligned, calculates new extrinsic parameters for that camera m to be recorrected, and obtains a new projection matrix Pm. On the other hand, if it is determined in step S409 that the second tolerance condition is satisfied for all cameras (YES), the process proceeds to step S410.

[0086] In this way, the recorrection target camera determination unit 272 detects changes over time in the positions of feature points in images captured using each of multiple cameras, and can detect as a correction target a camera whose cumulative value of velocity or acceleration calculated based on the changes over time in the positions of the detected feature points satisfies predetermined conditions.

[0087] According to the embodiments described above, it is possible to provide an information processing device, an information processing system, a calibration method, and a program that are capable of determining which imaging means require re-correction in three-dimensional measurement using multiple imaging means, and correcting parameters in three-dimensional measurement using those imaging means.

[0088] In particular, according to the three-dimensional measurement of the above-described embodiment, even if a positional deviation occurs in the camera after the start of the three-dimensional measurement, it is possible to achieve high-speed and highly accurate three-dimensional measurement.

[0089] In the described embodiment, the three-dimensional measurement system 100 including multiple high-speed vision cameras 160, a three-dimensional measurement device 110, and a camera synchronizer 150 has been described as an example of an information processing system. However, the information processing system is not limited to the above configuration, and components may be added or removed. For example, a device that combines the functions of the three-dimensional measurement device 110 and the camera synchronizer 150 may be provided, and the three-dimensional measurement device 110 and the camera synchronizer 150 may be omitted. In other embodiments, the system may be configured to include, in addition to the multiple high-speed vision cameras 160, the three-dimensional measurement device 110, and the camera synchronizer 150, a robotic device such as a robot arm that physically manipulates the object 102, and a controller that controls the robotic device. In this case, a device that combines the functions of the three-dimensional measurement device 110 and the controller may be provided.

[0090] In the embodiment described, the n high-speed vision cameras 160 are fixed and the target object is moving, but the present invention can be applied to any configuration as long as the relative positional relationship between the n high-speed vision cameras 160 and the target object changes.

[0091] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above.

[0092] The devices described in the embodiments represent only one of multiple computing environments for implementing the embodiments disclosed herein. In one embodiment, the 3D measurement apparatus includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

[0093] The above has described the embodiments and examples of the present invention, but the embodiments and examples of the present invention are not limited to the above-described embodiments and examples, and may be modified within the scope of what a person skilled in the art could conceive, such as other embodiments, other examples, additions, changes, deletions, etc., and any aspect is included in the scope of the present invention as long as it exhibits the functions and effects of the present invention. [Explanation of symbols]

[0094] 100... three-dimensional measurement system, 102... object, 110... three-dimensional measurement device, 150... camera synchronization device, 160... high-speed vision camera, 170... captured image, 172... object image, 210... image input unit, 220... object search unit, 230... tracking unit, 232... center of gravity calculation unit, 234... area setting unit, 250... three-dimensional measurement unit, 260... calibration unit, 262... correction transformation matrix calculation unit, 264... fundamental matrix calculation unit, 270... calibration correction unit, 27 2...recorrection target determination unit, 274...parameter calculation unit, 500...personal computer, 501...CPU, 502...ROM, 503...RAM, 504...HDD, 505...HDD controller, 506...display, 508...external device connection I / F, 509...network I / F, 511...keyboard, 512...pointing device, 514...DVD-RW drive, 513...DVD-RW media, 516...media I / F, 515...recording media [Prior art documents] [Patent documents]

[0095] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-233639

Claims

1. An information processing device for performing calibration of three-dimensional measurement using a plurality of imaging means, an object determination means for determining an imaging means to be recorrected among the plurality of imaging means based on a velocity or acceleration based on a time change in a position of a feature point in an image of an object captured by each of the plurality of imaging means; a calculation means for calculating parameters of three-dimensional measurement using the imaging means to be recorrected, based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; An information processing device comprising:

2. The information processing device described in claim 1, wherein the object includes one or more objects, and the three-dimensional position information acquired using a combination of imaging means other than the object to be recorrected indicates different positions of the one or more objects at one time point or multiple time points in a three-dimensional coordinate system obtained by three-dimensional measurement, and the corresponding position information indicates corresponding feature point positions at the one time point or multiple time points in an image coordinate system obtained by imaging by the imaging means of the object to be recorrected, and the calculation means calculates a translation matrix and a rotation matrix as parameters of the imaging means of the object to be recorrected based on the multiple positions in the three-dimensional coordinate system and the multiple feature point positions in the image coordinate system.

3. 2. The information processing device according to claim 1, wherein the three-dimensional position information acquired using a combination of imaging means other than the object to be recorrected indicates one or more positions including a position of the object at one point in time in a three-dimensional coordinate system obtained by three-dimensional measurement, the corresponding position information indicates one or more feature point positions including corresponding feature point positions at one point in time in an image coordinate system obtained by imaging by the imaging means to be recorrected, and the calculation means calculates a translation matrix as a parameter of the imaging means to be recorrected based on the one or more positions in the three-dimensional coordinate system and the one or more feature point positions in the image coordinate system.

4. An information processing device described in any one of claims 1 to 3, wherein the target determination means detects an imaging means whose speed, acceleration, cumulative value of the speed, or cumulative value of the acceleration meets predetermined conditions as the imaging means to be recorrected.

5. An information processing device for calibrating three-dimensional measurement using multiple imaging means, comprising: an object determining means for determining an imaging means to be recorrected among the plurality of imaging means based on information obtained from a captured image of an object captured using the imaging means; a calculation means for calculating parameters of three-dimensional measurement using the imaging means to be recorrected, based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; wherein the object determination means detects the imaging means to be recorrected by comparing, as the information, relative distances between two points in a three-dimensional coordinate system obtained using each set of the plurality of imaging means.

6. An information processing device for calibrating three-dimensional measurement using multiple imaging means, comprising: an object determining means for determining an imaging means to be recorrected among the plurality of imaging means based on information obtained from a captured image of an object captured using the imaging means; a calculation means for calculating parameters of three-dimensional measurement using the imaging means to be recorrected, based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; wherein the target determination means detects, as the information, changes over time in positions of feature points in images captured using each of the plurality of imaging means, and detects, as the imaging means to be recorrected, an imaging means whose speed calculated based on the changes over time in the positions of the detected feature points satisfies a predetermined condition.

7. An information processing device for calibrating three-dimensional measurement using multiple imaging means, comprising: an object determining means for determining an imaging means to be recorrected among the plurality of imaging means based on information obtained from a captured image of an object captured using the imaging means; a calculation means for calculating parameters of three-dimensional measurement using the imaging means to be recorrected, based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; wherein the target determination means detects, as the information, changes over time in positions of feature points in images captured using each of the plurality of imaging means, and detects, as the imaging means to be recorrected, an imaging means for which acceleration calculated based on the detected changes over time in the positions of the feature points satisfies a predetermined condition.

8. An information processing device for calibrating three-dimensional measurement using multiple imaging means, comprising: an object determining means for determining an imaging means to be recorrected among the plurality of imaging means based on information obtained from a captured image of an object captured using the imaging means; a calculation means for calculating parameters of three-dimensional measurement using the imaging means to be recorrected, based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; wherein the target determination means detects changes over time in positions of feature points in images captured using each of the plurality of imaging means, and detects, as the imaging means to be recorrected, an imaging means for which a cumulative value of velocity or acceleration calculated based on the changes over time in the positions of the detected feature points satisfies a predetermined condition.

9. An information processing system for performing three-dimensional measurement using a plurality of imaging means, A plurality of imaging means arranged at different positions; an object determination means for determining an imaging means to be recorrected among the plurality of imaging means based on a velocity or acceleration based on a time change in a position of a feature point in an image of an object captured by each of the plurality of imaging means; a calculation means for calculating parameters of three-dimensional measurement using the imaging means to be recorrected, based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; An information processing system comprising:

10. A calibration method for three-dimensional measurement using a plurality of imaging means, comprising: determining which of the plurality of imaging means is to be recorrected based on a velocity or acceleration based on a change over time in a position of a feature point in an image of an object captured using each of the plurality of imaging means; calculating parameters of the imaging means to be recorrected based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; Calibration methods, including:

11. A program for realizing an information processing device for performing calibration of three-dimensional measurement using a plurality of imaging means, the program comprising: a target determination means for determining which of the plurality of imaging means is to be recorrected based on a velocity or acceleration based on a time change in a position of a feature point in an image of an object captured by each of the plurality of imaging means; and a calculation means for calculating parameters of the imaging means to be recorrected based on three-dimensional position information of the object acquired using a combination of imaging means other than the imaging means to be recorrected and corresponding position information in an image captured by the imaging means to be recorrected; A program to function as a

Images