Information processing device, information processing method, and information processing program

Abstract

Provided is a new method for evaluating the accuracy of registration.　This information processing device includes: a first point cloud generation unit for extracting a first point cloud included in a first region determined with reference to a reference marker among point clouds generated on the basis of information pertaining to a reference object; a second point cloud generation unit for extracting a second point cloud included in a second region determined with reference to the reference marker among point clouds included in a composite image; a search unit for searching, with regard to a first point pair selected from within the first point cloud, for a second point pair corresponding to the first point pair in the second point cloud; and an evaluation unit for evaluating the accuracy of the correspondence relationship on the basis of the shape dimensions of the first point pair and the second point pair.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to registration for multiple cameras.

[0002] For example, to realize an autonomous robot, spatial recognition is necessary. For spatial recognition, for example, a method using multiple images captured by multiple cameras may be adopted. When multiple cameras are used, alignment between the multiple cameras (registration or camera registration) is required. Registration determines the extrinsic parameters (homogeneous transformation matrix) of the cameras.

[0003] For example, Japanese Patent Laid-Open Publication No. 2022-039906 (Patent Document 1) discloses a multi-sensor composite calibration method including internal parameter calibration of a camera and external parameter calibration of a laser radar camera.

[0004] Japanese Patent Application Laid-Open No. 2022-039906

[0005] The above-mentioned prior art documents disclose that the accuracy of calibration has been improved, but do not disclose a method for evaluating the accuracy of calibration itself.

[0006] An object of the present invention is to provide a new method for evaluating the accuracy of registration.

[0007] An information processing device according to one example of the present invention includes: a synthesis unit that generates a synthetic image by synthesizing a first image obtained by capturing an image of a reference object on which a reference marker is arranged with a first camera and a second image obtained by capturing an image of the reference object with a second camera on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; a first point cloud generation unit that extracts a first point cloud included in a first region determined based on the reference marker from the point cloud generated based on information of the reference object; a second point cloud generation unit that extracts a second point cloud included in a second region determined based on the reference marker from the point cloud included in the synthetic image; a search unit that searches for a second point pair in the second point cloud that corresponds to the first point pair selected from the first point cloud; and an evaluation unit that evaluates the accuracy of the correspondence based on the shape and dimensions of the first point pair and the second point pair.

[0008] According to this configuration, by acquiring a first pair of points selected from the first point cloud and a second pair of points in the second point cloud that corresponds to the first pair of points, based on a reference object on which a reference marker is placed, the accuracy of the correspondence can be evaluated based on the shape and dimensions of the first pair of points and the second pair of points. Since the absolute error can be evaluated based on the shape and dimensions of the first pair of points and the second pair of points, the accuracy of safety design, etc. can be improved.

[0009] The information processing device may further include a determination unit that determines the correspondence based on a difference between the position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the first camera and the position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the second camera. With this configuration, the correspondence between the camera coordinate systems can also be determined using the reference marker used to evaluate the accuracy of the correspondence.

[0010] The information processing device may further include an optimization unit that optimizes the correspondence relationship so as to reduce a difference in shape and size between the first point pair and the second point pair. With this configuration, it is possible to improve the accuracy of the correspondence relationship between the camera coordinate systems based on the shape and size of the first point pair and the second point pair.

[0011] The search unit may determine coordinates in the second point cloud corresponding to each point of the first point pair, and determine each point of the second point pair based on the center of gravity of one or more points that exist within a predetermined distance from the determined coordinates. This configuration makes it possible to determine points in the second point cloud that correspond to each point of the first point pair in a statistically robust manner.

[0012] Each of the first and second regions may be determined based on a surface of the reference object on which a reference marker is disposed. With this configuration, by using the surface on which the reference marker is disposed as a reference, it is possible to optimize the point cloud required for associating the first point pair with the second point pair.

[0013] The geometric dimensions may include length and angle. The evaluation unit may output a histogram of length errors and a histogram of angle errors. With this configuration, accuracy can be evaluated from the perspective of each of length and angle included in the geometric dimensions.

[0014] The evaluation unit may calculate the variation in the shape and dimension error by repeating the calculation of the shape and dimension error of the second point pair. This configuration makes it possible to predict fluctuations that occur during measurement.

[0015] The first point pair may be selected based on the geometric features of the reference object, which allows the first point pair to be selected more stably.

[0016] An information processing method according to another example of the present invention includes: generating a composite image by combining a first image obtained by capturing an image of a reference object on which a reference marker is arranged with a first camera and a second image obtained by capturing an image of the reference object with a second camera, on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; extracting a first point cloud included in a first region determined based on the reference marker from the point cloud generated based on information of the reference object; extracting a second point cloud included in a second region determined based on the reference marker from the point cloud included in the composite image; searching for a second point pair in the second point cloud that corresponds to the first point pair selected from the first point cloud; and evaluating the accuracy of the correspondence based on the shape and dimensions of the first point pair and the second point pair.

[0017] An information processing program according to yet another example of the present invention causes a computer to perform the following operations: generate a composite image by combining a first image obtained by capturing an image of a reference object on which a reference marker is placed with a first camera and a second image obtained by capturing an image of the reference object with a second camera, on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; extract a first point cloud included in a first area determined based on the reference marker from the point cloud generated based on information of the reference object; extract a second point cloud included in a second area determined based on the reference marker from the point cloud included in the composite image; for a first point pair selected from the first point cloud, search for a second point pair in the second point cloud that corresponds to the first point pair; and evaluate the accuracy of the correspondence based on the shape and dimensions of the first point pair and the second point pair.

[0018] According to the present invention, the accuracy of registration can be evaluated.

[0019] FIG. 7 is a schematic diagram showing an example of the configuration of an information processing system according to the present embodiment. FIG. 8 is a schematic diagram showing an example of the hardware configuration of an information processing device according to the present embodiment. FIG. 9 is a flowchart showing an example of the processing procedure of an information processing method according to the present embodiment. FIG. 10 is a flowchart showing another example of the processing procedure of an information processing method according to the present embodiment. FIG. 11 is a diagram for explaining an example of rough calibration processing according to the present embodiment. FIG. 12 is a diagram for explaining another example of rough calibration processing according to the present embodiment. FIG. 13 is a diagram showing an example of AprilTag. FIG. 14 is a diagram for explaining processing for calculating a homogeneous transformation matrix in the rough calibration processing shown in FIGS. 5 and 6. FIG. 15 is a diagram showing an example of a marker coordinate system of markers arranged on each surface of an object. FIG. 16 is a diagram showing an example of the relationship between rotation and translation between markers based on a CAD canonical coordinate system. FIG. 17 is a diagram for explaining an example of ROI point cloud extraction processing according to the present embodiment. FIG. 18 is a diagram for explaining an example of CAD-derived point cloud generation processing according to the present embodiment. FIG. 19 is a diagram for explaining an example of key point search in identification processing according to the present embodiment. FIG. 19 is a diagram for explaining a method of searching for key points by a minimum distance method. FIG. 19 is a diagram for explaining a method of searching for key points by a centroid method. FIG. 19 is a diagram for explaining a method of searching for key points by an average centroid method. FIG. 1 is a diagram for explaining a method of searching for key points by the average center of gravity method and M-estimation. FIG. 2 is a diagram for explaining a method of searching for key points by the average center of gravity method and RANSAC method. FIG. 3 is a diagram for explaining a method of searching for key points by the average center of gravity method and RANSAC method. FIG. 4 is a diagram for explaining an example of shape dimension calculation in the identification processing according to the present embodiment. FIG. 5 is a diagram for explaining a processing example of information processing according to the present embodiment. FIG. 6 is a diagram for explaining a processing example of information processing according to the present embodiment. FIG. 7 is a diagram for explaining a processing example of information processing according to the present embodiment. FIG. 8 is a schematic diagram showing an example of the configuration of a reference object used in the information processing system according to the present embodiment. FIG. 9 is a schematic diagram showing an example of the configuration of a reference object used in the information processing system according to the present embodiment.FIG. 1 is a schematic diagram showing an example of the configuration of a reference object used in an information processing system according to the present embodiment. FIG. 2 is a diagram for explaining an example of absolute error evaluation in the accuracy evaluation process according to the present embodiment. FIG. 3 is a diagram for explaining an example of iterative error evaluation in the accuracy evaluation process according to the present embodiment. FIG. 4 is a schematic diagram showing an example of output of an accuracy evaluation result according to the present embodiment. FIG. 5 is a diagram for explaining an example of a penalty term in the fine tuning process according to the present embodiment. FIG. 6 is a diagram showing an example of change in length error due to the fine tuning process according to the present embodiment. FIG. 7 is a diagram showing an example of change in angle error due to the fine tuning process according to the present embodiment. FIG. 8 is a diagram for explaining accuracy confirmation using point cloud thickness in the information processing method according to the present embodiment. FIG. 9 is a flowchart showing an example of the processing procedure of the method for calculating accuracy spatial distribution according to the present embodiment. FIG. 10 is a diagram showing an example of visualization of the accuracy spatial distribution obtained by the method for calculating accuracy spatial distribution according to the present embodiment. FIG. 11 is a schematic diagram showing an example of the functional configuration of an information processing system according to the present embodiment. FIG. 12 is a diagram for explaining a method of setting an arbitrary world coordinate system in the information processing system according to the present embodiment.

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings, in which the same or corresponding parts are designated by the same reference numerals and will not be described repeatedly.

[0021] <A. Application Example> Fig. 1 is a schematic diagram showing an example of the configuration of an information processing system 1 according to the present embodiment. Referring to Fig. 1, information processing system 1 includes an information processing device 100 and a plurality of cameras 10-1, 10-2, ..., 10-N (N is an integer of 2 or more) (hereinafter, sometimes collectively referred to as "cameras 10").

[0022] The camera 10 captures, for example, a two-dimensional image (RGB image) and a depth image simultaneously. The camera 10 may also capture a three-dimensional image.

[0023] The information processing device 100 performs alignment (registration or camera registration) between multiple cameras 10. The cameras 10-1, 10-2, ..., 10-N each have a camera coordinate system 12-1, 12-2, ..., 12-N (hereinafter, sometimes collectively referred to as "camera coordinate system 12"). The camera coordinate system 12 corresponds to the field of view of each camera 10.

[0024] Registration is a process for determining a correspondence relationship between the camera coordinate systems 12 for multiple cameras 10. In registration, the correspondence relationship between the camera coordinate systems 12 is determined based on one or more points in the space SP. The correspondence relationship between the camera coordinate systems 12 may be, for example, a homogeneous transformation matrix, which is an external parameter of the camera. In the following description, a homogeneous transformation matrix will be described as an example of the correspondence relationship between the camera coordinate systems 12, but any parameter, transformation matrix, or transformation function may be used.

[0025] Once registration is successful, it is possible to calculate a homogeneous transformation matrix T that indicates the correspondence between the camera coordinate system 12 of one camera 10 and the camera coordinate system 12 of another camera 10. The homogeneous transformation matrix T is an extrinsic parameter of one camera 10 relative to another camera 10.

[0026] For example, if the camera coordinate system 12-1 of the camera 10-1 is used as a criterion, the homogeneous transformation matrix T １ ２ Similarly, a homogeneous transformation matrix T from the camera coordinate system 12-1 to the camera coordinate system 12-N is calculated. １ Ｎ is calculated. A homogeneous transformation matrix is ​​a matrix that indicates rotation and translation (position).

[0027] The information processing system 1 according to the present embodiment evaluates the accuracy of the correspondence (homogeneous transformation matrix T) between the camera coordinate systems 12 to be calculated.

[0028] <B. Example of Hardware Configuration> Next, an example of a hardware configuration of information processing device 100 in information processing system 1 according to the present embodiment will be described.

[0029] 2 is a schematic diagram showing an example of a hardware configuration of information processing device 100 according to the present embodiment. Referring to FIG. 2, information processing device 100 is an example of a computer, and includes one or more processors 102, a memory 104, an input interface 106, a display interface 110, a camera interface 114, a network controller 116, and storage 120.

[0030] The one or more processors 102 provide the processes and functions described below by executing computer-readable instructions included in a program stored in storage 120. When the one or more processors 102 execute the computer-readable instructions included in the program, part or all of the program may be loaded into memory 104.

[0031] The storage 120 is an example of a non-transitory computer-readable medium. The storage 120 stores, for example, an operating system (OS) 122 and an information processing program 124. The OS 122 includes computer-readable instructions for providing functions required by a computer and computer-readable instructions for generating an environment in which the information processing program 124 is executed.

[0032] The information processing program 124 includes computer readable instructions for the processes described below.

[0033] The input interface 106 receives input commands from an input device 108 such as a keyboard, a touch panel, a mouse, or a tablet. The input device 108 may be a part of the configuration of the information processing device 100.

[0034] The display interface 110 outputs the video signal to a display 112. The display 112 may be a part of the configuration of the information processing device 100.

[0035] The camera interface 114 exchanges data with the multiple cameras 10. The network controller 116 exchanges data with other information processing devices via the network.

[0036] In this specification, the term "processor" encompasses not only arithmetic circuits that sequentially execute computer-readable instructions, such as a central processing unit (CPU) and a graphics processing unit (GPU), but also hardwired circuits. Examples of hardwired circuits include an application specific integrated circuit (ASIC) and a field programmable gate array (FPGA). The term "processor" encompasses a system on a chip (SoC) that integrates a core, memory, and peripheral circuits.

[0037] <C. Example of Processing Procedure> Next, an example of the processing procedure executed in information processing system 1 according to the present embodiment will be described.

[0038] 3 is a flowchart showing an example of a processing procedure of an information processing method according to the present embodiment. The processing shown in FIG. 3 includes registration and accuracy evaluation. Each step shown in FIG. 3 may be realized by processor 102 of information processing device 100 executing information processing program 124.

[0039] 3, the information processing device 100 executes a rough calibration process (step S1). After the rough calibration process is executed, a correspondence relationship between the camera coordinate systems 12 is determined. Step S1 may be executed in advance, independently of the processes of the other steps.

[0040] The information processing device 100 executes a ROI (Region of Interest) point cloud extraction process (step S2). By executing the ROI point cloud extraction process, the information processing device 100 extracts a point cloud included in the ROI (hereinafter also referred to as an "ROI point cloud").

[0041] The information processing device 100 executes a CAD (Computer Aided Design)-derived point cloud generation process (step S3). By executing the CAD-derived point cloud generation process, the information processing device 100 generates a point cloud based on CAD information (hereinafter also referred to as a "CAD-derived point cloud"). Step S3 may be executed in advance, independently of the processing of other steps.

[0042] In this specification, a "point cloud" refers to a collection of points including position information. A "point cloud" may include color information in addition to position information. The position information may be three-dimensional data or 2.5-dimensional data. The position information may include two-dimensional data and depth information (distance information) independently of each other.

[0043] Next, the information processing apparatus 100 executes an identification process (step S4). The identification process includes a process of determining a correspondence between the ROI point cloud extracted by the ROI point cloud extraction process and the CAD-derived point cloud generated by the CAD-derived point cloud generation process. The identification process includes a process of calculating the geometry of key point pairs of the CAD-derived point cloud and the geometry of key point pairs of the ROI point cloud.

[0044] Next, the information processing device 100 executes an accuracy evaluation process (step S5). The accuracy evaluation process includes a process of calculating accuracy based on the shape dimensions calculated in the identification process.

[0045] In this specification, "accuracy" refers to the direct or indirect accuracy of the homogeneous transformation matrix T (external parameters) that is the result of registration. However, "accuracy" may also refer to the accuracy of spatial information reconstructed based on images captured by multiple cameras. Furthermore, "accuracy" may also refer to the accuracy of the result of combining images captured by multiple cameras.

[0046] Subsequently, the information processing device 100 outputs the accuracy evaluation result (step S6). The output of the accuracy evaluation result includes, for example, a spatial visualization of the calculated accuracy.

[0047] In addition to the rough calibration process, a fine-tuning process may be performed to optimize the correspondence between the camera coordinate systems 12 .

[0048] 4 is a flowchart showing another example of the processing procedure of the information processing method according to the present embodiment. The processing shown in FIG. 4 includes fine tuning processing. Each step shown in FIG. 4 may be realized by the processor 102 of the information processing device 100 executing the information processing program 124.

[0049] The information processing apparatus 100 executes a rough calibration process (step S1). Step S1 may be executed in advance, independently of the processes of the other steps.

[0050] The information processing device 100 executes an ROI point cloud extraction process (step S2). The information processing device 100 executes a CAD-derived point cloud generation process (step S3). Step S3 may be executed in advance, independently of the processes of the other steps.

[0051] The information processing device 100 executes an identification process (step S4), and then executes a fine-tuning process (step S7).

[0052] The information processing apparatus 100 determines whether or not the end condition of the fine tuning process is satisfied (step S8). If the end condition of the fine tuning process is not satisfied (NO in step S8), the processes from step S2 onwards are repeated.

[0053] If the condition for terminating the fine-tuning process is satisfied (YES in step S8), the information processing device 100 executes the accuracy evaluation process (step S5). Subsequently, the information processing device 100 outputs the accuracy evaluation result (step S6). Then, the process ends.

[0054] Each step will be described in detail below. <D. Rough Calibration Process (Step S1)> Next, the rough calibration process (Step S1) will be described.

[0055] As described above, in the rough calibration process, the correspondence relationship between the camera coordinate systems 12 is determined. More specifically, the homogeneous transformation matrix T of the camera coordinate system 12C of the reference camera (hereinafter also referred to as the "reference camera 10C") to the camera coordinate system 12T of another camera (hereinafter also referred to as the "target camera 10T") is determined. Ｃ Ｔ is calculated.

[0056] Homogeneous transformation matrix T Ｃ Ｔ Any method may be used for the calculation of , but below, a method using a reference marker and a method using a point cloud will be exemplified.

[0057] (d1: Method Using Reference Markers) Fig. 5 is a diagram for explaining an example of rough calibration processing according to this embodiment. Fig. 5 shows an example of rough calibration processing using reference markers (hereinafter also simply referred to as "markers").

[0058] 5, a reference object (hereinafter also simply referred to as "object") 20 exists within a field of view (FOV) 14C of each of a reference camera 10C and a target camera 10T. A marker 22 is disposed on the object 20. The marker 22 exists within the field of view 14C of the reference camera 10C and within the field of view 14T of the target camera 10T.

[0059] The marker 22 may be, for example, a marker whose position and orientation in three-dimensional space can be detected. The marker 22 may be, for example, a two-dimensional marker. When the marker 22 is a two-dimensional marker, a surface on which the two-dimensional marker is arranged (hereinafter also referred to as an "effective surface") is defined.

[0060] 6 is a diagram for explaining another example of the rough calibration process according to the present embodiment. In FIG. 6, a case is shown in which a plurality of cameras 10 cannot measure the same marker 22.

[0061] 6, a marker 22-1 is placed on one surface of the object 20, and a marker 22-2 is placed on another surface of the object 20. With this arrangement, the marker 22-1 is present within the field of view 14C of the reference camera 10C, and the marker 22-2 is present within the field of view 14T of the target camera 10T.

[0062] When multiple markers are used at the same time, it may be possible to detect identification information indicating the marker 22. As the marker 22, for example, AprilTag developed by The APRIL Robotics Laboratory at the University of Michigan may be used.

[0063] Fig. 7 is a diagram showing an example of an AprilTag. Three AprilTags are shown as examples in Fig. 7. The position and orientation of each AprilTag in three-dimensional space, as well as its identification information, can be visually measured.

[0064] 5 and 6, a homogeneous transformation matrix T is calculated based on the difference in the position and orientation of the marker 22 in each camera coordinate system 12 measured from the images captured by each camera 10. The position and orientation of the marker 22 are expressed, for example, by a six-dimensional vector of position coordinates (three dimensions) and orientation angle (three dimensions).

[0065] The rough calibration process includes a process of determining the correspondence (homogeneous transformation matrix T) between the camera coordinate systems 12 based on the difference between the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 with the reference camera 10C and the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 with the target camera 10T.

[0066] The position and orientation of the marker 22 can also be defined as a marker coordinate system 26 based on the surface (effective surface 24) on which the marker 22 is arranged. In the following explanation, a process for determining the correspondence relationship (homogeneous transformation matrix T) between the camera coordinate systems 12 using the marker coordinate system 26 will be described.

[0067] FIG. 8 is a diagram for explaining the process of calculating a homogeneous transformation matrix in the rough calibration process shown in FIGS.

[0068] A homogeneous transformation matrix (rotation and translation) from a CAD canonical coordinate system 21, which is set in advance based on CAD information, to the markers 22 arranged on each surface of the object 20 is calculated in advance. By using the CAD canonical coordinate system 21, it is possible to facilitate the handling of geometric calculations for the rotation and translation of the markers 22.

[0069] For example, as shown in FIG. 8A, a homogeneous transformation matrix T from the CAD canonical coordinate system 21 to the marker coordinate system 26-1 of the marker 22-1 is ＣＡＤ ｍｒｋ１ As shown in FIG. 8B, a homogeneous transformation matrix T ＣＡＤ ｍｒｋ２ is calculated.

[0070] A homogeneous transformation matrix T from the camera coordinate system 12C of the reference camera 10C to the camera coordinate system 12T of the target camera 10T. Ｃ Ｔ Is T Ｃ ｍｒｋＣ ×T ｍｒｋＣ ｍｒｋＴ × (T Ｔ ｍｒｋＴ ) －１ It is calculated as follows.

[0071] Homogeneous transformation matrix T Ｃ ｍｒｋＣ corresponds to the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 with the reference camera 10C. Ｔ ｍｒｋＴ corresponds to the position and orientation of the marker 22 calculated based on the image obtained by capturing an image of the object 20 by the target camera 10T.

[0072] Homogeneous transformation matrix T ｍｒｋＣ ｍｒｋＴ corresponds to the rotation and translation from the marker 22 present in the field of view 14C of the reference camera 10C to the marker 22 present in the field of view 14T of the target camera 10T.

[0073] As shown in FIG. 8A, when the same marker 22 exists within the field of view of both the reference camera 10C and the target camera 10T, the homogeneous transformation matrix T ｍｒｋＴＣ ｍｒｋＴ corresponds to a rotation and a translation to the same marker. That is, the homogeneous transformation matrix T ｍｒｋＣ ｍｒｋＴ = E (unit matrix).

[0074] As a result, the homogeneous transformation matrix T Ｃ Ｔ =T Ｃ ｍｒｋ１ × (T Ｔ ｍｒｋ２ ) －１ On the other hand, as shown in FIG. 8B, when the marker 22 present in the field of view 14C of the reference camera 10C is different from the marker 22 present in the field of view 14T of the target camera 10T, the homogeneous transformation matrix T ｍｒｋＣ ｍｒｋＴ is not E (the identity matrix) but is determined as follows:

[0075] Fig. 9 is a diagram showing an example of a marker coordinate system 26 for the markers 22 arranged on each surface of the object 20. Referring to Fig. 9, when the markers 22 have identification information such as AprilTag, the identification information of the markers 22 appearing in the captured image is acquired.

[0076] For each piece of identification information of the marker 22, a homogeneous transformation matrix from the CAD canonical coordinate system 21 to the marker 22 is calculated in advance.

[0077] 10 is a diagram showing an example of the relationship between rotation and translation between markers based on the CAD canonical coordinate system. Referring to FIG. 10, information indicating the amount of rotation and translation is prepared in advance in association with the identification information (ID: 0 to ID: 5) of the marker 22.

[0078] The rotation amount and translation amount shown in Fig. 10 are values ​​calculated based on the CAD canonical coordinate system 21 of the marker 22 with ID: 0. The information shown in Fig. 10 may be stored in the information processing device 100 as a homogeneous transformation matrix table.

[0079] For example, if the identification information (ID) of the marker 22 acquired based on an image obtained by capturing an image of the object 20 with the reference camera 10C is “1” and the identification information (ID) of the marker 22 acquired based on an image obtained by capturing an image of the object 20 with the target camera 10T is “2”, then the homogeneous transformation matrix T ｍｒｋＣ ｍｒｋＴ =T ｍｒｋ１ ｍｒｋ２ = (T ＣＡＤ ｍｒｋ１ ) －１ ×T ＣＡＤ ｍｒｋ２ It is calculated as follows.

[0080] That is, in the homogeneous transformation matrix table shown in FIG. 10, the homogeneous transformation matrix table T ＣＡＤ ｍｒｋｉ and a homogeneous transformation matrix table T corresponding to the identification information of the marker 22 present within the field of view 14T of the target camera 10T. ＣＡＤ ｍｒｋｉ and are selected, and the homogeneous transformation matrix T ｍｒｋＴＣ ｍｒｋＴ is calculated.

[0081] The calculated homogeneous transformation matrix T ｍｒｋＴＣ ｍｒｋＴ Using the homogeneous transformation matrix T Ｃ Ｔ =T Ｃ ｍｒｋＣ ×T ｍｒｋＣ ｍｒｋＴ × (T Ｔ ｍｒｋＴ ) －１ is calculated.

[0082] In this way, in the rough calibration process, the object 20 on which one or more markers 22 are arranged is imaged by a plurality of cameras, and a homogeneous transformation matrix, which is an external parameter of the camera, is calculated.

[0083] (d2: Method using point cloud) As shown in Fig. 5 above, a case is assumed in which an object 20 exists in both the field of view 14C of the reference camera 10C and the field of view 14T of the target camera 10T. After the object 20 to be used in the rough calibration process is determined, a point cloud indicating the outline of the object 20 is extracted from each of the images captured by the reference camera 10C and the target camera 10T.

[0084] By combining the FPFH (Fast Point Feature Histogram) method and the ICP (Iterative Closest Point) method, the information processing device 100 determines the correspondence between the camera coordinate system 12C of the reference camera 10C and the camera coordinate system 12T of the target camera 10T based on the point clouds (representing the common object 20) extracted from each image. That is, the information processing device 100 determines the homogeneous transformation matrix T of the camera coordinate system 12T from the camera coordinate system 12C of the reference camera 10C to the camera coordinate system 12T of the target camera 10T. Ｃ Ｔ Estimate.

[0085] By the above processing, the correspondence relationship (homogeneous transformation matrix T) between the camera coordinate systems 12 is calculated.

[0086] <E. ROI Point Cloud Extraction Processing (Step S2)> Next, the ROI point cloud extraction processing (Step S2) will be described.

[0087] As described above, in the ROI point cloud extraction process, a first image obtained by capturing an image of the object 20 on which the markers 22 are placed using the reference camera 10C and a second image obtained by capturing an image of the object 20 using the target camera 10T are converted into a composite image. Note that the two images may be a collection of point clouds. Then, an ROI for extracting a point cloud to be used in the identification process is determined, and a point cloud included in the ROI (ROI point cloud) is extracted from the composite image.

[0088] As shown in FIG. 5, two images are obtained by capturing images of the object 20 on which the marker 22 is placed using the reference camera 10C and the target camera 10T. The two images obtained are based on the correspondence relationship (homogeneous transformation matrix T Ｃ Ｔ ) are synthesized on the same coordinate system (camera coordinate system 12C). The ROI is determined and an ROI point cloud is extracted for the synthesized image generated by such a transformation.

[0089] As shown in FIG. 6, even if a common marker 22 does not exist within the field of view 14C of the reference camera 10C and the field of view 14T of the target camera 10T, the homogeneous transformation matrix T Ｃ Ｔ The ROI point cloud extraction process can be performed using the above.

[0090] 11A and 11B are diagrams illustrating an example of ROI point cloud extraction processing according to this embodiment. Referring to Fig. 11A, once the positions and orientations of markers 22 of object 20 in three-dimensional space are measured, a marker coordinate system 26 is determined, with the surface (effective surface 24) on which markers 22 are arranged as a reference. The origin of marker coordinate system 26 is set to a predetermined position (e.g., the center position) of marker 22.

[0091] 11(B), a predetermined region is determined as ROI 50, with the origin of the determined marker coordinate system 26 as its center. ROI 50 may be a hexahedron of any size. The size of ROI 50 may be a predetermined fixed value, or may be a variable value that changes according to any condition.

[0092] 11(C), a point cloud (ROI point cloud 56) included in the determined ROI 50 is extracted. The ROI point cloud 56 may be expressed, for example, in a 2.5-dimensional data format. The 2.5-dimensional data format adds depth information (distance information) to two-dimensional image data.

[0093] In this way, the ROI point cloud extraction process includes a process of extracting, from the point clouds included in the composite image, an ROI point cloud 56 (corresponding to the second point cloud) included in the ROI 50 (corresponding to the second region) determined based on the markers 22. The ROI 50 may be determined based on the surface (effective surface 24) on which the markers 22 of the object 20 are arranged. An identification process is performed based on the extracted ROI point cloud 56.

[0094] For ease of explanation, an example of an object 20 on which one marker 22 is placed is shown, but if multiple markers 22 are placed on the object 20, an ROI may be determined for each marker 22 (effective surface 24) and an ROI point cloud may be extracted.

[0095] <F. CAD-Derived Point Cloud Generation Process (Step S3)> Next, the CAD-derived point cloud generation process (Step S3) will be described.

[0096] As described above, in the CAD-derived point cloud generation process, a CAD-derived point cloud is generated. Fig. 12 is a diagram for explaining an example of the CAD-derived point cloud generation process according to the present embodiment.

[0097] 12 , information on the object 20 used in the rough calibration process (step S1) and the ROI point cloud extraction process (step S2) is prepared. The information on the object 20 may be design information or actual measurement information. CAD information 30 of the object 20 may be used as an example of design information. The design information is not limited to the term "CAD" and may be data in any format. An example of actual measurement information may be information obtained by optically measuring the object 20. The actual measurement information may be information measured by any method and may be data in any format.

[0098] A three-dimensional point cloud 32 representing the shape of the object 20 is generated based on the CAD information 30. Of the generated three-dimensional point cloud 32, a point cloud existing within a target region 34 corresponding to the effective surface 24 of the marker 22 is extracted as a CAD-derived point cloud 36. The CAD-derived point cloud 36 does not need to include all points of the object 20, but only needs to include at least a point cloud in the vicinity of the effective surface 24 of the marker 22.

[0099] The three-dimensional point cloud 32 is represented, for example, in a three-dimensional data format. To simplify the identification process, the point cloud present in the region of interest 34 may be converted from the three-dimensional data format to a two-and-a-half-dimensional data format.

[0100] If the CAD-derived point cloud 36 is in a 2.5-dimensional data format, the ROI point cloud 56 and the CAD-derived point cloud 36 can be associated with each other in the 2.5-dimensional data format.

[0101] In this way, the CAD-derived point cloud generation process includes a process of extracting, from a point cloud generated based on information about the object 20 (e.g., CAD information 30), a CAD-derived point cloud 36 (corresponding to a first point cloud) included in a target region 34 (corresponding to a first region) determined based on the markers 22. The CAD-derived point cloud 36 may be determined based on a surface (effective surface 24) of the object 20 on which the markers 22 are arranged.

[0102] For ease of explanation, an example of an object 20 on which one marker 22 is placed is shown, but if multiple markers 22 are placed on the object 20, a CAD-derived point cloud 36 may be generated for each marker 22 (effective surface 24).

[0103] <G. Identification Processing (Step S4)> Next, the identification processing (Step S4) will be described.

[0104] As described above, the identification process determines the correspondence between the ROI point cloud 56 and the CAD-derived point cloud 36. The identification process also calculates the geometry of key point pairs in the CAD-derived point cloud and the geometry of key point pairs in the ROI point cloud.

[0105] In the identification process, key points in the CAD-derived point cloud 36 are selected. Key points in the ROI point cloud 56 that correspond to the selected key points in the CAD-derived point cloud 36 are searched for. The key points in the CAD-derived point cloud 36 are determined in units of "key point pairs," each consisting of two key points. In the process described below, the shape dimensions of the key point pairs are calculated. That is, the key point pairs are the targets for calculating the shape dimensions.

[0106] 13 is a diagram illustrating an example of key point search in the identification process according to this embodiment. Referring to FIG. 13, key point pairs 40-1 and 40-2 (hereinafter sometimes collectively referred to as "key point pairs 40") are selected in the CAD-derived point cloud 36. The key point pairs 40 (and key points 42) may be selected based on the shape characteristics of the object 20 on which the markers 22 are placed.

[0107] The key point pair 40 (and the key point 42) may be automatically selected according to one or more coordinates preset for the CAD information. The key point pair 40 (and the key point 42) may also be selected in response to a user instruction on the CAD-derived point cloud 36. In this case, the information processing device 100 may provide a user interface screen for the user to select the key point pair 40 (and the key point 42).

[0108] 13, key point pair 40-1 is a set of key points 42-1 and 42-2. Key point pair 40-2 is a set of key points 42-3 and 42-4. Hereinafter, key points 42-1 to 42-4 may be collectively referred to as "key points 42."

[0109] Key points 62-1 to 62-4 (hereinafter sometimes collectively referred to as "key points 62") in the ROI point cloud 56 are searched for, respectively, corresponding to key points 42-1 to 42-4 in the CAD-derived point cloud 36. That is, the identification process includes a process of searching for a set of key points 62 (corresponding to a second point pair) in the ROI point cloud 56 that corresponds to a key point pair 40 (corresponding to a first point pair) selected from the CAD-derived point cloud 36 (first point cloud).

[0110] Below, several exemplary key point search methods for searching for key points 62 in the ROI point cloud 56 that correspond to key points 42 in the CAD-derived point cloud 36 are described.

[0111] (g1: Minimum Distance Method) First, the minimum distance method, which is an example of the key point search method, will be described.

[0112] 14 is a diagram illustrating a method for searching for key points using the minimum distance method. With reference to FIG. 14 , coordinates corresponding to key points 42 in the CAD-derived point cloud 36 are provided for the ROI point cloud 56. The distance from the coordinates corresponding to the key points 42 in the CAD-derived point cloud 36 to each point is calculated. The point that is the smallest distance from the coordinates corresponding to the key points 42 in the CAD-derived point cloud 36 is estimated as the key point 62.

[0113] (g2: Center of Gravity Method) Next, the center of gravity method, which is another example of the key point search method, will be described.

[0114] 15 is a diagram illustrating a method for searching for key points using the centroid method. Referring to FIG. 15 , coordinates corresponding to key points 42 in the CAD-derived point cloud 36 are assigned to the ROI point cloud 56. A plurality of points within a predetermined radius R are determined from the coordinates corresponding to key points 42 in the CAD-derived point cloud 36. The centroid of the plurality of points within radius R is estimated as key point 62.

[0115] In this way, the centroid method involves determining coordinates in the ROI point cloud 56 (second point cloud) corresponding to each point of the key point pair 40 (first point pair) in the CAD-derived point cloud 36, and determining a key point 62 of the key point pair 60 (second point pair) based on the centroid of one or more points that exist within a predetermined distance from the determined coordinates. The same applies to various methods based on the centroid method described below.

[0116] (g3: Average Center of Gravity Method) Next, the average center of gravity method, which is yet another example of the key point search method, will be described.

[0117] 16 is a diagram illustrating a method for searching for key points using the average center of gravity method. Similar to the above-described center of gravity method, multiple points within a predetermined radius R from the coordinates corresponding to the key point 42 in the CAD-derived point cloud 36 are selected. A center of gravity 64 of the multiple points within the radius R is calculated.

[0118] When multiple samples of the ROI point cloud 56 can be prepared by capturing images multiple times with the camera 10, the centroid 64 can be calculated for each ROI point cloud 56. For example, multiple centroids 64-1 to 64-6 are calculated, and then the average of the multiple centroids 64-1 to 64-6 is estimated as the key point 62.

[0119] (g4: Average Center of Gravity Method + M-Estimation) Next, a method in which M-estimation is added to the average center of gravity method, which is yet another example of the key point search method, will be described.

[0120] 17 is a diagram illustrating a method for searching for key points using the average centroid method and M-estimation. Referring to FIG. 17 , when multiple samples of the ROI point cloud 56 can be prepared by using the camera 10 to capture images multiple times, a centroid 64 can be calculated for each ROI point cloud 56, similar to the above-described average centroid method. Once the multiple centroids 64 are calculated, the average of the calculated multiple centroids 64 is calculated. Based on the calculated average, the deviation and standard deviation σ of each of the multiple centroids 64 are calculated.

[0121] Among the multiple centroids 64, a centroid 64 having a deviation exceeding the standard deviation σ is determined to be an outlier, and the other centroids 64 are determined to be inliers. In the example shown in Fig. 17, three centroids 64-1, 64-3, and 64-4 are determined to be inliers.

[0122] One of the centroids 64 determined as the optimum value is estimated as the key point 62. Alternatively, the average of the centroids 64 determined as the optimum value is estimated as the key point 62.

[0123] (g5: Mean center of gravity method + Random sample consensus (RANSAC) method) Next, a method in which the RANSAC method is added to the mean center of gravity method, which is yet another example of the key point search method, will be described.

[0124] 18 and 19 are diagrams for explaining the method of searching for key points using the average centroid method and the RANSAC method. Referring to Fig. 18, a plurality of centroids 64 are calculated from a plurality of samples of the ROI point cloud 56. The plurality of centroids 64 are classified into outliers and inliers.

[0125] In sample set 1, three centroids 64-5, 64-3, and 64-2 are determined to be correct values, and the remaining three centroids 64-4, 64-1, and 64-4 are determined to be outliers. The average of the centroids determined to be correct values ​​and the average of the centroids determined to be outliers are determined as RANSAC key points. The difference between the RANSAC key points of the correct values ​​and the RANSAC key points of the outliers is calculated as the error for key points 42 in CAD-derived point cloud 36.

[0126] Similarly, in sample set 2, three centroids 64-1, 64-5, and 64-6 are determined to be correct values, and the remaining three centroids 64-3, 64-2, and 64-4 are determined to be outliers. The difference between the correct RANSAC key points and the outlier RANSAC key points is calculated as the error for key point 42 in CAD-derived point cloud 36.

[0127] 19 , the error relative to the key point 42 in the CAD-derived point cloud 36 is calculated for a plurality of sample groups, and the sample group with the smallest error is determined as the optimal sample group. The average of the centroids of the optimal sample group is estimated as the key point 62.

[0128] (g5: Summary) The key point search method is not limited to the above-mentioned methods, and any method can be used. However, since the key point search method is an important factor that affects the accuracy evaluation process described below, multiple key point search methods may be evaluated and then one or more appropriate key point search methods may be determined.

[0129] Using a key point search method, key points 62 in the ROI point cloud 56 that correspond to key points 42 in the CAD-derived point cloud 36 are found, and the geometry is calculated.

[0130] 20 is a diagram for explaining an example of geometry calculation in the identification process according to the present embodiment. Referring to FIG. 20, key points 62-1 to 62-4 in ROI point cloud 56 corresponding to key points 42-1 to 42-4 in CAD-derived point cloud 36, respectively, are searched for.

[0131] Based on key points 62-1 to 62-4 in the ROI point cloud 56, a key point pair 60-1 corresponding to the key point pair 40-1, 40-2 in the CAD-derived point cloud 36 is determined, and a key point pair 60-2 corresponding to the key point pair 40-3, 40-4 in the CAD-derived point cloud 36 is determined.

[0132] The geometry dimensions of each of the key point pairs 40-1 and 40-2 in the CAD-derived point cloud 36 are calculated. The geometry dimensions of each of the key point pairs 60-1 and 60-2 (hereinafter sometimes collectively referred to as "key point pairs 60") in the ROI point cloud 56 are also calculated. The geometry dimensions may be, for example, the length of the key point pair (one-dimensional: scalar quantity), or the length and angle of the key point pair (vector quantity). The length and direction of the geometry dimensions may be expressed using a component indicating the length and a rotational component indicating the angle.

[0133] <H. Processing Examples> Next, processing examples of the ROI point cloud extraction processing, the CAD-derived point cloud generation processing, and the identification processing will be described.

[0134] 21 to 23 are diagrams showing an example of information processing according to this embodiment. After the rough calibration process, a homogeneous transformation matrix T Ｃ Ｔ is calculated.

[0135] Referring to FIG. 21(A), the homogeneous transformation matrix T Ｃ ＴThe marker coordinate system 26 in the camera coordinate system 12C of the target camera 10T is coordinate-transformed into the marker coordinate system 26 in the camera coordinate system 12C of the reference camera 10C using the homogeneous transformation matrix T Ｃ Ｔ The point clouds are aggregated after matching the criteria using coordinate transformation using

[0136] 21B, an ROI is determined based on the effective surface 24, and a point cloud included in the ROI (ROI point cloud 56) is extracted. That is, a point cloud of the surface layer in the vicinity of the effective surface 24 is extracted.

[0137] 21C, the ROI point cloud 56 is then coordinate-transformed into the CAD canonical coordinate system 21. This coordinate transformation results in a consistent standard for calculating the shape dimensions in the identification process.

[0138] 22, key points 42 and key point pairs 40 are selected based on CAD information of the object 20 on which the markers 22 are placed. The key points 42 may be selected based on the geometric features of the object 20.

[0139] Referring to FIG. 23A, coordinates corresponding to key points 42 in CAD-derived point cloud 36 are assigned to ROI point cloud 56 .

[0140] Referring to FIG. 23B, a key point 62 in ROI point cloud 56 that corresponds to key point 42 in CAD-derived point cloud 36 is searched for by the key point search method.

[0141] <I. Reference Object> Next, an example of the shape of the reference object (object 20) will be described.

[0142] As mentioned above, the object 20 has an active surface 24 on which the markers 22 are placed, and may have geometric features for selecting the key points 42. Some examples of shapes that can serve as reference objects are given below.

[0143] Fig. 24 is a schematic diagram showing an example of the configuration of reference object 20A used in information processing system 1 according to this embodiment. Referring to Fig. 24, reference object 20A is based on a cube. Markers 22 are placed on each face of reference object 20A (six faces in total). Fig. 24 shows markers 22-1 to 22-3 positioned on only three faces, but markers 22 are also placed on each of the remaining faces not shown.

[0144] The reference object 20A has portions cut out that correspond to the four vertices of a cube. More specifically, cube-shaped cutouts 28-1 and 28-2 are provided at a pair of opposing vertices on the top surface of the reference object 20A. Chamfered portions 29-1 and 29-2 are provided at another pair of opposing vertices on the top surface of the reference object 20A.

[0145] The notches 28-1 and 28-2 and the chamfers 29-1 and 29-2 are geometric features of the reference object 20A and can be used as criteria for selecting the key points 42 and key point pairs 40.

[0146] Fig. 25 is a schematic diagram showing an example of the configuration of reference object 20B used in information processing system 1 according to the present embodiment. Referring to Fig. 25, reference object 20B includes a main quadrangular pyramid 250 and quadrangular pyramids 252 and 254 whose bases coincide with one face of quadrangular pyramid 250. Markers may be placed on the faces of quadrangular pyramid 250 where quadrangular pyramids 252 and 254 are not present.

[0147] The vertices of the pyramid 250 and the pyramids 252 and 254 are geometric features of the reference object 20B and can be used as criteria for selecting the key points 42 and key point pairs 40.

[0148] FIG. 26 is a schematic diagram showing an example of the configuration of reference objects 20C and 20D used in information processing system 1 according to the present embodiment.

[0149] 26A, the reference object 20C is based on a cube, and markers 22-1 to 22-3 are arranged on three surfaces of the reference object 20C, respectively.

[0150] The reference object 20C has cutouts in the portions corresponding to the sides of its top surface. More specifically, cutouts 28-3 to 28-6 are provided in the portions corresponding to the sides of the top surface of the reference object 20C.

[0151] The cutouts 28-3 to 28-6 are formed so that the cutout sections and lengths are different from one another, that is, the cutouts 28-3 to 28-6 have shapes that can be distinguished from one another.

[0152] The notches 28-3 to 28-6 are geometric features of the reference object 20C and can serve as criteria for selecting the key points 42 and key point pairs 40.

[0153] 26B, the reference object 20D is based on a cube, and markers 22-1 to 22-3 are arranged on three surfaces of the reference object 20D.

[0154] The reference object 20D has two sides partially cut out. More specifically, the two sides of the reference object 20D are provided with cutouts 28-7 and 28-8, respectively.

[0155] The cross section of the cutout 28-7 is rectangular, and the cross section of the cutout 28-8 is triangular. The cutouts 28-7 and 28-8 may have shapes such that the cross-sectional area decreases toward the inside of the reference object 20D. In this way, the cross-sectional shapes of the cutouts 28-7 and 28-8 are different, making them distinguishable from each other.

[0156] The notches 28-7 and 28-8 are geometric features of the reference object 20D and can be used as criteria for selecting the key points 42 and key point pairs 40.

[0157] 27 is a schematic diagram showing an example of the configuration of reference object 20E used in information processing system 1 according to the present embodiment. Referring to Fig. 27, reference object 20E includes a main pillar 260 and five flat plates 261 to 265 on which markers 22 can be arranged. Flat plate 262 is supported by main pillar 260. Flat plates 261, 263, 264, and 265 are supported by support members 271, 273, 274, and 275 connected to main pillar 260.

[0158] The arrangement of the planar plates 261 to 265 is a geometric feature of the reference object 20E and can be the basis for selecting the key points 42 and key point pairs 40.

[0159] The reference object (object 20) may be a cube or rectangular parallelepiped without any cutouts, etc. The reference object (object 20) is not limited to the above-mentioned shapes and may have any shape.

[0160] As described above, the key points 42 and key point pairs 40 may be selected based on the geometric characteristics of the reference object. A reference object having geometric characteristics may be employed to set the key points 42 and key point pairs 40.

[0161] <J. Accuracy Evaluation Process (Step S5)> Next, the accuracy evaluation process (Step S5) will be described.

[0162] As described above, in the accuracy evaluation process, accuracy is calculated based on the geometry calculated in the identification process. That is, the accuracy evaluation process includes a process of evaluating the accuracy of the homogeneous transformation matrix T (the correspondence between the camera coordinate systems 12) based on the geometry of the key point pair 40 (first point pair) in the CAD-derived point cloud 36 and the key point pair 60 (second point pair) in the ROI point cloud 56.

[0163] The calculated accuracy includes, for example, at least one of absolute error and repeat error.

[0164] (j1: absolute error) Fig. 28 is a diagram for explaining an example of absolute error evaluation in the accuracy evaluation process according to the present embodiment. With reference to Fig. 28, for each key point pair, the difference between the shape and dimensions of key point pair 40 in CAD-derived point cloud 36 and key point pair 60 in ROI point cloud 56 is calculated. The calculated difference becomes the absolute error.

[0165] The absolute error calculated for each of the multiple key point pairs is statistically processed to calculate the absolute error in the target information processing system 1. Also, the absolute error in the target information processing system 1 is calculated by statistically processing the shape error of each key point pair calculated from the multiple samples.

[0166] For example, for key point pair p1, the length of the geometry at key point pair 40 is L ｃａｄ 1, and the length of the geometry at key point pair 60 is L ＲＯＩ If 1, the absolute error Δp1=|L ｃａｄ 1-L ＲＯＩ Similarly, for key point pair p2, the shape dimension of key point pair 40 is Lc2, and the shape dimension of key point pair 60 is L ＲＯＩ If 2, the absolute error Δp2 = |L ｃａｄ 2-L ＲＯＩ 2|The result is

[0167] Alternatively, the angle between the key point pair p1 and the key point pair p2 is θ ｃａｄ 1, and the angle at the key point pair 60 is θ ＲＯＩ If 1, the absolute error Δpθ1=|θ ｃａｄ 1-θ ＲＯＩ 1|

[0168] In this way, an absolute error is calculated as accuracy based on the difference between the geometric dimensions of key point pairs 40 in CAD-derived point cloud 36 and key point pairs 60 in ROI point cloud 56 .

[0169] (j2: Repetitive Error) Fig. 29 is a diagram for explaining an example of repetitive error evaluation in the accuracy evaluation process according to the present embodiment. With reference to Fig. 29, for a plurality of samples, the variation in the shape and dimensions of key point pairs 60 in ROI point cloud 56 is calculated for each key point pair. The calculated variation (e.g., standard deviation) becomes the repetitive error.

[0170] For example, for key point pair p1, the geometry of sample 1 is L ＲＯＩ 11, and the shape dimension of sample 2 is L ＲＯＩ If 12, then L ＲＯＩ 11 and L ＲＯＩ The error with 12 is a repeat error.

[0171] Similarly, for key point pair p2, the geometric dimensions of sample 1 are L ＲＯＩ 21, and the shape dimension of sample 2 is L ＲＯＩ If 22, then L ＲＯＩ 21 and L ＲＯＩ The error with 22 is a repeat error.

[0172] Alternatively, for key point pair p1, the angle at sample 1 is θ ｃａｄ 11, and the angle in sample 2 is θ ＲＯＩ If θ is 12, ＲＯＩ 21 and θ ＲＯＩ The error with 22 is a repeat error.

[0173] If the repeat error is calculated for more than two samples, the standard deviation of the geometry of the same key point pair is calculated as the repeat error.

[0174] In this way, the accuracy evaluation process includes a process of calculating the variation in the geometric dimension error by repeatedly calculating the geometric dimension error of the key point pairs 60 (second point pairs) in the ROI point cloud 56.

[0175] (j3: Others) For each key point pair, the shape dimensions of the key point pair 40 in the CAD-derived point cloud 36 and the shape dimensions of the key point pair 60 in the ROI point cloud 56 are obtained, so any accuracy evaluation process can be performed using at least one of the two types of shape dimensions.

[0176] <K. Output of Accuracy Evaluation Results (Step S6)> Next, output of accuracy evaluation results (Step S6) will be described.

[0177] 30 is a schematic diagram showing an example of output of the accuracy evaluation result according to the present embodiment. Referring to Fig. 30, an output screen 200 of the accuracy evaluation result includes information 210 indicating the error in the length of the geometric dimension and information 220 indicating the error in the angle of the geometric dimension.

[0178] The information 210 includes statistical information 212 and a histogram 214 for length errors. The histogram 214 indicates the number (frequency) of key point pairs for each magnitude of length error. The information 220 includes statistical information 222 and a histogram 224 for angle errors. The histogram 224 indicates the number (frequency) of key point pairs for each magnitude of angle error.

[0179] In this way, the information processing device 100 may output a histogram 214 for length errors and a histogram 224 for angle errors.

[0180] In the histogram, the more the distribution is biased to the left, the better the accuracy can be determined. As shown in Figure 30, by referring to the distribution shape of the histogram for the length and angle errors, the variation in the errors can also be evaluated.

[0181] <L. Fine-tuning Process (Step S7)> Next, the fine-tuning process (Step S7) will be described. If the calculated accuracy (absolute error and repeat error) does not satisfy a predetermined condition, a process including the fine-tuning process may be performed.

[0182] The fine-tuning process is a process of optimizing the homogeneous transformation matrix T so as to minimize errors in the geometric dimensions calculated by the identification process. That is, the fine-tuning process includes a process of optimizing the homogeneous transformation matrix T (the correspondence between the camera coordinate systems 12) so as to reduce the difference in the geometric dimensions between the key point pair 40 (first point pair) in the CAD-derived point cloud 36 and the key point pair 60 (second point pair) in the ROI point cloud 56. The optimization of the homogeneous transformation matrix T can be reduced to a nonlinear programming problem for minimizing (or maximizing) an evaluation function using the geometric dimensions.

[0183] In the fine-tuning process, an evaluation function indicating the cost is defined, and each parameter of the homogeneous transformation matrix T is searched for so as to minimize the cost indicated by the evaluation function.

[0184] More specifically, the homogeneous transformation matrix T ＊ (R ＊ , t ＊ ) the following evaluation function can be used:

[0185]

[0186] The evaluation function includes the sum of squares of the absolute errors between the geometry of key point pairs 40 in CAD-derived point cloud 36 and the geometry of key point pairs 60 in ROI point cloud 56 for all key point pairs. In other words, minimizing the evaluation function means minimizing the sum of squares of the absolute errors between the geometry of key point pairs 40 in CAD-derived point cloud 36 and the geometry of key point pairs 60 in ROI point cloud 56.

[0187] The above evaluation function may include a penalty term such as:

[0188]

[0189] FIG. 31 is a diagram for explaining an example of the penalty term in the fine tuning process according to the present embodiment.

[0190] 31 , depending on the accuracy of the rough calibration process, fluctuations in the point cloud, and the like, there may be cases where a predetermined number of points do not exist within a predetermined radius R from the coordinates corresponding to the key point 42 in the CAD-derived point cloud 36. In the example on the left side (real world) of FIG.

[0191] In such a case, the radius R may be increased stepwise until a predetermined number of points are found. However, the increase in the search radius from the reference radius R to radius R' may be reflected in the penalty term. For example, the penalty term may be a value obtained by multiplying |radius R' - radius R| by a coefficient.

[0192] Furthermore, an evaluation function may be employed that uses an L2 regularization term as a penalty term in addition to the length error and angle error of the shape dimensions. For example, the following evaluation function may be used.

[0193]

[0194] As described above, the evaluation function used in the fine tuning process may be any function as long as it uses the calculation results of the geometry of key point pairs obtained in the identification process.

[0195] An example of the effect of the fine-tuning process will now be described. FIG. 32 is a diagram showing an example of how the length error changes as a result of the fine-tuning process according to this embodiment. Referring to FIG. 32, it can be seen that the histogram before the fine-tuning process exhibited a multi-peak distribution, whereas after the fine-tuning process, the histogram exhibits an exponential distribution. In the histogram, the more the distribution is biased to the left, the better the accuracy. In this way, it can be seen that the fine-tuning process brings the error distribution closer to an ideal one.

[0196] 33 is a diagram showing an example of how the angle error changes due to the fine-tuning process according to this embodiment. Referring to FIG. 33, it can be seen that the histogram before the fine-tuning process deviated from the exponential distribution, whereas after the fine-tuning process, the histogram shows an exponential distribution. For the exponential distribution p(x)=λexp(−λ), the parameter λ becomes larger after the fine-tuning process. In other words, the histogram shows a sharper exponential distribution.

[0197] In this way, the fine tuning process can reduce errors in geometric dimensions.

[0198] <M. Application Examples of Accuracy Evaluation> Next, several application examples of accuracy evaluation of the information processing method according to the present embodiment will be described. The output of the accuracy evaluation result described above is based on the results of execution of the rough calibration process and the fine tuning process, but the information processing method according to the present embodiment can also be used for managing changes in accuracy over time and for periodic accuracy management.

[0199] (m1: Accuracy Management Based on Histograms) As shown in Fig. 32 above, accuracy is reflected in the distribution shape of the histograms. By comparing the distribution shapes of the histograms using a predetermined algorithm, changes in accuracy, etc. can be quantitatively calculated.

[0200] Examples of algorithms for comparing the distribution shapes of histograms include a method using a histogram intersection kernel and a method using a histogram Euclidean distance.

[0201] For example, a facility maintenance person may generate and save a histogram of errors before starting operation, and then regenerate the histogram of errors during daily inspections, etc., and compare it with the saved histogram. Based on changes in the distribution shape of the histogram, the facility maintenance person can determine whether accuracy has decreased.

[0202] Observation based on changes in the distribution shape of the histogram of such errors may be used as equipment maintenance work.

[0203] Alternatively, accuracy trend management may be performed based on a parameter λ estimated by approximating a histogram of errors using an exponential distribution p(x)=λexp(−λ).

[0204] For example, before starting operation, a facility maintainer generates a histogram of the error, approximates the generated histogram with an exponential distribution, and saves the determined parameter λ. During routine inspections, the facility maintainer regenerates the histogram of the error, approximates the generated histogram with an exponential distribution, and calculates the parameter λ. The facility maintainer can manage the trend of accuracy based on changes in the value of the parameter λ.

[0205] For example, the accuracy trend management is performed using the margin I calculated according to the following formula: λｍａｒｇｉｎ may also be used.

[0206]

[0207] where λ Ｅｘｐ＿ｃｒｉｔｅ denotes the reference value of the parameter λ, and λ Ｅｘｐ＿ｃｕｒｒ denotes the value of the parameter λ calculated from the histogram to be evaluated.

[0208] If the parameter λ does not change from the reference value, the margin I λｍａｒｇｉｎ = 0, and if it is in the direction of improvement, the margin I λｍａｒｇｉｎ >0, and the deterioration direction is margin I λｍａｒｇｉｎ <0. Margin I λｍａｒｇｉｎ The absolute value of indicates the degree of trend (improvement or deterioration).

[0209] The parameter λ calculated from such a histogram of errors can also be used to trend the accuracy.

[0210] (m2: Accuracy Control Based on Point Cloud Thickness) After the fine tuning process, the thickness of the point cloud (point cloud thickness) may be used as a method for checking the accuracy in detail.

[0211] FIG. 34 is a diagram for explaining accuracy confirmation using point cloud thickness in the information processing method according to the present embodiment.

[0212] Referring to Figure 34 (A), after the rough calibration process or the fine tuning process is performed, a composite point cloud 270 is generated by combining the point cloud extracted from the image captured by the reference camera 10C and the point cloud extracted from the image captured by the target camera 10T.

[0213] For the composite point cloud 270, a point cloud thickness is calculated from each of a region 282 corresponding to the composite seam and a region 284 corresponding to the area other than the composite seam.

[0214] The point cloud thickness indicates the degree of dispersion of the point cloud within a region, i.e., the smaller the calculated point cloud thickness, the higher the accuracy.

[0215] The point cloud thickness of the region 284 corresponding to the area other than the synthesis seam depends on the accuracy of the camera 10 (sensor) itself. Ideally, the point cloud thickness of the region 282 corresponding to the synthesis seam is preferably the same as the point cloud thickness of the region 284 corresponding to the area other than the synthesis seam. In other words, it is possible to quantitatively evaluate how appropriately the rough calibration process or the fine-tuning process was performed based on how close the point cloud thickness of the region 282 is to the point cloud thickness of the region 284.

[0216] Referring to FIG. 34(B), a reference plane Q is estimated from the point cloud included in the specified region. The reference plane Q is determined based on the coordinates of the point cloud included in the specified region, for example, by the least squares method. The Euclidean distance L(i) between a point P(i) included in the point cloud and the reference plane Q is calculated. Finally, the variance of the Euclidean distance L(i) is determined as the point cloud thickness. That is, the point cloud thickness is calculated as Thickness=variance(L(i:1→n)).

[0217] A larger value of the point cloud thickness indicates that the point cloud is more disorganized relative to the reference plane Q. The synthesis accuracy is calculated by comparing the point cloud thickness between an area 282 corresponding to the synthesis seam and an area 284 corresponding to the non-synthesis seam.

[0218] <N. Spatial Distribution of Accuracy> As described above, the identification process is based on a point cloud extracted from the vicinity of the effective surface 24 of the marker 22. Therefore, the calculated accuracy is for the vicinity of the effective surface 24. If the object 20 on which the marker 22 is arranged is placed at multiple positions in the space SP and the accuracy at each position is calculated, the spatial distribution of accuracy in the space SP can be obtained.

[0219] By using the spatial distribution of accuracy in the space SP, the uniformity (flatness) of accuracy in the space SP can be evaluated. Furthermore, the spatial distribution of accuracy in the space SP may be visualized.

[0220] 35 is a diagram for explaining a method of calculating the accuracy spatial distribution according to the present embodiment. Referring to Fig. 35, a reference camera 10C and a target camera 10T are arranged in a space SP.

[0221] While maintaining the positions of the reference camera 10C and the target camera 10T, the object 20 on which the marker 22 is placed is sequentially placed at different measurement positions Pos1, Pos2, Pos3, .... At each measurement position, the accuracy is calculated by the process described above.

[0222] 36 is a flowchart showing an example of the processing procedure of the method for calculating the spatial distribution of precision according to the present embodiment. Each step shown in FIG. 36 may be realized by the processor 102 of the information processing device 100 executing the information processing program 124.

[0223] The information processing device 100 executes rough calibration processing (step S1). At this time, the object 20 on which the markers 22 are placed is placed at an arbitrary measurement position. Step S3 may be executed in advance, independently of the processing of the other steps.

[0224] The information processing device 100 executes an ROI point cloud extraction process (step S2). The information processing device 100 executes a CAD-derived point cloud generation process (step S3). Step S3 may be executed in advance, independently of the processes of the other steps. Alternatively, step S3 may be executed only the first time during a loop process including step S10.

[0225] The information processing device 100 executes an identification process (step S4). Subsequently, the information processing device 100 executes an accuracy evaluation process (step S5). The information processing device 100 stores the calculated accuracy and the coordinates obtained by converting the marker coordinate system 26 used in the accuracy calculation into the global coordinate system in association with each other (step S9).

[0226] The information processing device 100 determines whether the object 20 on which the marker 22 is placed has been moved to another measurement position (step S10). The user may give an instruction to the information processing device 100, which may cause the information processing device 100 to determine whether the object 20 has been moved to another measurement position.

[0227] If the object 20 on which the marker 22 is placed has been moved to another measurement position (YES in step S10), the processes from step S2 onwards are repeated.

[0228] If the object 20 on which the marker 22 is placed has not been moved to another measurement position (NO in step S10), the information processing device 100 calculates the accuracy spatial distribution based on a pair of the calculated accuracy and coordinates obtained by converting the marker coordinate system 26 used to calculate the accuracy into the global coordinate system (step S11).

[0229] Since it may not be possible to move the object 20 evenly within the space SP, the information processing device 100 may use a known interpolation method to calculate the accuracy of each measurement position within the space SP from the stored information.

[0230] The information processing device 100 visualizes and outputs the calculated spatial distribution of precision in the space SP (step S12), and the processing then ends.

[0231] In this way, the process of calculating the spatial distribution of accuracy includes a process of calculating the accuracy of the correspondence (homogeneous transformation matrix T) between the camera coordinate systems 12 based on the shape and dimensions of the key point pair 40 (first point pair) of the CAD-derived point cloud 36 and the key point pair 60 (second point pair) of the ROI point cloud 56, and a process of visualizing the accuracy of the correspondence (homogeneous transformation matrix T) between the camera coordinate systems 12 calculated for each measurement position when the object 20 is placed at multiple measurement positions in space, in relation to each measurement position.

[0232] 37 is a diagram showing an example of visualization of the spatial distribution of accuracy obtained by the method for calculating the spatial distribution of accuracy according to the present embodiment. In the spatial distribution of accuracy shown in FIG. 37, the accuracy of each measurement position is expressed by shading or color. In this way, an image may be output in which the accuracy value of the correspondence relationship (homogeneous transformation matrix T) between camera coordinate systems 12 is expressed by at least one of shading and color.

[0233] The visualized spatial distribution of accuracy can help the user recognize areas in the space SP where error variation and error bias exist. The areas where error variation and error bias exist provide useful information for optimizing the installation location of the robot, for example. Furthermore, information on the presence or absence of areas where error variation and error bias exist provides useful information for determining whether or not it is necessary to add a camera.

[0234] <O. Functional Configuration Example> Next, a functional configuration example of information processing system 1 according to the present embodiment will be described.

[0235] 38 is a schematic diagram showing an example of the functional configuration of an information processing system 1 according to the present embodiment. Each component shown in FIG. 38 may be realized by the processor 102 of the information processing device 100 executing an information processing program 124.

[0236] Referring to FIG. 38, the information processing device 100 includes, as components, a synthesis unit 150, a determination unit 152, a first point cloud generation unit 154, a second point cloud generation unit 156, a search unit 158, an evaluation unit 160, an output unit 164, and an optimization unit 166.

[0237] The synthesis unit 150 synthesizes a homogeneous transformation matrix T Ｃ Ｔ The synthesis unit 150 converts the two images captured by the reference camera 10C and the target camera 10T into a composite image using the above formula. The synthesis unit 150 executes the process of generating the composite image of the ROI point cloud extraction process (step S3).

[0238] The determination unit 152 calculates a homogeneous transformation matrix T based on the difference between the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 with the reference camera 10C and the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 with the target camera 10T. Ｃ Ｔ The determination unit 152 executes a rough calibration process (step S1).

[0239] The first point cloud generator 154 extracts the CAD-derived point cloud 36 from the point cloud generated based on the information (CAD information 30) of the object 20. The first point cloud generator 154 executes a CAD-derived point cloud generation process (step S3).

[0240] The second point cloud generation unit 156 extracts, from the point clouds included in the composite image, an ROI point cloud 56 included in the ROI 50 determined based on the marker 22. The second point cloud generation unit 156 executes the process of determining the ROI and the process of extracting the ROI point cloud in the ROI point cloud extraction process (step S3).

[0241] The search unit 158 ​​searches for key points in the ROI point cloud 56 that correspond to the selected key points in the CAD-derived point cloud 36. The search unit 158 ​​calculates the geometry of the searched key point pairs. The search unit 158 ​​calculates the geometry of the key point pairs in the CAD-derived point cloud and the geometry of the key point pairs in the ROI point cloud. The search unit 158 ​​executes an identification process (step S4).

[0242] The evaluation unit 160 calculates the accuracy based on the geometric dimensions of the key point pairs calculated by the search unit 158. The evaluation unit 160 may include a calculation unit 162 for calculating the accuracy. The evaluation unit 160 executes an accuracy evaluation process (step S5).

[0243] The output unit 164 outputs the calculation result obtained by the evaluation unit 160. The output unit 164 executes a process of outputting the accuracy evaluation result (step S6).

[0244] The output unit 164 may generate and output the accuracy spatial distribution. The output unit 164 performs the following steps: associating accuracy with coordinates (step S9), calculating the accuracy spatial distribution (step S11), and visualizing the accuracy spatial distribution (step S12).

[0245] The optimization unit 166 optimizes the homogeneous transformation matrix T so as to reduce the difference in shape and size between the key point pairs 40 of the CAD-derived point cloud 36 and the key point pairs 60 of the ROI point cloud 56. Ｃ Ｔ The optimization unit 166 executes a fine tuning process (step S7).

[0246] <P. Setting of World Coordinate System> Next, a process for setting an arbitrary world coordinate system will be described. After the correspondence between the camera coordinate systems 12 is determined by the rough calibration process or the fine tuning process, the world coordinate system may be set at an arbitrary position using the reference object 20.

[0247] 39 is a diagram for illustrating a method for setting an arbitrary world coordinate system in the information processing system according to the present embodiment. Referring to Fig. 39, cameras 10-1, 10-2, and 10-3 are arranged in a space SP.

[0248] For example, assume that the camera 10-1 is the reference camera and the cameras 10-2 and 10-3 are target cameras. A homogeneous transformation matrix T from (the camera coordinate system of) the camera 10-1 to (the camera coordinate system of) the camera 10-2 is obtained by rough calibration processing or fine tuning processing. １ ２ Similarly, a homogeneous transformation matrix T from (the camera coordinate system of) the camera 10-1 to (the camera coordinate system of) the camera 10-3 is determined. １ ３ has been decided.

[0249] First, a reference object 20 is placed so that it is included in at least a portion of the field of view of each of cameras 10-1 to 10-3. The reference object 20 is placed at a position where the user wants to set a world coordinate system. The reference object 20 has a CAD canonical coordinate system 21 that is set in advance based on CAD information. The CAD canonical coordinate system 21 of the reference object 20 is used as the world coordinate system.

[0250] The reference object 20 is imaged by the cameras 10-1 to 10-3, and based on the images, the position and orientation of the CAD canonical coordinate system 21 in the camera coordinate system of each camera 10 are calculated. The position and orientation of the CAD canonical coordinate system 21 are determined based on the surface (effective surface) of the reference object 20 on which the markers 22 are arranged.

[0251] The position and orientation of the CAD canonical coordinate system 21 calculated based on the image captured by the camera 10-1, which is the reference camera, may be set as the world coordinate system as is.

[0252] When the reference object 20 is not present within the field of view of the camera 10-1, which is the reference camera, but is present within the field of view of the camera 10-2 or 10-3, the world coordinate system may be determined using a homogeneous transformation matrix. More specifically, the position and orientation of the CAD canonical coordinate system 21 calculated based on the image captured by the camera 10-2 or 10-3 are transformed into the position and orientation of the camera coordinate system of the camera 10-1 using a homogeneous transformation matrix. In this way, the world coordinate system may be determined via the camera coordinate system of the reference camera.

[0253] When a world coordinate system is not set, the camera coordinate system of the reference camera can be used as the world coordinate system.

[0254] <Q. Modifications> In the above description, a configuration example using AprilTag, which is an example of a two-dimensional marker, has been described. However, the marker 22 is not limited to AprilTag, and any AR (Augmented Reality) marker can be used. The AR marker is generated using, for example, "ARToolKit," which is an example of an AR development library. The marker 22 may also be any QR code (registered trademark).

[0255] The markers 22 used in the rough calibration process described above may be different from the markers 22 used in other processes. That is, in the rough calibration process, a homogeneous transformation matrix T Ｃ Ｔ Any method may be adopted as long as it can determine the homogeneous transformation matrix T Ｃ Ｔ may be determined.

[0256] In the above description, an example of a configuration in which the information processing device 100 executes the main processing has been shown, but multiple information processing devices 100 may execute processing in cooperation with each other. Also, some or all of the processing may be executed using computing resources on a cloud.

[0257] <R. Supplementary Note> The present embodiment as described above includes the following technical idea.

[0258] [Configuration 1] A synthesis unit (S2; 150) that generates a synthetic image by synthesizing a first image obtained by capturing an image of a reference object (20) on which a reference marker (22) is arranged with a first camera (10C) and a second image obtained by capturing an image of the reference object with a second camera (10T) on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; a first point cloud generation unit (S3; 154) that extracts a first point cloud (36) included in a first region (34) determined based on the reference marker from the point clouds generated based on information (30) of the reference object; and a second point cloud generation unit (S2; 156) that extracts a second point cloud (56) included in a second region (50) determined based on the reference marker from the point clouds included in the synthetic image. an information processing device comprising: a search unit (S4; 158) that searches for a second point pair (60) in the second point group that corresponds to a first point pair (40) selected from the first point group; and an evaluation unit (S6; 160) that evaluates accuracy of the correspondence based on shape dimensions of the first point pair and the second point pair.

[0259] [Configuration 2] The information processing device according to Configuration 1, further comprising a determination unit (S1; 152) that determines the correspondence based on a difference between a position and an orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the first camera and a position and an orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the second camera.

[0260] [Configuration 3] The information processing device according to Configuration 1 or 2, further comprising an optimization unit (S7; 166) that optimizes the correspondence relationship so as to reduce a difference in shape dimensions between the first point pair and the second point pair.

[0261] [Configuration 4] The information processing device according to any one of configurations 1 to 3, wherein the search unit determines coordinates in the second point cloud corresponding to each point of the first point pair, and determines each point of the second point pair based on a center of gravity (64) of one or more points that exist within a predetermined distance from the determined coordinates.

[0262] [Configuration 5] The information processing device according to any one of configurations 1 to 4, wherein each of the first region and the second region is determined based on a surface (24) of the reference object on which the reference marker is arranged.

[0263] [Configuration 6] The information processing device according to any one of configurations 1 to 5, wherein the shape dimensions include a length and an angle, and the evaluation unit outputs a histogram (214) of length errors and a histogram (224) of angle errors.

[0264] [Configuration 7] The information processing device according to any one of configurations 1 to 6, wherein the evaluation unit calculates a variation in the shape and dimension error by repeating the calculation of the shape and dimension error of the second point pair.

[0265] [Configuration 8] The information processing device according to any one of configurations 1 to 7, wherein the first point pair is selected based on a shape feature of the reference object.

[0266] [Configuration 9] A method for generating a composite image by combining a first image obtained by capturing an image of a reference object (20) on which a reference marker (22) is arranged with a first camera (10C) and a second image obtained by capturing an image of the reference object with a second camera (10T) on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera (S2); extracting a first point cloud (36) included in a first region (34) determined based on information about the reference object (30) from the point clouds generated based on the information about the reference object (30) (S3); extracting a second point cloud (56) included in a second region (50) determined based on the reference marker from the point clouds included in the composite image (S2); and for a first point pair (40) selected from the first point cloud, searching for a second point pair (60) corresponding to the first point pair in the second point cloud (S4). and evaluating (S6) the accuracy of the correspondence based on the geometric dimensions of the first point pair and the second point pair.

[0267] [Configuration 10] A computer (100) is caused to: generate a composite image by combining a first image obtained by capturing an image of a reference object (20) on which a reference marker (22) is placed with a first camera (10C) and a second image obtained by capturing an image of the reference object with a second camera (10T) on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera (S2); extract (S3) a first point cloud included in a first region (34) determined based on information about the reference object (30); and extract (S2) a second point cloud included in a second region (50) determined based on the reference marker from the point clouds included in the composite image; An information processing program (124) that executes the steps of: searching for a second point pair (60) in the second point group that corresponds to a first point pair selected from the first point group (S4); and evaluating the accuracy of the correspondence based on the shape dimensions of the first point pair and the second point pair (S6).

[0268] <S. Advantages> According to information processing system 1 according to the present embodiment, by performing registration using a reference object whose shape, etc. is defined in advance, it is possible to evaluate the accuracy of the homogeneous transformation matrix that is the result of registration. Since the evaluated accuracy is based on absolute error, it is possible to ensure the safety of the robot design.

[0269] According to the information processing system 1 of the present embodiment, a spatial accuracy distribution can be generated by placing reference objects at multiple measurement positions in a target space and calculating the accuracy. By using the spatial accuracy distribution, it is possible to efficiently optimize the camera positions and consider whether or not to add cameras.

[0270] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims.

[0271] 10 Camera, 10C Reference camera, 10T Target camera, 12, 12C, 12T Camera coordinate system, 14C, 14T Field of view, 20, 20A, 20B, 20C, 20D, 20E Reference object (object), 21 CAD canonical coordinate system, 22 Marker, 24 Effective surface, 26 Marker coordinate system, 28 Notch, 29 Chamfer, 30 CAD information, 32 3D point cloud, 34 Target area, 36 CAD-derived point cloud, 40, 60 Key point pair, 42, 62 Key point, 50 ROI, 56 ROI point cloud, 64 Center of gravity, 100 Information processing device, 102 Processor, 104 Memory, 106 Input interface, 108 Input device, 110 Display interface, 112 Display, 114 Camera interface, 116 Network controller, 120 Storage, 122 OS, 124 Information processing program, 150 Synthesis unit, 152 Determination unit, 154 First point cloud generation unit, 156 Second point cloud generation unit, 158 Search unit, 160 Evaluation unit, 162 Calculation unit, 164 Output unit, 166 Optimization unit, 200 Output screen, 210, 220 Information, 212, 222 Statistical information, 214, 224 Histogram, 250, 252, 254 Square pyramid, 260 Main pillar, 261, 262, 263, 264, 265 Planar plate, 270 Synthesis point cloud, 271, 273, 274, 275 Support member, 282, 284 Area.

Claims

1. An information processing device comprising: a synthesis unit that generates a synthetic image by synthesizing a first image obtained by capturing an image of a reference object on which a reference marker is placed with a first camera and a second image obtained by capturing an image of the reference object with a second camera on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; a first point cloud generation unit that extracts a first point cloud included in a first area determined based on the reference marker from a point cloud generated based on information of the reference object; a second point cloud generation unit that extracts a second point cloud included in a second area determined based on the reference marker from a point cloud included in the synthetic image; a search unit that searches for a second point pair in the second point cloud that corresponds to the first point pair for a first point pair selected from the first point cloud; and an evaluation unit that evaluates the accuracy of the correspondence based on the shape and dimensions of the first point pair and the second point pair.

2. The information processing device of claim 1, further comprising a determination unit that determines the correspondence based on the difference between the position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the first camera and the position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the second camera.

3. The information processing apparatus according to claim 1, further comprising an optimization section that optimizes the correspondence relationship so that a difference in shape dimensions between the first point pair and the second point pair becomes small.

4. An information processing device according to any one of claims 1 to 3, wherein the search unit determines coordinates in the second point cloud corresponding to each point of the first point pair, and determines each point of the second point pair based on the center of gravity of one or more points that exist within a predetermined distance from the determined coordinates.

5. An information processing device according to any one of claims 1 to 3, wherein each of the first region and the second region is determined based on a surface of the reference object on which the reference marker is arranged.

6. An information processing device according to any one of claims 1 to 3, wherein the shape dimensions include length and angle, and the evaluation unit outputs a histogram of length errors and a histogram of angle errors.

7. The information processing device according to claim 1, wherein the evaluation unit calculates the variation in the shape dimension error by repeating the calculation of the shape dimension error of the second point pair.

8. An information processing device according to any one of claims 1 to 3, wherein the first point pair is selected based on a geometric feature of the reference object.

9. An information processing method comprising: generating a composite image by combining a first image obtained by capturing an image of a reference object on which a reference marker is placed with a first camera and a second image obtained by capturing an image of the reference object with a second camera on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; extracting a first point cloud included in a first area determined based on the reference marker from the point cloud generated based on information of the reference object; extracting a second point cloud included in a second area determined based on the reference marker from the point cloud included in the composite image; searching for a second point pair in the second point cloud that corresponds to the first point pair for a first point pair selected from the first point cloud; and evaluating the accuracy of the correspondence based on the shape and dimensions of the first point pair and the second point pair.

10. An information processing program that causes a computer to execute the following steps: generate a composite image by combining a first image obtained by capturing an image of a reference object on which a reference marker is placed with a first camera and a second image obtained by capturing an image of the reference object with a second camera on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; extracting a first point cloud included in a first area determined based on the reference marker from the point cloud generated based on information of the reference object; extracting a second point cloud included in a second area determined based on the reference marker from the point cloud included in the composite image; searching for a second point pair in the second point cloud that corresponds to the first point pair for a first point pair selected from the first point cloud; and evaluating the accuracy of the correspondence based on the shape and dimensions of the first point pair and the second point pair.

Images