Information processing apparatus, information processing method, and information processing program

Abstract

To provide an information processing apparatus and an information processing method for visualizing the distribution of the accuracy of a correspondence relation in a space.SOLUTION: An information processing apparatus 100 includes: a first point group generation unit which generates a first point group based on a reference marker based on design information of a reference object; a second point group generation unit which generates a second point group based on the reference marker from a composite image; a search unit for searching a second point pair corresponding to a first point pair in the second point group with respect to the first point pair selected from the first point group; an evaluation unit including a calculation unit for calculating the accuracy of a correspondence based on the shape dimensions of the first point pair and the second point pair; and an output unit for, when a reference object is arranged at a plurality of measurement positions in space, visualizing the accuracy of a correspondence calculated for the respective measurement positions for the respective measurement positions.SELECTED DRAWING: Figure 38

Description

[Technical field]

[0001] The present invention relates to registration for multiple cameras. [Background technology]

[0002] For example, in order to realize an autonomous robot, spatial recognition is necessary. For example, a method using a plurality of images captured by a plurality of cameras may be adopted for spatial recognition. When a plurality of cameras are used, alignment between the plurality of cameras (registration or camera registration) is required. The external parameters (homogeneous transformation matrix) of the cameras are determined by the registration.

[0003] For example, JP 2022-039906 A (Patent Document 1) discloses a multi-sensor composite calibration method including internal parameter calibration of a camera and external parameter calibration of a laser radar camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-039906 Summary of the Invention [Problem to be solved by the invention]

[0005] The above-mentioned prior art documents disclose that the accuracy of calibration has been improved, but do not disclose a method for evaluating the accuracy of calibration itself.

[0006] An object of the present invention is to provide a new method for visualizing the distribution of the accuracy of correspondence relationships in a space. [Means for solving the problem]

[0007] An information processing device according to one example of the present invention includes a synthesis unit that generates a synthetic image by synthesizing a first image obtained by capturing an image of a reference object on which a reference marker is arranged with a first camera and a second image obtained by capturing an image of the reference object with a second camera on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; a first point cloud generation unit that generates a first point cloud based on the reference marker based on design information of the reference object; a second point cloud generation unit that generates a second point cloud based on the reference marker from the synthetic image; a search unit that searches for a second point pair in the second point cloud that corresponds to the first point pair for a first point pair selected from the first point cloud; a calculation unit that calculates the accuracy of the correspondence based on the shape dimensions of the first point pair and the second point pair; and an output unit that visualizes the accuracy of the correspondence calculated for each measurement position in association with each measurement position when the reference object is placed at a plurality of measurement positions in space.

[0008] According to this configuration, the accuracy of the correspondence relationship at each measurement position is calculated based on the geometric dimensions of the first point pair and the second point pair obtained for the reference object arranged at each measurement position. Since the accuracy of the correspondence relationship at each measurement position is visualized, the accuracy distribution in space is quantified. The visualized result makes it easy for the user to grasp the accuracy distribution.

[0009] The output unit may output an image in which the value of the accuracy of the correspondence is expressed by at least one of shading and color. This configuration makes it easier to grasp the spatial accuracy distribution.

[0010] The first point cloud generation unit may extract, from among the point clouds generated based on the design information, a point cloud included in a first region determined based on the reference marker as the first or second point cloud. The second point cloud generation unit may extract, from among the point clouds included in the composite image, a point cloud included in a second region determined based on the reference marker as the second point cloud. With this configuration, by using the surface on which the reference marker is arranged as a reference, it is possible to optimize the point clouds required for associating the first point pair with the second point pair.

[0011] Each of the first region and the second region may be determined based on a surface of the reference object on which a reference marker is arranged. With this configuration, by using the surface on which the reference marker is arranged as a reference, it is possible to optimize a point group required for associating the first point pair with the second point pair.

[0012] The search unit may determine coordinates in the second point cloud corresponding to each point of the first point pair, and determine each point of the second point pair based on a center of gravity of one or more points existing within a predetermined distance from the determined coordinates. With this configuration, it is possible to determine points in the second point cloud corresponding to each point of the first point pair in a statistically robust manner.

[0013] The information processing device may further include a determination unit that determines the correspondence relationship based on a difference between a position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the first camera and a position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the second camera. With this configuration, the correspondence relationship between the camera coordinate systems can also be determined using the reference marker used to evaluate the accuracy of the correspondence relationship.

[0014] An information processing method according to another example of the present invention includes: generating a composite image by combining a first image obtained by capturing an image of a reference object on which a reference marker is arranged with a first camera and a second image obtained by capturing an image of the reference object with a second camera on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; generating a first point cloud based on the reference marker based on design information of the reference object; generating a second point cloud based on the reference marker from the composite image; searching for a second point pair in the second point cloud that corresponds to the first point pair for a first point pair selected from the first point cloud; calculating accuracy of the correspondence based on shape dimensions of the first point pair and the second point pair; and visualizing the accuracy of the correspondence calculated for each measurement position in association with each measurement position when the reference object is placed at a plurality of measurement positions in space.

[0015] An information processing program according to yet another example of the present invention causes a computer to execute the following steps: generate a composite image by combining a first image obtained by capturing an image of a reference object on which a reference marker is arranged with a first camera and a second image obtained by capturing an image of the reference object with a second camera on the same coordinate system based on a correspondence between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; generate a first point cloud based on the reference marker based on design information of the reference object; generate a second point cloud based on the reference marker from the composite image; for a first point pair selected from the first point cloud, search for a second point pair in the second point cloud that corresponds to the first point pair; calculate accuracy of the correspondence based on shape dimensions of the first point pair and the second point pair; and visualize the accuracy of the correspondence calculated for each measurement position in association with each measurement position when the reference object is placed at a plurality of measurement positions in space. Effect of the Invention

[0016] According to the present invention, it is possible to visualize the distribution of the accuracy of correspondence relationships in space. [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic diagram showing a configuration example of an information processing system according to an embodiment of the present invention. [Diagram 2] 1 is a schematic diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present invention; [Diagram 3] 10 is a flowchart showing an example of a processing procedure of an information processing method according to the present embodiment. [Figure 4] 13 is a flowchart showing another example of the processing procedure of the information processing method according to the present embodiment. [Diagram 5] 11A to 11C are diagrams for explaining an example of rough calibration processing according to the present embodiment. [Figure 6] 10A to 10C are diagrams for explaining another example of the rough calibration process according to the present embodiment. [Figure 7] FIG. 13 is a diagram showing an example of AprilTag. [Figure 8] 7 is a diagram for explaining the process of calculating a homogeneous transformation matrix in the rough calibration process shown in FIGS. 5 and 6. FIG. [Figure 9] FIG. 2 is a diagram showing an example of a marker coordinate system for markers arranged on each surface of an object. [Figure 10] 11A and 11B are diagrams illustrating examples of relationships between markers in terms of rotation and translation based on a CAD canonical coordinate system. [Figure 11] 11A to 11C are diagrams for explaining an example of the ROI point cloud extraction process according to the present embodiment. [Figure 12] 5A to 5C are diagrams for explaining an example of the CAD-derived point cloud generation process according to the present embodiment. [Figure 13] 11 is a diagram for explaining an example of a key point search in the identification process according to the present embodiment. FIG. [Figure 14] FIG. 13 is a diagram for explaining a method for searching for key points by a minimum distance method. [Figure 15]FIG. 13 is a diagram for explaining a method for searching for key points by the centroid method. [Figure 16] FIG. 13 is a diagram for explaining a method for searching for key points by the average center of gravity method. [Figure 17] FIG. 13 is a diagram for explaining a method of searching for key points by the average center of gravity method and M-estimation. [Figure 18] FIG. 13 is a diagram for explaining a method of searching for key points by the average center of gravity method and the RANSAC method. [Figure 19] FIG. 13 is a diagram for explaining a method of searching for key points by the average center of gravity method and the RANSAC method. [Figure 20] 11A to 11C are diagrams for explaining an example of geometry calculation in the identification processing according to the present embodiment. [Figure 21] FIG. 11 is a diagram showing an example of information processing according to the present embodiment. [Figure 22] FIG. 11 is a diagram showing an example of information processing according to the present embodiment. [Figure 23] FIG. 11 is a diagram showing an example of information processing according to the present embodiment. [Figure 24] 1 is a schematic diagram showing a configuration example of a reference object used in an information processing system according to the present embodiment. FIG. [Diagram 25] 1 is a schematic diagram showing a configuration example of a reference object used in an information processing system according to the present embodiment. FIG. [Figure 26] 1 is a schematic diagram showing a configuration example of a reference object used in an information processing system according to the present embodiment. FIG. [Figure 27] 1 is a schematic diagram showing a configuration example of a reference object used in an information processing system according to the present embodiment. FIG. [Figure 28] 11 is a diagram for explaining an example of absolute error evaluation in the accuracy evaluation process according to the present embodiment. FIG. [Figure 29] 11 is a diagram for explaining an example of repeated error evaluation of the accuracy evaluation process according to the present embodiment. FIG. [Diagram 30] FIG. 13 is a schematic diagram showing an example of an output of an accuracy evaluation result according to the present embodiment. [Diagram 31] It is a diagram for explaining an example of a penalty term in the fine-tuning process according to this embodiment. [Diagram 32] It is a diagram showing an example of changes in length error due to the fine-tuning process according to this embodiment. [Diagram 33] It is a diagram showing an example of changes in angle error due to the fine-tuning process according to this embodiment. [Diagram 34] It is a diagram for explaining accuracy confirmation using the point cloud thickness of the information processing method according to this embodiment. [Diagram 35] It is a diagram for explaining a method of calculating the accuracy spatial distribution according to this embodiment. [Diagram 36] It is a flowchart showing an example of a processing procedure of a method of calculating the accuracy spatial distribution according to this embodiment. [Figure 37] It is a diagram showing an example of visualizing the accuracy spatial distribution obtained by the method of calculating the accuracy spatial distribution according to this embodiment. [Figure 38] It is a schematic diagram showing an example of the functional configuration of the information processing system according to this embodiment. [Figure 39] It is a diagram for explaining a method of setting an arbitrary world coordinate system in the information processing system according to this embodiment.

Embodiments for Carrying Out the Invention

[0018] Embodiments of the present invention will be described in detail with reference to the drawings. For the same or corresponding parts in the drawings, the same reference numerals are given and the description thereof will not be repeated.

[0019] <A. Application Example> FIG. 1 is a schematic diagram showing a configuration example of an information processing system 1 according to this embodiment. Referring to FIG. 1, the information processing system 1 includes an information processing apparatus 100 and a plurality of cameras 10-1, 10-2,..., 10-N (N is an integer of 2 or more) (hereinafter, may also be collectively referred to as "camera 10").

[0020] The camera 10 captures, for example, a two-dimensional image (RGB image) and a depth image simultaneously. The camera 10 may capture a three-dimensional image.

[0021] The information processing device 100 performs alignment (registration or camera registration) between the multiple cameras 10. The cameras 10-1, 10-2, ..., 10-N each have a camera coordinate system 12-1, 12-2, ..., 12-N (hereinafter, sometimes collectively referred to as "camera coordinate system 12"). The camera coordinate system 12 corresponds to the field of view of each camera 10.

[0022] Registration is a process for determining a correspondence relationship between the camera coordinate systems 12 for multiple cameras 10. In registration, the correspondence relationship between the camera coordinate systems 12 is determined based on one or more points in the space SP. The correspondence relationship between the camera coordinate systems 12 may be, for example, a homogeneous transformation matrix that is an external parameter of the camera. In the following description, a homogeneous transformation matrix will be described as an example of the correspondence relationship between the camera coordinate systems 12, but any parameter, transformation matrix, or transformation function may be used.

[0023] Once the registration is successful, it is possible to calculate a homogeneous transformation matrix T that indicates the correspondence between the camera coordinate system 12 of one camera 10 and the camera coordinate system 12 of another camera 10. The homogeneous transformation matrix T is the extrinsic parameters of the one camera 10 relative to the other camera 10.

[0024] For example, if the camera coordinate system 12-1 of the camera 10-1 is taken as the criteria, the homogeneous transformation matrix T 1 2 is calculated. Similarly, the homogeneous transformation matrix T 1 N is calculated. A homogeneous transformation matrix is ​​a matrix that indicates rotation and translation (position).

[0025] The information processing system 1 according to this embodiment evaluates the accuracy of the correspondence relationship (homogeneous transformation matrix T) between the calculated camera coordinate systems 12.

[0026] <B. Hardware Configuration Example> Next, a hardware configuration example of the information processing apparatus 100 of the information processing system 1 according to this embodiment will be described.

[0027] FIG. 2 is a schematic diagram showing a hardware configuration example of the information processing apparatus 100 according to this embodiment. Referring to FIG. 2, the information processing apparatus 100 is an example of a computer and includes one or more processors 102, a memory 104, an input interface 106, a display interface 110, a camera interface 114, a network controller 116, and a storage 120.

[0028] The one or more processors 102 provide the processes and functions as described later by executing computer-readable instructions included in the programs stored in the storage 120. When the one or more processors 102 execute the computer-readable instructions included in the program, part or all of the program may be expanded in the memory 104.

[0029] The storage 120 is an example of a non-transitory computer-readable medium. For example, an OS (Operating System) 122 and an information processing program 124 are stored in the storage 120. The OS 122 includes computer-readable instructions for providing functions necessary for the computer and computer-readable instructions for generating an environment for executing the information processing program 124.

[0030] The information processing program 124 includes computer-readable instructions for the processes described later.

[0031] The input interface 106 receives input commands from an input device 108 such as a keyboard, a touch panel, a mouse, and a tablet. The input device 108 may be a part of the configuration of the information processing apparatus 100.

[0032] The display interface 110 outputs a video signal to the display 112. The display 112 may be a part of the configuration of the information processing apparatus 100.

[0033] The camera interface 114 exchanges data with a plurality of cameras 10.

[0034] The network controller 116 exchanges data with other information processing apparatuses and the like via a network.

[0035] In this specification, the term "processor" includes a hardwired circuit in addition to an arithmetic circuit that sequentially executes computer-readable instructions such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Examples of hardwired circuits include an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). The "processor" includes a SoC (System on a chip) that integrates a core, memory, and peripheral circuits.

[0036] <C. Example of processing procedure> Next, an example of the processing procedure executed in the information processing system 1 according to the present embodiment will be described.

[0037] FIG. 3 is a flowchart showing an example of the processing procedure of the information processing method according to the present embodiment. The processing shown in FIG. 3 includes registration and accuracy evaluation. Each step shown in FIG. 3 may be realized by the processor 102 of the information processing apparatus 100 executing the information processing program 124.

[0038] 3, the information processing device 100 executes a rough calibration process (step S1). After the rough calibration process is executed, a correspondence relationship between the camera coordinate systems 12 is determined. Step S1 may be executed in advance, independently of the processes of other steps.

[0039] The information processing device 100 executes a ROI (Region of Interest) point cloud extraction process (step S2). The information processing device 100 executes the ROI point cloud extraction process to extract a point cloud included in the ROI (hereinafter, also referred to as an "ROI point cloud").

[0040] The information processing device 100 executes a CAD (Computer Aided Design) derived point cloud generation process (step S3). The information processing device 100 executes the CAD derived point cloud generation process to generate a point cloud based on CAD information (hereinafter also referred to as a "CAD derived point cloud"). Step S3 may be executed in advance, independently of the processing of other steps.

[0041] In this specification, a "point cloud" refers to a collection of points including position information. A "point cloud" may include color information in addition to position information. The position information may be three-dimensional data or 2.5-dimensional data. The position information may include two-dimensional data and depth information (distance information) independently of each other.

[0042] Next, the information processing device 100 executes an identification process (step S4). The identification process includes a process of determining a correspondence relationship between the ROI point cloud extracted by the ROI point cloud extraction process and the CAD-derived point cloud generated by the CAD-derived point cloud generation process. The identification process includes a process of calculating the shape dimensions of the key point pairs of the CAD-derived point cloud and the shape dimensions of the key point pairs of the ROI point cloud.

[0043] Subsequently, the information processing device 100 executes an accuracy evaluation process (step S5). The accuracy evaluation process includes a process of calculating accuracy based on the geometric dimensions calculated in the identification process.

[0044] In this specification, "accuracy" refers to the direct or indirect accuracy of the homogeneous transformation matrix T (extrinsic parameters) that is the result of registration. However, "accuracy" may also refer to the accuracy of spatial information reconstructed based on images captured by multiple cameras. Also, "accuracy" may refer to the accuracy of the result of synthesizing images captured by multiple cameras.

[0045] Subsequently, the information processing device 100 outputs the accuracy evaluation result (step S6). The output of the accuracy evaluation result includes, for example, a spatial visualization of the calculated accuracy.

[0046] In addition to the rough calibration process, a fine-tuning process to optimize the correspondence between the camera coordinate systems 12 may be performed.

[0047] Fig. 4 is a flowchart showing another example of the processing procedure of the information processing method according to the present embodiment. The processing shown in Fig. 4 includes fine tuning processing. Each step shown in Fig. 4 may be realized by the processor 102 of the information processing device 100 executing the information processing program 124.

[0048] The information processing device 100 executes a rough calibration process (step S1). Step S1 may be executed in advance, independently of the processes of other steps.

[0049] The information processing device 100 executes an ROI point cloud extraction process (step S2). The information processing device 100 executes a CAD-derived point cloud generation process (step S3). Step S3 may be executed in advance, independently of the processes of the other steps.

[0050] The information processing apparatus 100 executes identification processing (step S4). The information processing apparatus 100 executes fine-tuning processing (step S7).

[0051] The information processing apparatus 100 determines whether or not the end condition of the fine-tuning processing is satisfied (step S8). If the end condition of the fine-tuning processing is not satisfied (NO in step S8), the processing from step S2 and below is repeated.

[0052] If the end condition of the fine-tuning processing is satisfied (YES in step S8), the information processing apparatus 100 executes accuracy evaluation processing (step S5). Subsequently, the information processing apparatus 100 outputs the accuracy evaluation result (step S6). Then, the processing ends.

[0053] Hereinafter, the details of each step will be described.

[0054] <D. Rough calibration processing (step S1)> Next, the rough calibration processing (step S1) will be described.

[0055] As described above, in the rough calibration processing, the correspondence relationship between the camera coordinate systems 12 is determined. More specifically, the homogeneous transformation matrix T of the camera coordinate system 12T from the camera coordinate system 12C of the reference camera (hereinafter, also referred to as the "reference camera 10C") to another camera (hereinafter, also referred to as the "target camera 10T") C T is calculated.

[0056] The homogeneous transformation matrix T C T may be calculated using any method. Hereinafter, a method using a reference marker and a method using a point cloud will be exemplified.

[0057] (d1: Method using a reference marker) Fig. 5 is a diagram for explaining an example of rough calibration processing according to the present embodiment. Fig. 5 shows an example of rough calibration processing using a reference marker (hereinafter also simply referred to as a "marker").

[0058] 5, a reference object (hereinafter also simply referred to as "object") 20 exists within a field of view (FOV) 14C of each of a reference camera 10C and a target camera 10T. A marker 22 is disposed on the object 20. The marker 22 exists within the field of view 14C of the reference camera 10C and within the field of view 14T of the target camera 10T.

[0059] The marker 22 is, for example, one that can detect the position and orientation in a three-dimensional space. The marker 22 may be, for example, a two-dimensional marker. When the marker 22 is a two-dimensional marker, a surface on which the two-dimensional marker is arranged (hereinafter also referred to as an "effective surface") is defined.

[0060] 6 is a diagram for explaining another example of the rough calibration process according to the present embodiment. In FIG. 6, a case is shown in which the multiple cameras 10 cannot measure the same marker 22.

[0061] 6, a marker 22-1 is placed on one surface of an object 20, and a marker 22-2 is placed on another surface of the object 20. With this arrangement, the marker 22-1 is present within the field of view 14C of the reference camera 10C, and the marker 22-2 is present within the field of view 14T of the target camera 10T.

[0062] When multiple markers are used at the same time, it may be possible to detect identification information indicating the marker 22. As the marker 22, for example, AprilTag developed by The APRIL Robotics Laboratory at the University of Michigan may be used.

[0063] Fig. 7 is a diagram showing an example of AprilTag. Three AprilTags are shown as examples in Fig. 7. The position and orientation in three-dimensional space, as well as the identification information, of each AprilTag can be visually measured.

[0064] 5 and 6, a homogeneous transformation matrix T is calculated based on the difference in position and orientation of the marker 22 in each camera coordinate system 12 measured from images captured by each camera 10. The position and orientation of the marker 22 are expressed, for example, by a six-dimensional vector of position coordinates (three dimensions) and orientation angle (three dimensions).

[0065] The rough calibration process includes a process of determining the correspondence (homogeneous transformation matrix T) between the camera coordinate systems 12 based on the difference between the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 with the reference camera 10C and the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 with the target camera 10T.

[0066] The position and orientation of the marker 22 can also be defined as a marker coordinate system 26 based on the surface (effective surface 24) on which the marker 22 is arranged. In the following description, a process for determining the correspondence relationship (homogeneous transformation matrix T) between the camera coordinate systems 12 using the marker coordinate system 26 will be described.

[0067] FIG. 8 is a diagram for explaining the process of calculating a homogeneous transformation matrix in the rough calibration process shown in FIGS.

[0068] A homogeneous transformation matrix (rotation and translation) from a CAD canonical coordinate system 21, which is set in advance based on CAD information, to the markers 22 arranged on each surface of the object 20 is calculated in advance. By using the CAD canonical coordinate system 21, the ease of handling of the geometric calculation of the rotation and translation of the markers 22 can be improved.

[0069] For example, as shown in FIG. 8A, a homogeneous transformation matrix TCAD mrk1 Also, as shown in FIG. 8B, a homogeneous transformation matrix T CAD mrk2 is calculated.

[0070] A homogeneous transformation matrix T from the camera coordinate system 12C of the reference camera 10C to the camera coordinate system 12T of the target camera 10T C T is T C mrkC ×T mrkC mrkT ×(T T mrkT ) -1 It is calculated as follows.

[0071] Homogeneous transformation matrix T C mrkC corresponds to the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 by the reference camera 10C. T mrkT corresponds to the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 by the target camera 10T.

[0072] Homogeneous transformation matrix T mrkC mrkT corresponds to a rotation and translation from a marker 22 present in the field of view 14C of the reference camera 10C to a marker 22 present in the field of view 14T of the target camera 10T.

[0073] As shown in FIG. 8A, when the same marker 22 is present in the field of view of the reference camera 10C and the target camera 10T, the homogeneous transformation matrix T mrkTC mrkT corresponds to the rotation and translation to the same marker. That is, the homogeneous transformation matrix T mrkC mrkT =E(unit matrix).

[0074] As a result, the homogeneous transformation matrix T C T =TC mrk1 ×(T T mrk2 ) -1 It is calculated as follows.

[0075] On the other hand, as shown in FIG. 8B, when the marker 22 existing within the field of view 14C of the reference camera 10C is different from the marker 22 existing within the field of view 14T of the target camera 10T, the homogeneous transformation matrix T mrkC mrkT is not E (the unit matrix) but is determined as follows:

[0076] Fig. 9 is a diagram showing an example of a marker coordinate system 26 of the markers 22 arranged on each surface of the object 20. Referring to Fig. 9, when the markers 22 having identification information such as AprilTag are used, the identification information of the markers 22 appearing in the captured image is acquired.

[0077] For each piece of identification information of the marker 22, a homogeneous transformation matrix from the CAD canonical coordinate system 21 to the marker 22 is calculated in advance.

[0078] Fig. 10 is a diagram showing an example of the relationship between rotation and translation between markers based on the CAD canonical coordinate system. With reference to Fig. 10, information indicating the amount of rotation and the amount of translation is prepared in advance in association with the identification information (ID:0 to ID:5) of the markers 22.

[0079] The rotation amount and translation amount shown in Fig. 10 are values ​​calculated based on the CAD canonical coordinate system 21 of the marker 22 with ID: 0. The information shown in Fig. 10 may be stored in the information processing device 100 as a homogeneous transformation matrix table.

[0080] For example, if the identification information (ID) of the marker 22 acquired based on an image obtained by capturing an image of the object 20 with the reference camera 10C is “1”, and the identification information (ID) of the marker 22 acquired based on an image obtained by capturing an image of the object 20 with the target camera 10T is “2”, then the homogeneous transformation matrix T mrkC mrkT =T mrk1 mrk2 =(TCAD mrk1 ) -1 ×T CAD mrk2 It is calculated as follows.

[0081] That is, in the homogeneous transformation matrix table shown in FIG. 10, the homogeneous transformation matrix table T CAD mrki and a homogeneous transformation matrix table T corresponding to the identification information of the marker 22 present within the field of view 14T of the target camera 10T. CAD mrki and are selected, and the homogeneous transformation matrix T mrkTC mrkT is calculated.

[0082] The calculated homogeneous transformation matrix T mrkTC mrkT Using the homogeneous transformation matrix T C T =T C mrkC ×T mrkC mrkT ×(T T mrkT ) -1 is calculated.

[0083] In this manner, in the rough calibration process, the object 20 on which one or more markers 22 are arranged is imaged by a plurality of cameras, respectively, and a homogeneous transformation matrix, which is an external parameter of the camera, is calculated.

[0084] (d2: Method using point clouds) 5, it is assumed that an object 20 exists in both the field of view 14C of the reference camera 10C and the field of view 14T of the target camera 10T. After the object 20 to be used in the rough calibration process is determined, a point cloud indicating the outline of the object 20 is extracted from each of the images captured by the reference camera 10C and the target camera 10T.

[0085] By combining the FPFH (Fast Point Feature Histogram) method and the ICP (Iterative Closest Point) method, the information processing apparatus 100 determines the correspondence between the camera coordinate system 12C of the reference camera 10C and the camera coordinate system 12T of the target camera 10T based on the point clouds (indicating the common object 20) extracted from the respective images. That is, the information processing apparatus 100 calculates the homogeneous transformation matrix T of the camera coordinate system 12T from the camera coordinate system 12C of the reference camera 10C to the target camera 10T C T for estimation.

[0086] Through the above processing, the correspondence (homogeneous transformation matrix T) between the camera coordinate systems 12 is calculated.

[0087] <E. ROI Point Cloud Extraction Process (Step S2)> Next, the ROI point cloud extraction process (step S2) will be described.

[0088] As described above, in the ROI point cloud extraction process, the first image obtained by imaging the object 20 on which the marker 22 is arranged with the reference camera 10C and the second image obtained by imaging the object 20 with the target camera 10T are converted into a composite image. Note that the two images may be a set of point clouds. Then, an ROI for extracting the point cloud used in the identification process is determined, and the point cloud (ROI point cloud) included in the ROI is extracted from the composite image.

[0089] As shown in FIG. 5 described above, two images are obtained by imaging the object 20 on which the marker 22 is arranged with the reference camera 10C and the target camera 10T. The two obtained images are synthesized on the same coordinate system (camera coordinate system 12C) based on the correspondence (homogeneous transformation matrix T C T ) between the camera coordinate system 12C of the reference camera 10C and the camera coordinate system 12T of the target camera 10T. For the composite image generated by such transformation, determination of the ROI and extraction of the ROI point cloud are performed.

[0090] As shown in FIG. 6, even if a common marker 22 does not exist within the field of view 14C of the reference camera 10C and the field of view 14T of the target camera 10T, the homogeneous transformation matrix T C T The ROI point cloud extraction process can be performed using the above.

[0091] FIG. 11 is a diagram for explaining an example of the ROI point cloud extraction process according to the present embodiment.

[0092] 11(A), when the position and orientation of the marker 22 of the object 20 in the three-dimensional space are measured, a marker coordinate system 26 is determined based on the surface (effective surface 24) on which the marker 22 is arranged. The origin of the marker coordinate system 26 is set to a predetermined position (for example, the center position) of the marker 22.

[0093] 11(B), a predetermined region is determined as ROI 50 with the origin of the determined marker coordinate system 26 as the center. ROI 50 may be a hexahedron of any size. The size of ROI 50 may be a predetermined fixed value, or may be a variable value that changes according to any condition.

[0094] 11(C), a point group (ROI point group 56) included in the determined ROI 50 is extracted. The ROI point group 56 may be expressed in, for example, a 2.5-dimensional data format. The 2.5-dimensional data format is a format in which depth information (distance information) is added to two-dimensional image data.

[0095] In this way, the ROI point cloud extraction process includes a process of extracting, from among the points contained in the composite image, an ROI point cloud 56 (corresponding to a second point cloud) included in an ROI 50 (corresponding to a second region) determined based on the marker 22. The ROI 50 may be determined based on the surface (effective surface 24) on which the marker 22 of the object 20 is arranged. An identification process is performed based on the extracted ROI point cloud 56.

[0096] For the sake of convenience in explanation, an object 20 with one marker 22 disposed thereon is illustrated. However, when a plurality of markers 22 are disposed on the object 20, the ROI may be determined for each marker 22 (effective surface 24), and the ROI point cloud may be extracted.

[0097] <F. Point cloud generation process from CAD (Step S3)> Next, the point cloud generation process from CAD (Step S3) will be described.

[0098] As described above, in the point cloud generation process from CAD, a point cloud from CAD is generated.

[0099] FIG. 12 is a diagram for explaining an example of the point cloud generation process from CAD according to the present embodiment.

[0100] Referring to FIG. 12, the design information of the object 20 used in the rough calibration process (Step S1) and the ROI point cloud extraction process (Step S2) is prepared. An example of the design information is CAD information 30. However, the design information is not limited to the term "CAD" and may be data in any format.

[0101] Based on the CAD information 30, a three-dimensional point cloud 32 indicating the shape of the object 20 is generated. Among the generated three-dimensional point cloud 32, the point cloud existing in the target region 34 corresponding to the effective surface 24 of the marker 22 is extracted as the point cloud from CAD 36. The point cloud from CAD 36 does not have to include all the point clouds of the object 20, and it is sufficient if it includes at least the point cloud in the vicinity of the effective surface 24 of the marker 22.

[0102] The three-dimensional point cloud 32 is expressed in, for example, a three-dimensional data format. In order to simplify the identification process, the point cloud existing in the target region 34 may be converted from the three-dimensional data format to a 2.5-dimensional data format.

[0103] If the point cloud from CAD 36 is in the 2.5-dimensional data format, in the 2.5-dimensional data format, the ROI point cloud 56 and the point cloud from CAD 36 can be associated with each other.

[0104] In this way, the CAD-derived point cloud generation process includes a process of extracting a CAD-derived point cloud 36 (corresponding to the first point cloud) included in a target region 34 (corresponding to the first region) determined based on a marker 22 from among the point clouds generated based on the design information of the object 20 (for example, CAD information 30). The CAD-derived point cloud 36 may be determined based on the surface (effective surface 24) on which the marker 22 of the object 20 is disposed.

[0105] For the sake of convenience in explanation, an object 20 with one marker 22 disposed thereon is illustrated. However, when a plurality of markers 22 are disposed on the object 20, CAD-derived point clouds 36 may be generated for each marker 22 (effective surface 24).

[0106] <G. Identification Process (Step S4)> Next, the identification process (Step S4) will be described.

[0107] As described above, in the identification process, the correspondence between the ROI point cloud 56 and the CAD-derived point cloud 36 is determined. Also, in the identification process, the shape dimensions of the key point pairs of the CAD-derived point cloud and the shape dimensions of the key point pairs of the ROI point cloud are calculated.

[0108] In the identification process, key points in the CAD-derived point cloud 36 are selected. Key points in the ROI point cloud 56 corresponding to the selected key points in the CAD-derived point cloud 36 are searched for. The key points in the CAD-derived point cloud 36 are determined in units of "key point pairs" formed by combining two key points. In the subsequent process, the shape dimensions of the key point pairs are calculated. That is, the key point pairs are the objects for which the shape dimensions are calculated.

[0109] Fig. 13 is a diagram for explaining an example of key point search in the identification process according to the present embodiment. With reference to Fig. 13, key point pairs 40-1 and 40-2 (hereinafter sometimes collectively referred to as "key point pair 40") are selected in the CAD-derived point cloud 36. The key point pair 40 (and the key point 42) may be selected based on the shape characteristics of the object 20 on which the marker 22 is placed.

[0110] The key point pair 40 (and the key point 42) may be automatically selected according to one or more coordinates preset for the CAD information. The key point pair 40 (and the key point 42) may be selected according to a user's instruction for the CAD-derived point cloud 36. In this case, the information processing device 100 may provide a user interface screen for the user to select the key point pair 40 (and the key point 42).

[0111] 13, the key point pair 40-1 is a set of key points 42-1 and 42-2. The key point pair 40-2 is a set of key points 42-3 and 42-4. Hereinafter, the key points 42-1 to 42-4 may be collectively referred to as "key points 42."

[0112] Key points 62-1 to 62-4 (hereinafter sometimes collectively referred to as "key points 62") are searched for in the ROI point group 56 corresponding to the key points 42-1 to 42-4 in the CAD-derived point group 36. That is, the identification process includes a process of searching for a set of key points 62 (corresponding to a second point pair) in the ROI point group 56 that corresponds to the key point pair 40 (corresponding to a first point pair) selected from the CAD-derived point group 36 (first point group).

[0113] Below, some examples of key point search methods for searching for key points 62 in the ROI point cloud 56 that correspond to key points 42 in the CAD-derived point cloud 36 are provided.

[0114] (g1: Minimum distance method) First, the minimum distance method, which is an example of the key point search method, will be described.

[0115] 14 is a diagram for explaining a method for searching for a key point by the minimum distance method. With reference to FIG. 14, coordinates corresponding to the key point 42 in the CAD-derived point group 36 are given to the ROI point group 56. The distance from the coordinates corresponding to the key point 42 in the CAD-derived point group 36 to each point is calculated. The point having the smallest distance from the coordinates corresponding to the key point 42 in the CAD-derived point group 36 is estimated as the key point 62.

[0116] (g2: Center of gravity method) Next, the centroid method, which is another example of the key point searching method, will be described.

[0117] 15 is a diagram for explaining a method for searching for a key point by the centroid method. With reference to FIG. 15, coordinates corresponding to a key point 42 in the CAD-derived point group 36 are given to the ROI point group 56. A plurality of points existing within a predetermined radius R are determined from the coordinates corresponding to the key point 42 in the CAD-derived point group 36. The centroid of the plurality of points existing within the radius R is estimated as a key point 62.

[0118] In this way, the centroid method includes a process of determining coordinates in the ROI point cloud 56 (second point cloud) corresponding to each point of the key point pair 40 (first point pair) in the CAD-derived point cloud 36, and determining a key point 62 of the key point pair 60 (second point pair) based on the centroid of one or more points existing within a predetermined distance from the determined coordinates. The same applies to various methods based on the centroid method described below.

[0119] (g3: Average Center of gravity method) Next, the average center of gravity method, which is yet another example of the key point search method, will be described.

[0120] Fig. 16 is a diagram for explaining a method for searching for a key point by the average center of gravity method. With reference to Fig. 16, similarly to the above-mentioned center of gravity method, a plurality of points existing within a predetermined radius R from the coordinates corresponding to the key point 42 in the CAD-derived point cloud 36 are selected. A center of gravity 64 of the plurality of points existing within the radius R is calculated.

[0121] When multiple samples of the ROI point cloud 56 can be prepared by the camera 10 capturing images multiple times, the center of gravity 64 can be calculated for each ROI point cloud 56. For example, multiple centers of gravity 64-1 to 64-6 are calculated, and then the average of the multiple centers of gravity 64-1 to 64-6 is estimated as the key point 62.

[0122] (g4: Average centroid method + M-estimation) Next, a method in which M-estimation is added to the average center of gravity method, which is yet another example of the key point search method, will be described.

[0123] Fig. 17 is a diagram for explaining a method of searching for key points by the average center of gravity method and M-estimation. With reference to Fig. 17, when a plurality of samples of the ROI point cloud 56 can be prepared by the camera 10 capturing images a plurality of times, the center of gravity 64 can be calculated for each of the ROI point clouds 56, similarly to the above-mentioned average center of gravity method. Once the plurality of center of gravity 64 are calculated, the average of the calculated plurality of center of gravity 64 is calculated. Based on the calculated average, the deviation and standard deviation σ of each of the plurality of center of gravity 64 are calculated.

[0124] Among the multiple centroids 64, a centroid 64 having a deviation exceeding the standard deviation σ is determined to be an outlier, and the other centroids 64 are determined to be inliers. In the example shown in Fig. 17, three centroids 64-1, 64-3, and 64-4 are determined to be inliers.

[0125] One of the centroids 64 determined as the optimum value is estimated as the key point 62. Alternatively, the average of the centroids 64 determined as the optimum value is estimated as the key point 62.

[0126] (g5: Average centroid method + RANSAC (Random sample consensus) method) Next, a method in which the RANSAC method is added to the average center of gravity method, which is yet another example of the key point search method, will be described.

[0127] 18 and 19 are diagrams for explaining a method of searching for key points by the average centroid method and the RANSAC method. Referring to Fig. 18, a plurality of centroids 64 are calculated from a plurality of samples of the ROI point cloud 56. The plurality of centroids 64 are classified into outliers and inliers.

[0128] In sample group 1, three centroids 64-5, 64-3, and 64-2 are determined as correct values, and the remaining three centroids 64-4, 64-1, and 64-4 are determined as outliers. The average of the centroids determined as correct values ​​and the average of the centroids determined as outliers are determined as RANSAC key points. The difference between the correct value RANSAC key points and the outlier value RANSAC key points is calculated as an error for key points 42 in CAD-derived point group 36.

[0129] Similarly, in sample set 2, three centroids 64-1, 64-5, and 64-6 are determined to be correct values, and the remaining three centroids 64-3, 64-2, and 64-4 are determined to be outliers. The difference between the correct RANSAC key points and the outlier RANSAC key points is calculated as an error for the key points 42 in the CAD-derived point cloud 36.

[0130] 19, the error for a number of sample sets relative to the key point 42 in the CAD-derived point cloud 36 is calculated, and the sample set with the smallest error is determined as the optimal sample set. The average of the centroids of the optimal sample sets is estimated as the key point 62.

[0131] (g5:Short summary) The key point search method is not limited to the methods described above, and any method can be used. However, since the key point search method is an important factor that affects the accuracy evaluation process described later, one or more appropriate key point search methods may be determined after evaluating a plurality of key point search methods.

[0132] When a key point 62 in the ROI point group 56 corresponding to the key point 42 in the CAD-derived point group 36 is searched using the key point search method, the shape dimensions are calculated.

[0133] FIG. 20 is a diagram for explaining an example of shape dimension calculation in the identification process according to the present embodiment. Referring to FIG. 20, key points 62-1 to 62-4 in the ROI point group 56 corresponding to the key points 42-1 to 42-4 in the CAD-derived point group 36 are searched.

[0134] Based on the key points 62-1 to 62-4 in the ROI point group 56, a key point pair 60-1 corresponding to the key point pair 40-1, 40-2 in the CAD-derived point group 36 is determined, and a key point pair 60-2 corresponding to the key point pair 40-3, 40-4 in the CAD-derived point group 36 is determined.

[0135] The shape dimensions of each of the key point pairs 40-1, 40-2 in the CAD-derived point group 36 are calculated. Also, the shape dimensions of each of the key point pairs 60-1, 60-2 (hereinafter, may be collectively referred to as "key point pair 60") in the ROI point group 56 are calculated. The shape dimensions may be, for example, the length (one-dimensional: scalar quantity) of the key point pair, or the length and angle (vector quantity) of the key point pair. The length and direction of the shape dimensions may be expressed using a component indicating the length and a rotation component indicating the angle.

[0136] <H. Processing Example> Next, processing examples of the ROI point group extraction process, the CAD-derived point group generation process, and the identification process will be described.

[0137] FIGS. 21 to 23 are diagrams showing processing examples of the information processing according to the present embodiment.

[0138] After the rough calibration process, a homogeneous transformation matrix T from the camera coordinate system 12C of the reference camera 10C to the camera coordinate system 12T of the target camera 10T is obtained. C T is calculated.

[0139] Referring to FIG. 21(A), the homogeneous transformation matrix T C T The marker coordinate system 26 in the camera coordinate system 12C of the target camera 10T is transformed to the marker coordinate system 26 in the camera coordinate system 12C of the reference camera 10C using the above. By this coordinate transformation, the point cloud obtained by the imaging of the reference camera 10C and the point cloud obtained by the imaging of the target camera 10T are synthesized on the same coordinate system. That is, the homogeneous transformation matrix T C T The points are aggregated after matching criteria using coordinate transformation using

[0140] 21(B), an ROI is determined based on the effective surface 24, and a point group included in the ROI (ROI point group 56) is extracted. That is, a point group of the surface layer in the vicinity of the effective surface 24 is extracted.

[0141] 21C, the ROI point group 56 is coordinate-transformed into the CAD canonical coordinate system 21. This coordinate transformation makes the standard for calculating the shape dimensions consistent in the identification process.

[0142] 22, the key points 42 and the key point pairs 40 are selected based on CAD information of the object 20 on which the markers 22 are placed. The key points 42 may be selected based on geometric features of the object 20.

[0143] 23(A), coordinates corresponding to key points 42 in CAD-derived point cloud 36 are given to ROI point cloud 56.

[0144] With reference to FIG. 23(B), key points 62 in the ROI point group 56 corresponding to the key points 42 in the CAD-derived point group 36 are searched by the key point search method.

[0145] <I. Reference Object> Next, an example of the shape of the reference object (object 20) will be described.

[0146] As described above, the object 20 has an effective surface 24 on which the marker 22 is disposed, and may have a shape feature for selecting the key point 42. Some shapes as the reference object will be exemplified.

[0147] FIG. 24 is a schematic diagram showing a configuration example of a reference object 20A used in the information processing system 1 according to the present embodiment. With reference to FIG. 24, the reference object 20A is based on a cube. Markers 22 are respectively disposed on each surface of the reference object 20A (a total of six surfaces). In FIG. 24, only the markers 22-1 to 22-3 located on three surfaces are shown, but markers 22 are also disposed on each of the remaining surfaces not shown.

[0148] The reference object 20A has portions corresponding to four vertices of the cube cut out. More specifically, cubic cutouts 28-1 and 28-2 are provided at a pair of opposing vertices on the upper surface of the reference object 20A. In addition, chamfered portions 29-1 and 29-2 are provided at another pair of opposing vertices on the upper surface of the reference object 20A.

[0149] The cutouts 28-1 and 28-2 and the chamfered portions 29-1 and 29-2 are shape features of the reference object 20A and can be criteria for selecting the key point 42 and the key point pair 40.

[0150] Fig. 25 is a schematic diagram showing a configuration example of reference object 20B used in information processing system 1 according to the present embodiment. With reference to Fig. 25, reference object 20B includes a main pyramid 250, and pyramids 252 and 254 whose bottom surfaces coincide with one face of pyramid 250. Markers may be placed on faces of pyramid 250 on which pyramids 252 and 254 are not present.

[0151] The vertices of the pyramid 250 and the pyramids 252 and 254 are geometric features of the reference object 20B and can serve as criteria for selecting the key points 42 and the key point pairs 40.

[0152] FIG. 26 is a schematic diagram showing a configuration example of reference objects 20C and 20D used in information processing system 1 according to the present embodiment.

[0153] 26(A), reference object 20C has a cube as a base. Markers 22-1 to 22-3 are arranged on three faces of reference object 20C, respectively.

[0154] The reference object 20C has portions corresponding to the sides of its upper surface cut out. More specifically, notches 28-3 to 28-6 are provided in the portions corresponding to the sides of the upper surface of the reference object 20C.

[0155] The cutout portions 28-3 to 28-6 are formed so that the cutout sections and lengths are different from one another, that is, the cutout portions 28-3 to 28-6 have shapes that can be distinguished from one another.

[0156] The notches 28-3 to 28-6 are shape features of the reference object 20C and can serve as criteria for selecting the key points 42 and the key point pairs 40.

[0157] 26(B), reference object 20D has a cube as a base. Markers 22-1 to 22-3 are arranged on three faces of reference object 20D, respectively.

[0158] The reference object 20D has two sides partially cut out. More specifically, the reference object 20D has two side surfaces each provided with a cutout portion 28-7, 28-8.

[0159] The cross section of the cutout portion 28-7 is rectangular, and the cross section of the cutout portion 28-8 is triangular. The cutout portions 28-7 and 28-8 may have a shape in which the cross-sectional area decreases toward the inside of the reference object 20D. In this way, the cross-sectional shapes of the cutout portions 28-7 and 28-8 are different, so that they can be distinguished from each other.

[0160] The notches 28-7 and 28-8 are geometric features of the reference object 20D and can serve as criteria for selecting the key points 42 and the key point pairs 40.

[0161] Fig. 27 is a schematic diagram showing a configuration example of a reference object 20E used in information processing system 1 according to the present embodiment. With reference to Fig. 27, reference object 20E includes a main pillar 260 and five flat plates 261-265 on which markers 22 can be arranged. Flat plate 262 is supported by main pillar 260. Flat plates 261, 263, 264, 265 are supported by support members 271, 273, 274, 275 connected to main pillar 260.

[0162] The arrangement of the planar plates 261 to 265 is a geometric feature of the reference object 20E, and can be a criterion for selecting the key points 42 and the key point pairs 40.

[0163] The reference object (object 20) may be a cube or a rectangular parallelepiped without any cutouts etc. The reference object (object 20) is not limited to the above-mentioned shapes and may be any shape.

[0164] As described above, the key points 42 and the key point pairs 40 may be selected based on the geometric characteristics of a reference object. A reference object having geometric characteristics may be employed to set the key points 42 and the key point pairs 40.

[0165] <J. Precision evaluation process (step S5)> Next, the precision evaluation process (step S5) will be described.

[0166] As described above, in the precision evaluation process, the precision is calculated based on the shape dimensions calculated in the identification process. That is, the precision evaluation process includes a process of evaluating the precision of the homogeneous transformation matrix T (the correspondence relationship between the camera coordinate systems 12) based on the shape dimensions of the key point pair 40 (the first point pair) in the CAD-derived point cloud 36 and the key point pair 60 (the second point pair) in the ROI point cloud 56.

[0167] The calculated precision includes, for example, at least one of the absolute error and the repeatability error.

[0168] (j1: Absolute error) FIG. 28 is a diagram for explaining an example of the absolute error evaluation of the precision evaluation process according to the present embodiment. Referring to FIG. 28, for each key point pair, the difference between the shape dimensions of the key point pair 40 in the CAD-derived point cloud 36 and the shape dimensions of the key point pair 60 in the ROI point cloud 56 is calculated. The calculated difference becomes the absolute error.

[0169] By statistically processing the absolute errors calculated for each of the plurality of key point pairs, the absolute error in the target information processing system 1 is calculated. Also, by statistically processing the shape errors of each key point pair calculated from a plurality of samples, the absolute error in the target information processing system 1 is calculated.

[0170] For example, for the key point pair p1, if the length of the shape dimension in the key point pair 40 is L cad 1 and the length of the shape dimension in the key point pair 60 is L ROI 1, then the absolute error Δp1 = |L cad 1 - L ROI 1|. Similarly, for the key point pair p2, if the shape dimension in the key point pair 40 is Lc2 and the shape dimension in the key point pair 60 is L ROIIf 2, then the absolute error Δp2 = |L cad 2-L ROI 2|The result is

[0171] Alternatively, the angle between the key point pair p1 and the key point pair p2 is θ cad 1, and the angle at the key point pair 60 is θ ROI If 1, then the absolute error Δpθ1=|θ cad 1-θ ROI 1|The result is

[0172] In this manner, an absolute error is calculated as the accuracy based on the difference between the geometric dimensions of the key point pairs 40 in the CAD-derived point cloud 36 and the geometric dimensions of the key point pairs 60 in the ROI point cloud 56 .

[0173] (j2: repeat error) Fig. 29 is a diagram for explaining an example of the repetitive error evaluation of the accuracy evaluation process according to the present embodiment. With reference to Fig. 29, for a plurality of samples, the variation in the shape and dimensions of key point pairs 60 in ROI point cloud 56 is calculated for each key point pair. The calculated variation (e.g., standard deviation) is the repetitive error.

[0174] For example, for key point pair p1, the feature size of sample 1 is L ROI 11, and the shape dimension of sample 2 is L ROI If it is 12, then L ROI 11 and L ROI The error with 12 is a repeatable error.

[0175] Similarly, for key point pair p2, the geometric dimension of sample 1 is L ROI 21, and the shape dimension of sample 2 is L ROI If 22, then L ROI 21 and L ROI The error with 22 is a repeatable error.

[0176] Alternatively, for key point pair p1, the angle at sample 1 is θ cad 11, and the angle in sample 2 is θROI When it is 12, θ ROI 21 and θ ROI The error between 22 becomes the repetitive error.

[0177] When calculating the repetitive error for three or more samples, the standard deviation of the shape dimensions of the same key point pair is calculated as the repetitive error.

[0178] In this way, the accuracy evaluation process includes a process of calculating the variation in the error of the shape dimensions by repeating the calculation of the error of the shape dimensions of the key point pair 60 (the second point pair) in the ROI point group 56.

[0179] (j3: Others) For each key point pair, the shape dimensions of the key point pair 40 in the CAD-derived point group 36 and the shape dimensions of the key point pair 60 in the ROI point group 56 can be obtained. Therefore, any accuracy evaluation process can be performed using at least one of the two types of shape dimensions.

[0180] <K. Output of Accuracy Evaluation Result (Step S6)> Next, the output of the accuracy evaluation result (step S6) will be described.

[0181] FIG. 30 is a schematic diagram showing an example of the output of the accuracy evaluation result according to the present embodiment. Referring to FIG. 30, the output screen 200 of the accuracy evaluation result includes information 210 indicating the error in the length of the shape dimension and information 220 indicating the error in the angle of the shape dimension.

[0182] The information 210 includes statistical information 212 and a histogram 214 for the error in length. The histogram 214 shows the number (frequency) of key point pairs for each magnitude of the error in length. The information 220 includes statistical information 222 and a histogram 224 for the error in angle. The histogram 224 shows the number (frequency) of key point pairs for each magnitude of the error in angle.

[0183] In this way, the information processing apparatus 100 may output a histogram 214 for the error in length and a histogram 224 for the error in angle.

[0184] In the histogram, it can be judged that the higher the accuracy is, the more the distribution is biased to the left. As shown in FIG. 30, by referring to the distribution shape of the histogram for the length and angle errors, it is also possible to evaluate the variation of the errors.

[0185] <L. Fine Tuning Process (Step S7)> Next, the fine tuning process (Step S7) will be described. When the calculated accuracy (absolute error and repeat error) does not satisfy the predetermined conditions, a process including the fine tuning process may be executed.

[0186] The fine tuning process is a process of optimizing the homogeneous transformation matrix T so as to minimize the error of the shape dimensions calculated by the identification process. That is, the fine tuning process includes a process of optimizing the homogeneous transformation matrix T (the correspondence relationship between the camera coordinate systems 12) so that the difference in shape dimensions from the key point pair 40 (the first point pair) in the CAD-derived point group 36 and the key point pair 60 (the second point pair) in the ROI point group 56 becomes small. The optimization of the homogeneous transformation matrix T can be reduced to a non-linear programming problem for minimizing (or maximizing) an evaluation function using the shape dimensions.

[0187] In the fine tuning process, an evaluation function indicating the cost is defined, and each parameter of the homogeneous transformation matrix T is searched so as to minimize the cost indicated by the evaluation function.

[0188] More specifically, for the homogeneous transformation matrix T * (R * , t * ), the following evaluation function can be used.

[0189]

Equation

[0190] The evaluation function includes the sum of squares of absolute errors between the geometric dimensions of the key point pairs 40 in the CAD-derived point cloud 36 and the geometric dimensions of the key point pairs 60 in the ROI point cloud 56 for all key point pairs. In other words, minimizing the evaluation function means minimizing the sum of squares of absolute errors between the geometric dimensions of the key point pairs 40 in the CAD-derived point cloud 36 and the geometric dimensions of the key point pairs 60 in the ROI point cloud 56.

[0191] The above evaluation function may include a penalty term such as:

[0192]

number

[0193] FIG. 31 is a diagram for explaining an example of the penalty term in the fine-tuning process according to the present embodiment.

[0194] 31, depending on the accuracy of the rough calibration process, fluctuations in the point cloud, and the like, there may be cases where a predetermined number of points do not exist within a predetermined radius R from the coordinates corresponding to the key point 42 in the CAD-derived point cloud 36. In the example on the left side (actual state) of FIG. 30, only one point exists within the radius R.

[0195] In such a case, the radius R may be increased stepwise until a predetermined number of points are found. However, the increase in the search radius from the reference radius R to radius R' may be reflected in the penalty term. For example, the value obtained by multiplying |radius R'-radius R| by a coefficient may be used as the penalty term.

[0196] In addition to the error in the length and the error in the angle of the shape dimensions, an evaluation function may be adopted that uses an L2 regularization term as a penalty term. For example, the following evaluation function may be used.

[0197]

number

[0198] As described above, the evaluation function used for the fine-tuning process may be any function as long as it uses the calculation results of the shape dimensions of the key point pairs obtained in the identification process.

[0199] Next, an example of the effect of the fine-tuning process will be described.

[0200] FIG. 32 is a diagram showing an example of changes in length error due to the fine-tuning process according to the present embodiment. Referring to FIG. 32, it can be seen that before the fine-tuning process, the histogram showed multimodality, whereas after the fine-tuning process, the histogram showed an exponential distribution. In the histogram, the more the distribution is skewed to the left, the better the accuracy. Thus, it can be seen that the fine-tuning process is approaching an ideal error distribution.

[0201] FIG. 33 is a diagram showing an example of changes in angle error due to the fine-tuning process according to the present embodiment. Referring to FIG. 33, it can be seen that before the fine-tuning process, the histogram deviated from the exponential distribution, whereas after the fine-tuning process, the histogram showed an exponential distribution. For the exponential distribution p(x)=λexp(-λ), after the fine-tuning process, the parameter λ has increased. That is, the histogram comes to show a sharper exponential distribution.

[0202] Thus, the fine-tuning process can reduce the error of the shape dimensions.

[0203] <M. Application Examples of Accuracy Evaluation> Next, some application examples of the accuracy evaluation of the information processing method according to the present embodiment will be described. The output of the above-described accuracy evaluation result was based on the execution results of the rough calibration process and the fine-tuning process, but the information processing method according to the present embodiment can also be used for the secular change of accuracy and the regular accuracy management.

[0204] (m1: Histogram-based quality control) As shown in the above-mentioned Figure 32, the accuracy is reflected in the distribution shape of the histogram. By comparing the distribution shapes of the histograms using a predetermined algorithm, the change in accuracy, etc. can be quantitatively calculated.

[0205] Examples of algorithms for comparing the distribution shapes of histograms include a method using a histogram intersection kernel and a method using a histogram Euclidean distance.

[0206] For example, a facility maintainer may generate and store a histogram of errors before starting operation, and then regenerate the histogram of errors during daily inspections, etc., and compare it with the stored histogram. The facility maintainer can determine whether or not accuracy has decreased based on a change in the distribution shape of the histogram.

[0207] Observation based on changes in the distribution shape of the histogram for such errors may be used as equipment maintenance work.

[0208] Alternatively, accuracy may be trended based on a parameter λ estimated by approximating a histogram of the error using an exponential distribution p(x)=λexp(−λ).

[0209] For example, before starting operation, a facility maintainer generates a histogram of the error, and stores the parameter λ determined by approximating the generated histogram with an exponential distribution. The facility maintainer regenerates the histogram of the error during routine inspections, and calculates the parameter λ by approximating the generated histogram with an exponential distribution. The facility maintainer can manage the trend of the accuracy based on the change in the value of the parameter λ.

[0210] For example, the margin I calculated according to the following formula is used for trend management of accuracy. λmargin may also be used.

[0211]

number

[0212] where λ Exp_crite denotes the reference value of the parameter λ, and λ Exp_curr denotes the value of the parameter λ calculated from the histogram to be evaluated.

[0213] If the parameter λ does not change from the reference value, the margin I λmargin = 0, and if there is an improvement, the margin I λmargin >0, and the deterioration direction is margin I λmargin <0. Margin I λmargin The absolute value indicates the degree of trend (improvement or deterioration).

[0214] The parameter λ, calculated from a histogram of such errors, can also be used to trend the accuracy.

[0215] (m2: Accuracy control based on point cloud thickness) After the fine tuning process, the thickness of the point cloud (point cloud thickness) may be used as a method for checking the accuracy in detail.

[0216] FIG. 34 is a diagram for explaining accuracy check using the point cloud thickness in the information processing method according to the present embodiment.

[0217] Referring to Figure 34 (A), after performing a rough calibration process or a fine tuning process, a composite point cloud 270 is generated by combining a point cloud extracted from an image captured by the reference camera 10C and a point cloud extracted from an image captured by the target camera 10T.

[0218] For the composite point cloud 270, a point cloud thickness is calculated from each of a region 282 corresponding to the composite seam and a region 284 corresponding to other than the composite seam.

[0219] The point cloud thickness indicates the degree of variation of the point cloud within the region. That is, the smaller the calculated point cloud thickness, the higher the accuracy.

[0220] The point cloud thickness of the region 284 corresponding to other than the splicing part of the synthesis depends on the accuracy of the camera 10 (sensor) itself. Ideally, it is preferable that the point cloud thickness of the region 282 corresponding to the splicing part of the synthesis is the same as the point cloud thickness of the region 284 corresponding to other than the splicing part of the synthesis. That is, based on how close the point cloud thickness of the region 282 is to the point cloud thickness of the region 284, it is possible to quantitatively evaluate how appropriately the rough calibration process or the fine tuning process is executed.

[0221] Referring to FIG. 34(B), the reference plane Q is estimated from the point cloud included in the specified region. The reference plane Q is determined based on the coordinates of the point cloud included in the specified region by, for example, the least squares method. The Euclidean distance L(i) between the reference plane Q and the point P(i) included in the point cloud is calculated. Finally, the variance of the Euclidean distance L(i) is determined as the point cloud thickness. That is, it is calculated as the point cloud thickness Thickness = variance(L(i:1→n)).

[0222] The larger the value of the point cloud thickness, the more the point cloud exists in a disordered state with respect to the reference plane Q. The synthesis accuracy is calculated by comparing the point cloud thickness between the region 282 corresponding to the splicing part of the synthesis and the region 284 corresponding to other than the splicing part of the synthesis.

[0223] <N. Precision Spatial Distribution> As described above, the identification process is based on the point cloud extracted from the vicinity of the effective surface 24 of the marker 22. Therefore, the calculated accuracy is for the vicinity of the effective surface 24. If the object 20 on which the marker 22 is arranged is arranged at a plurality of positions in the space SP and the accuracy at each position is calculated, the precision spatial distribution in the space SP can be obtained.

[0224] By using the spatial distribution of precision in the space SP, the uniformity (flatness) of precision in the space SP can be evaluated. In addition, the spatial distribution of precision in the space SP may be visualized.

[0225] Fig. 35 is a diagram for explaining a method of calculating accuracy spatial distribution according to the present embodiment. With reference to Fig. 35, a reference camera 10C and a target camera 10T are arranged in a space SP.

[0226] While maintaining the positions of the reference camera 10C and the target camera 10T, the object 20 on which the marker 22 is placed is successively moved to different measurement positions Pos1, Pos2, Pos3, .... At each measurement position, the accuracy is calculated by the process described above.

[0227] Fig. 36 is a flowchart showing an example of a processing procedure of a calculation method for accuracy spatial distribution according to the present embodiment. Each step shown in Fig. 36 may be realized by causing the processor 102 of the information processing device 100 to execute the information processing program 124.

[0228] The information processing device 100 executes a rough calibration process (step S1). At this time, the object 20 on which the markers 22 are arranged is placed at an arbitrary measurement position. Step S3 may be executed in advance, independently of the processes of the other steps.

[0229] The information processing device 100 executes an ROI point cloud extraction process (step S2). The information processing device 100 executes a CAD-derived point cloud generation process (step S3). Step S3 may be executed in advance, independent of the processes of other steps. Also, step S3 may be executed only the first time in a loop process including step S10.

[0230] The information processing device 100 executes an identification process (step S4). Then, the information processing device 100 executes an accuracy evaluation process (step S5). The information processing device 100 stores the calculated accuracy and the coordinates obtained by converting the marker coordinate system 26 used in the accuracy calculation into the global coordinate system in association with each other (step S9).

[0231] The information processing device 100 determines whether the object 20 on which the marker 22 is placed has been moved to another measurement position (step S10). The user may give an instruction to the information processing device 100, which may cause the information processing device 100 to determine whether the object 20 has been moved to another measurement position.

[0232] If the object 20 on which the marker 22 is placed has been moved to another measurement position (YES in step S10), the processes from step S2 onwards are repeated.

[0233] If the object 20 on which the marker 22 is placed has not been moved to another measurement position (NO in step S10), the information processing device 100 calculates the accuracy spatial distribution based on a pair of the calculated accuracy and coordinates obtained by converting the marker coordinate system 26 used to calculate the accuracy into the global coordinate system (step S11).

[0234] Since it may not be possible to move the object 20 evenly throughout the space SP, the information processing device 100 may use a known interpolation method to calculate the accuracy of each measurement position within the space SP from the stored information.

[0235] The information processing device 100 visualizes and outputs the calculated spatial distribution of precision in the space SP (step S12), and the process then ends.

[0236] As described above, the process of calculating the accuracy spatial distribution includes a process of calculating the accuracy of the correspondence relationship (homogeneous transformation matrix T) between the camera coordinate systems 12 based on the shape dimensions of the key point pair 40 (first point pair) of the CAD-derived point group 36 and the key point pair 60 (second point pair) of the ROI point group 56, and a process of visualizing the accuracy of the correspondence relationship (homogeneous transformation matrix T) between the camera coordinate systems 12 calculated for each measurement position in association with each measurement position when the object 20 is arranged at a plurality of measurement positions in the space.

[0237] FIG. 37 is a diagram showing an example of visualizing the accuracy spatial distribution obtained by the method for calculating the accuracy spatial distribution according to the present embodiment. In the accuracy spatial distribution shown in FIG. 37, the accuracy of each measurement position is represented by shading or color. Thus, an image in which the accuracy value of the correspondence relationship (homogeneous transformation matrix T) between the camera coordinate systems 12 is represented by at least one of shading and color may be output.

[0238] According to the visualized accuracy spatial distribution, in the space SP, it is possible to assist the user in recognizing regions where there are variations and biases in errors. Regions where there are variations and biases in errors are, for example, effective information for optimizing the installation location of the robot. Also, information on the presence or absence of regions where there are variations and biases in errors is effective information for determining whether it is necessary to add a camera.

[0239] <O. Functional Configuration Example> Next, a functional configuration example of the information processing system 1 according to the present embodiment will be described.

[0240] FIG. 38 is a schematic diagram showing a functional configuration example of the information processing system 1 according to the present embodiment. Each component shown in FIG. 38 may be realized by the processor 102 of the information processing apparatus 100 executing the information processing program 124.

[0241] Referring to FIG. 38, the information processing device 100 includes, as components, a synthesis unit 150, a determination unit 152, a first point cloud generation unit 154, a second point cloud generation unit 156, a search unit 158, an evaluation unit 160, an output unit 164, and an optimization unit 166.

[0242] The synthesis unit 150 synthesizes a homogeneous transformation matrix T C T The synthesis unit 150 converts two images captured by the reference camera 10C and the target camera 10T into a composite image using the above-mentioned synthesis unit 150. The synthesis unit 150 executes the process of generating a composite image of the ROI point cloud extraction process (step S3).

[0243] The determination unit 152 calculates a homogeneous transformation matrix T based on a difference between the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 by the reference camera 10C and the position and orientation of the marker 22 calculated based on an image obtained by capturing an image of the object 20 by the target camera 10T. C T The determining unit 152 executes a rough calibration process (step S1).

[0244] The first point cloud generation unit 154 extracts the CAD-derived point cloud 36 from a point cloud generated based on the design information (CAD information 30) of the object 20. The first point cloud generation unit 154 executes a CAD-derived point cloud generation process (step S3).

[0245] The second point cloud generation unit 156 extracts, from the point clouds included in the composite image, an ROI point cloud 56 included in the ROI 50 determined based on the marker 22. The second point cloud generation unit 156 executes a process of determining the ROI and a process of extracting the ROI point cloud in the ROI point cloud extraction process (step S3).

[0246] The search unit 158 searches for key points in the ROI point cloud 56 corresponding to the key points in the selected CAD-derived point cloud 36. The search unit 158 calculates the shape dimensions of the searched key point pairs. The search unit 158 calculates the shape dimensions of the key point pairs of the CAD-derived point cloud and the shape dimensions of the key point pairs of the ROI point cloud. The search unit 158 executes an identification process (step S4).

[0247] The evaluation unit 160 calculates the accuracy based on the shape dimensions of the key point pairs calculated by the search unit 158. The evaluation unit 160 may include a calculation unit 162 for calculating the accuracy. The evaluation unit 160 executes an accuracy evaluation process (step S5).

[0248] The output unit 164 outputs the calculation results by the evaluation unit 160. The output unit 164 executes a process of outputting the accuracy evaluation result (step S6).

[0249] The output unit 164 may generate and output an accuracy spatial distribution. The output unit 164 executes the association of accuracy with coordinates (step S9), the calculation of the accuracy spatial distribution (step S11), and the visualization of the accuracy spatial distribution (step S12).

[0250] The optimization unit 166 optimizes the homogeneous transformation matrix T so that the difference in the shape dimensions between the key point pair 40 of the CAD-derived point cloud 36 and the key point pair 60 of the ROI point cloud 56 becomes small. The optimization unit 166 executes a fine-tuning process (step S7). C T To optimize. The optimization unit 166 executes a fine-tuning process (step S7).

[0251] <P. Setting of the world coordinate system> Next, a process of setting an arbitrary world coordinate system will be described. After the correspondence between the camera coordinate systems 12 is determined by the rough calibration process or the fine-tuning process, an arbitrary world coordinate system may be set at an arbitrary position using the reference object 20.

[0252] Fig. 39 is a diagram for explaining a method of setting an arbitrary world coordinate system in the information processing system according to the present embodiment. With reference to Fig. 39, cameras 10-1, 10-2, and 10-3 are arranged in a space SP.

[0253] For example, assume that the camera 10-1 is a reference camera and the cameras 10-2 and 10-3 are target cameras. A homogeneous transformation matrix T from the camera 10-1 (in its camera coordinate system) to the camera 10-2 (in its camera coordinate system) is obtained by rough calibration processing or fine tuning processing. 1 Similarly, a homogeneous transformation matrix T from the camera 10-1 (in the camera coordinate system) to the camera 10-3 (in the camera coordinate system) is determined. 1 3 has been decided.

[0254] First, a reference object 20 is placed so as to be included in at least a part of the field of view of each of the cameras 10-1 to 10-3. The reference object 20 is placed at a position where a user wants to set a world coordinate system. The reference object 20 has a CAD canonical coordinate system 21 that is set in advance based on CAD information. The CAD canonical coordinate system 21 of the reference object 20 is used as the world coordinate system.

[0255] The reference object 20 is captured by the cameras 10-1 to 10-3, and based on the images, the position and orientation of the CAD canonical coordinate system 21 in the camera coordinate system of each camera 10 are calculated. The position and orientation of the CAD canonical coordinate system 21 are determined based on the surface (effective surface) on which the markers 22 of the reference object 20 are arranged.

[0256] The position and orientation of the CAD canonical coordinate system 21 calculated based on the image captured by the camera 10-1, which is the reference camera, may be set as the world coordinate system as it is.

[0257] If the reference object 20 does not exist within the field of view of the reference camera, camera 10-1, and exists within the field of view of camera 10-2 or camera 10-3, the world coordinate system may be determined using the homogeneous transformation matrix. More specifically, the position and orientation of the CAD alignment coordinate system 21 calculated based on the image captured by camera 10-2 or camera 10-3 are transformed to the position and orientation of the camera coordinate system of camera 10-1 using the homogeneous transformation matrix. In this way, the world coordinate system may be determined via the camera coordinate system of the reference camera.

[0258] Note that when the world coordinate system is not set, the camera coordinate system of the reference camera can be used as the world coordinate system.

[0259] <Q. Variant example> In the above description, a configuration example using AprilTag, which is an example of a 2D marker, has been described. However, the marker 22 is not limited to AprilTag, and any AR (Augmented Reality) marker can be used. The AR marker is generated, for example, using "ARToolKit", which is an example of an AR development library. The marker 22 may be any QR code (registered trademark).

[0260] The marker 22 used in the above rough calibration process and the marker 22 used in other processes may be different. That is, in the rough calibration process, as long as the homogeneous transformation matrix T C T from the camera coordinate system 12C of the reference camera 10C to the camera coordinate system 12T of the target camera 10T can be determined, any method may be adopted. Therefore, the homogeneous transformation matrix T C T may be determined using any pre-trained machine learning model.

[0261] In the above description, a configuration example in which the information processing apparatus 100 executes main processing has been shown. However, a plurality of information processing apparatuses 100 may execute processing in cooperation. Further, part or all of the processing may be executed using computing resources on the cloud.

[0262] <R. Appendix> The present embodiment as described above includes the following technical ideas.

[0263] [Configuration 1] A synthesizing unit (S2; 150) that synthesizes a first image obtained by imaging a reference object (20) on which a reference marker (22) is disposed with a first camera (10C) and a second image obtained by imaging the reference object with a second camera (10T) on the same coordinate system based on the correspondence between the first camera coordinate system of the first camera and the second camera coordinate system of the second camera to generate a synthesized image; A first point group generation unit (S3; 154) that generates a first point group with the reference marker as a reference based on design information (30) of the reference object; A second point group generation unit (S2; 156) that generates a second point group with the reference marker as a reference from the synthesized image; A search unit (S4; 158) that searches for a second point pair (60) corresponding to the first point pair in the second point group for a first point pair (40) selected from the first point group; A calculation unit (S5; 160) that calculates the accuracy of the correspondence based on the shape dimensions of the first point pair and the second point pair; An information processing apparatus comprising: an output unit (S9, S11, S12; 164) that visualizes the accuracy of the correspondence calculated for each measurement position in association with each measurement position when the reference object is disposed at a plurality of measurement positions in space.

[0264] [Configuration 2] The information processing apparatus according to Configuration 1, wherein the output unit outputs an image in which the accuracy value of the correspondence is represented by at least one of shading and color.

[0265] [Configuration 3] the first point cloud generation unit extracts, from the point cloud generated based on the design information, a point cloud included in a first region (34) determined based on the reference marker as the first point cloud; The information processing device according to configuration 1 or 2, wherein the second point cloud generation unit extracts the second point cloud from among the point clouds included in the composite image, the second point cloud being a point cloud included in a second region (50) determined based on the reference marker.

[0266] [Configuration 4] 4. The information processing device according to configuration 3, wherein each of the first region and the second region is determined based on a surface (24) of the reference object on which the reference marker is arranged.

[0267] [Configuration 5] The information processing device according to any one of configurations 1 to 4, wherein the search unit determines coordinates in the second point group corresponding to each point of the first point pair, and determines each point of the second point pair based on a center of gravity (64) of one or more points existing within a predetermined distance from the determined coordinates.

[0268] [Configuration 6] The information processing device according to any one of configurations 1 to 5, further comprising a determination unit (S1;152) that determines the correspondence relationship based on a difference between a position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the first camera, and a position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the second camera.

[0269] [Configuration 7] generating a composite image by combining (S2) a first image obtained by capturing an image of a reference object (20) on which a reference marker (22) is arranged with a first camera (10C) and a second image obtained by capturing an image of the reference object with a second camera (10T) on the same coordinate system based on a correspondence relationship between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; generating (S3) a first point cloud based on design information (30) of the reference object and using the reference marker as a reference; generating a second point cloud based on the reference marker from the composite image (S2); for a first pair of points (40) selected from the first set of points, searching for a second pair of points (60) in the second set of points that corresponds to the first pair of points (S4); Calculating the accuracy of the correspondence based on the geometric dimensions of the first point pair and the second point pair (S5); and visualizing (S9, S11, S12) the accuracy of the correspondence calculated for each measurement position in association with the respective measurement positions when the reference object is placed at multiple measurement positions in space.

[0270] [Configuration 8] A computer (100) generating a composite image by combining (S2) a first image obtained by capturing an image of a reference object (20) on which a reference marker (22) is arranged with a first camera (10C) and a second image obtained by capturing an image of the reference object with a second camera (10T) on the same coordinate system based on a correspondence relationship between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; generating (S3) a first point cloud based on design information (30) of the reference object and using the reference marker as a reference; generating a second point cloud based on the reference marker from the composite image (S2); For a first point pair (40) selected from among the first point group, searching for a second point pair (60) corresponding to the first point pair in the second point group (S4); Calculating the accuracy of the correspondence based on the shape dimensions of the first point pair and the second point pair (S5); An information processing program that, when the reference object is arranged at a plurality of measurement positions in space, associates and visualizes the accuracy of the correspondence calculated for each measurement position with each measurement position (S9, S11, S12).

[0271] <S. Advantage> According to the information processing system 1 according to the present embodiment, by performing registration using a reference object whose shape and the like are defined in advance, the accuracy of the homogeneous transformation matrix that is the result of registration can be evaluated. Since the evaluated accuracy is based on the absolute error, the safety design of the robot can be ensured.

[0272] According to the information processing system 1 according to the present embodiment, by arranging a reference object at a plurality of measurement positions in the target space and calculating the accuracy, an accuracy spatial distribution can be generated. By using the accuracy spatial distribution, it is possible to streamline the optimization of the camera position and the consideration of the necessity of adding a camera.

[0273] It should be considered that all the embodiments disclosed this time are illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Reference Numerals

[0274] 10 camera, 10C reference camera, 10T target camera, 12, 12C, 12T camera coordinate system, 14C, 14T field of view, 20, 20A, 20B, 20C, 20D, 20E reference object (object), 21 CAD canonical coordinate system, 22 marker, 24 effective surface, 26 marker coordinate system, 28 cutout, 29 chamfer, 30 CAD information, 32 3D point cloud, 34 target area, 36 CAD-derived point cloud, 40, 60 key point pair, 42, 62 key point, 50 ROI, 56 ROI point cloud, 64 center of gravity, 100 information processing device, 102 processor, 104 memory, 106 input interface, 108 input device, 110 display interface, 112 display, 114 camera interface, 116 network controller, 120 storage, 122 OS, 124 information processing program, 150 synthesis unit, 152 determination unit, 154 first point cloud generation unit, 156 second point cloud generation unit, 158 search unit, 160 evaluation unit, 162 calculation unit, 164 output unit, 166 optimization unit, 200 output screen, 210, 220 information, 212, 222 statistical information, 214, 224 histogram, 250, 252, 254 square pyramid, 260 main pillar, 261, 262, 263, 264, 265 plane plate, 270 synthesis point cloud, 271, 273, 274, 275 support member, 282, 284 area.

Claims

1. a synthesis unit that synthesizes a first image obtained by capturing an image of a reference object, on which a reference marker is arranged, with a first camera and a second image obtained by capturing an image of the reference object with a second camera, on the same coordinate system based on a correspondence relationship between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera, to generate a synthetic image; a first point cloud generating unit that generates a first point cloud based on design information of the reference object and using the reference marker as a reference; a second point cloud generation unit that generates a second point cloud based on the reference marker from the composite image; a search unit that searches for a second point pair in the second point cloud that corresponds to a first point pair selected from the first point cloud; a calculation unit that calculates accuracy of the correspondence relationship based on geometric dimensions of the first point pair and the second point pair; and an output unit that, when the reference object is placed at a plurality of measurement positions in space, visualizes the accuracy of the correspondence calculated for each measurement position in association with each measurement position.

2. The information processing device according to claim 1 , wherein the output unit outputs an image in which the value of the accuracy of the correspondence is expressed by at least one of shading and color.

3. the first point cloud generation unit extracts, from the point clouds generated based on the design information, a point cloud included in a first region determined based on the reference marker as the first or second point cloud; The information processing device according to claim 1 , wherein the second point cloud generation unit extracts, from the point clouds included in the composite image, a point cloud included in a second region determined based on the reference marker, as the second point cloud.

4. The information processing apparatus according to claim 3 , wherein each of the first region and the second region is determined based on a surface of the reference object on which the reference marker is arranged.

5. The information processing device according to any one of claims 1 to 4, wherein the search unit determines coordinates in the second point cloud corresponding to each point of the first point pair, and determines each point of the second point pair based on a center of gravity of one or more points that exist within a predetermined distance from the determined coordinates.

6. The information processing device according to any one of claims 1 to 4, further comprising a determination unit that determines the correspondence based on a difference between a position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the first camera and a position and orientation of the reference marker calculated based on an image obtained by capturing an image of the reference object with the second camera.

7. generating a composite image by combining a first image obtained by capturing an image of a reference object, on which a reference marker is arranged, with a first camera and a second image obtained by capturing an image of the reference object with a second camera, on the same coordinate system based on a correspondence relationship between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; generating a first point cloud based on design information of the reference object and using the reference marker as a reference; generating a second point cloud based on the reference marker from the composite image; For a first pair of points selected from the first cloud of points, searching for a second pair of points in the second cloud of points that corresponds to the first pair of points; calculating an accuracy of the correspondence based on geometric dimensions of the first point pair and the second point pair; and visualizing, when the reference object is placed at a plurality of measurement positions in space, the accuracy of the correspondence calculated for each measurement position in association with each measurement position.

8. On the computer, generating a composite image by combining a first image obtained by capturing an image of a reference object, on which a reference marker is arranged, with a first camera and a second image obtained by capturing an image of the reference object with a second camera, on the same coordinate system based on a correspondence relationship between a first camera coordinate system of the first camera and a second camera coordinate system of the second camera; generating a first point cloud based on design information of the reference object and using the reference marker as a reference; generating a second point cloud based on the reference marker from the composite image; For a first pair of points selected from the first cloud of points, searching for a second pair of points in the second cloud of points that corresponds to the first pair of points; calculating an accuracy of the correspondence based on geometric dimensions of the first point pair and the second point pair; and visualizing, when the reference object is placed at a plurality of measurement positions in space, the accuracy of the correspondence calculated for each measurement position in association with each measurement position.

Images