Information processing system, information processing method, and program

Abstract

To make processing of capturing distance images of an object more efficient while maintaining accuracy of measurement, when measuring the object based on distance images using a principle of triangulation.SOLUTION: With respect to a measuring device that performs three-dimensional measurement of an object, an information processing system that determines imaging positions of each of M distance images to be used for measuring the object includes: imaging control means for obtaining N distance images of an object for teaching captured by using a distance image sensor from each of N imaging positions (N>M); configuration means for constructing a plurality of integrated data from each of a plurality of different combinations of the N distance images; and determination means for calculating accuracy representing a matching degree of each of the plurality of integrated data with the teaching object, and determining M imaging positions from the N imaging positions based on the accuracy of each of the plurality of integrated data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there exist inspection systems in which a sensor measures an object and inspects the shape and size of the object.

[0003] Patent document 1 discloses a technology for measuring an object by suppressing measurement variation errors by having a sensor detect a line of light (line-shaped measurement light) irradiated onto the object at multiple positions where the relative positions between the sensor and the object are different from one another. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-220338 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 1 requires expensive components for irradiating and detecting line light with high accuracy, which makes the entire inspection system expensive. In contrast, if the inspection system is configured to measure an object based on a distance image using the principle of triangulation instead of detecting line light, the inspection system can be configured more inexpensively. When the inspection system measures an object based on a distance image, the inspection system integrates multiple distance images to generate integrated data, which is one piece of three-dimensional data, as a measurement result of the object, and inspects the object based on the integrated data.

[0006] On the other hand, when an inspection system measures an object based on distance images, there is a problem that the accuracy of each distance image is not very high, and noise is likely to occur in the integrated data that is the measurement result of the object. For this reason, in the past, in order to reduce such noise, it was necessary to image the object from a large number of imaging positions, and it took a lot of processing (time) to capture (acquire) the distance images for measuring the object.

[0007] Therefore, the present invention aims to provide a technology that, when measuring an object based on a distance image using the principle of triangulation, improves the efficiency of the processing of capturing a distance image of the object while maintaining measurement accuracy. [Means for solving the problem]

[0008] In order to achieve the above object, the present invention employs the following configuration.

[0009] That is, an information processing system according to one aspect of the present invention is an information processing system for determining imaging positions of M distance images to be used for measuring an object using a measuring device that performs three-dimensional measurement of the object using a plurality of distance images acquired at different imaging positions by a distance image sensor that acquires distance images using the principle of triangulation, the information processing system including: an imaging control means for acquiring N distance images of a teaching object using the distance image sensor from each of N imaging positions (N>M); a configuration means for configuring a plurality of integrated data from each of a plurality of mutually different combinations of the N distance images; and a determination means for calculating an accuracy representing the degree of match of each of the plurality of integrated data with the teaching object and determining M imaging positions from the N imaging positions based on the accuracy of each of the plurality of integrated data. The step includes steps (Step 1 and Step 2 in FIG. 7A).

[0010] With this configuration, even if the number of imaging positions for imaging the object is small (the value of M is small), the object can be measured with high accuracy. That is, when measuring an object (workpiece) using distance images, the number of distance images taken of the object can be reduced while maintaining high measurement accuracy. Note that in the above, the teaching object and the object are, for example, objects with the same model number.

[0011] In the above information processing system, each of the multiple integrated data may be point cloud data that represents an object in three-dimensional space using a point cloud, and the accuracy may be a value corresponding to the variation of the point cloud for the teaching object.

[0012] In the above information processing system, the variation of the point cloud may be based on the variation of each point of the integrated data relative to a preset reference plane and the variation of each point of the integrated data relative to the surface of the teaching object. According to this configuration, since it is only necessary to know the reference plane and the surface of the teaching object in advance, the variation of the point cloud can be determined without the user having to input detailed information about the teaching object. In other words, the burden on the user in determining the imaging position of M point can be reduced.

[0013] The information processing system may further include a path generating means for generating a movement path for changing the position of the range image sensor to the M imaging positions so that the range image sensor can complete imaging of the object from the M imaging positions in the shortest time. With this configuration, a path for the range image sensor to move to the M imaging positions more efficiently can be generated, so that imaging processing can be performed more efficiently when measuring the object (Step 3 in FIG. 7A).

[0014] In the above information processing system, the determining means may determine the M imaging positions based on the accuracy and a time required for the range image sensor to capture an image. With this configuration, it is possible to achieve a good balance between the measurement accuracy when measuring an object and the time required for capturing an image.

[0015] The information processing system may further include an acquisition means for acquiring a plurality of tentative movement paths, each of which includes the N imaging positions and is a movement path of the range image sensor, and for each of the plurality of tentative movement paths, the imaging control means acquires the N distance images, the configuration means configures a plurality of the integrated data, and the determination means determines candidates for the M imaging positions based on the accuracy of each of the plurality of integrated data, and the determination means selects one tentative movement path from the plurality of tentative movement paths and determines the M imaging position candidates corresponding to the selected tentative movement path as the M imaging positions. With this configuration, one optimal actual movement path can be generated from many tentative movement paths (the actual imaging positions of M points can be determined), and therefore the actual imaging positions of M points that can be measured with higher accuracy can be determined (FIG. 8).

[0016] In the above information processing system, an upper limit of the time required for the range image sensor to capture an image may be set in advance, and the M may be a maximum number of range images that the range image sensor can acquire before reaching the upper limit of the time required for the range image sensor to capture an image. With this configuration, it is possible to achieve a good balance between the measurement accuracy when measuring an object and the time required for capturing an image.

[0017] In the above information processing system, the information processing system further comprises a control means for controlling the display means to display a graph showing a relationship between the number of the imaging positions determined by the determination means and the accuracy, The number M may be a number according to a user input input after the graph is displayed. According to this configuration, the displayed graph allows the user to easily understand the relationship between the number of imaging positions and the measurement accuracy. Therefore, even when the user determines the number M of imaging positions, the user can easily determine M.

[0018] In the above information processing system, the control means may control the display means to display a graph showing the relationship between the time required for the range image sensor to capture an image and the number of imaging positions and the accuracy determined by the determination means. According to this configuration, the displayed graph allows the user to easily understand the relationship between the number of imaging positions, the measurement accuracy, and the time required for imaging. Therefore, the user can determine an appropriate number M of imaging positions by further considering the time required for imaging.

[0019] In the above information processing system, the determination means may select the most accurate distance image from the N distance images, and further select one distance image from the unselected distance images based on the accuracy of each integrated data obtained by integrating each unselected distance image from the N distance images with all the selected distance images from the N distance images, repeating this process until the M distance images are selected, and determine the M imaging positions corresponding to the selected M distance images from the N imaging positions. According to this configuration, by using greedy selection, if K distance images have been selected, the K+1th distance image can be efficiently selected, and therefore the imaging positions of M points can be determined more efficiently. This also makes it possible to simplify the configuration of the information processing system.

[0020] In the above information processing system, the imaging control means may further obtain distance images of the teaching object captured by using the distance image sensor from peripheral positions that are in a certain relative positional relationship with the N imaging positions, and the configuration means may configure the multiple integrated data by integrating N image sets, each set being a distance image captured from one of the N imaging positions and a distance image captured from the peripheral position of the one imaging position, in multiple different combinations. With this configuration, the imaging position of M point can be determined based on distance images captured from imaging positions around the imaging position of N point. Therefore, even if there is a slight difference in imaging position between when the object is imaged and when the teaching object is imaged, the object can be measured with stable accuracy.

[0021] In the above information processing system, the imaging control means may further obtain distance images by individually imaging W teaching objects including the teaching object from the N imaging positions using the distance image sensor, and the configuration means may configure the plurality of integrated data by integrating N image sets, each set being W distance images individually imaging the W teaching objects from one imaging position among the N imaging positions, in a plurality of different combinations. According to this configuration, the imaging position of M point can be determined based on distance images of two or more teaching objects from the imaging positions of N points. Therefore, even if there are individual differences in the teaching objects, the objects can be measured with stable accuracy.

[0022] In the above information processing system, the information processing system may have the measuring device, and the measuring device may acquire integrated data as a measurement result of the object by integrating the M distance images captured by the distance image sensor from the M imaging positions.

[0023] The present invention may be understood as an apparatus having at least a part of the above means, or as an electronic device, an imaging system, an inspection system, a measurement system, an information processing device, an inspection device, or a measurement device. The present invention may also be understood as a control method, an information processing method, a measurement method, or an inspection method including at least a part of the above processing. The present invention may also be understood as a method for realizing such a method. The present invention can also be understood as a program for performing the above-mentioned operations, or a recording medium (storage medium) on which the program is non-temporarily recorded. Note that the above-mentioned means and processes can be combined with each other as much as possible to constitute the present invention. Effect of the Invention

[0024] According to the present invention, when an object is measured based on a distance image obtained using the principle of triangulation, the accuracy of the measurement can be maintained while improving the efficiency of the process of capturing the distance image of the object. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a configuration diagram of an inspection system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing the internal configuration of the information processing device according to the first embodiment. [Diagram 3] FIG. 3A is a flowchart showing a process in a decision phase according to the first embodiment, and FIG. 3B is a flowchart showing a process in an inspection phase according to the first embodiment. [Figure 4] 4A to 4C are diagrams illustrating the variation error according to the first embodiment. [Diagram 5] FIG. 5A is a flowchart showing the process of the decision phase according to the fifth embodiment, and FIG. 5B is a flowchart showing the process of the decision phase according to the sixth embodiment. [Figure 6] 6A and 6B are graphs according to the fifth embodiment. [Figure 7] 7A and 7B are diagrams for explaining the process of the decision phase according to the first embodiment. [Figure 8] FIG. 8 is a diagram for explaining the process of the decision phase according to the third embodiment. [Figure 9] FIG. 9 is a diagram illustrating integrated data according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating the effects according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

[0027] <Application Examples> In the following, an inspection system 1 that inspects an object using multiple distance images obtained from a distance image sensor 20 that acquires distance images of the object will be described. The distance image sensor 20 acquires distance images by capturing an image of the object using the principle of triangulation. The processing of the inspection system 1 includes processing of a determination phase and an inspection phase. In the inspection phase, the inspection system 1 measures the object by acquiring multiple distance images by capturing images of the object while changing the position of the distance image sensor 20 to each of the multiple imaging positions determined in the determination phase, and inspects the object according to the measurement results.

[0028] In addition, in multiple distance images acquired by the distance image sensor 20 from the same imaging position, sensor position-dependent coordinate noise due to the BRDF (Bidirectional Reflectance Distribution Function) characteristics of the object (workpiece) occurs as a measurement error, and similar noise occurs in each distance image. Therefore, the noise is also affected in integrated data, which is a measurement result obtained by integrating multiple distance images. On the other hand, as described in each of the following embodiments, when multiple distance images acquired by the distance image sensor 20 from multiple different imaging positions are integrated to form integrated data, the multiple distance images cancel each other's coordinate noise. Therefore, it is possible to inspect an object with high accuracy.

[0029] In the determination phase, the inspection system 1 determines (sets) multiple imaging positions at which the range image sensor 20 captures an image of an object in the inspection phase. Here, the inspection system 1 determines (sets) the multiple imaging positions based on the accuracy of integrated data obtained by integrating multiple range images captured from multiple candidate imaging positions.

[0030] According to this, in the inspection phase, when the number of imaging positions at which the object is imaged is small, Even if the distance image is not measured, the object can be measured with high accuracy. In other words, when measuring an object using a distance image, the processing for capturing the distance image of the object can be made more efficient while maintaining high measurement accuracy. Specifically, the number of captured images (inspection tact) in the inspection phase and the amount of calculation required for processing can be reduced.

[0031] <Embodiment 1> [Inspection system configuration] The configuration of an inspection system 1 according to the first embodiment will be described with reference to FIG. 1. The inspection system 1 is an information processing system that inspects an object. The inspection system 1 includes a determination phase and an inspection phase, and images of objects of the same type (for example, objects with the same product number) are captured in the determination phase and the inspection phase. Thus, an object for teaching (object for teaching) imaged in the determination phase is referred to as a workpiece 50A, and an object imaged (inspected) in the inspection phase is referred to as a workpiece 50B. The workpiece 50A and the workpiece 50B are collectively referred to as a workpiece 50. The workpiece 50A and the workpiece 50B are arranged in the same position and orientation.

[0032] The inspection system 1 includes a robot 10, a range image sensor 20, an information processing device 30, and a control server 40.

[0033] The robot 10 controls the position and posture of the range image sensor 20 by controlling its own posture (movement; angle of the joint of the arm). Here, the posture of the robot 10 is controlled by the information processing device 30. The range image sensor 20 is provided at the tip of the arm of the robot 10. Therefore, it may be considered that the robot 10 has the range image sensor 20. Note that, although the "position and posture" refers to the position and posture in this embodiment, it may be the position or the posture as long as no technical contradiction occurs. In addition, in the following, in cases where the posture corresponding to the position is uniquely determined once the position is determined, only the position will be mentioned and the posture will not be mentioned.

[0034] The distance image sensor 20 captures a distance image by capturing an image of the workpiece 50. Here, the distance image is an image in which each pixel indicates the distance to the subject. More specifically, in this embodiment, the distance image is point cloud data (three-dimensional data) in which each pixel represents a point indicating the presence or absence of an object at each distance from the distance image sensor 20. The distance image represents an object in a three-dimensional space by a point cloud (a set of multiple points). In this embodiment, the distance image sensor 20 captures the distance image by a so-called active stereo method. The active stereo method includes a method in which, for example, a projector projects an image of a pattern encoded temporally or spatially onto the subject, and the camera image and the projector image are associated by analyzing the image of the pattern captured by the camera, thereby identifying a three-dimensional position based on the principle of triangulation.

[0035] It should be noted that the distance image sensor 20 is not limited to capturing distance images using the active stereo method, and may capture distance images using any method, such as a passive stereo method or a phase shift method, as long as the method identifies a three-dimensional position based on the principles of triangulation.

[0036] The information processing device 30 generates integrated data, which is three-dimensional data of the workpiece 50B, based on the distance image of the workpiece 50B captured by the distance image sensor 20, and inspects the workpiece 50B from the integrated data. The information processing device 30 also determines an imaging position at which the distance image sensor 20 captures an image of the workpiece 50, and controls the distance image sensor 20 to be positioned at the imaging position. Note that in this embodiment, once the position of the distance image sensor 20 is determined, the attitude of the distance image sensor 20 is uniquely determined such that the optical axis of the distance image sensor 20 passes through the center (or any point) of the workpiece 50. Then, when the information processing device 30 moves the distance image sensor 20 to a certain imaging position, it controls the distance image sensor 20 to be in an attitude corresponding to the imaging position. This will change the position of 20.

[0037] The control server 40 controls a plurality of information processing devices 30 via a network 60. The control server 40 may have all or a part of the components that the information processing devices 30 have.

[0038] [Configuration of information processing device] The internal configuration of the information processing device 30 will be described with reference to the configuration diagram of Fig. 2. The information processing device 30 has a control unit 301, a storage unit 302, an information acquisition unit 303, an imaging control unit 304, a configuration unit 305, a determination unit 306, a path generation unit 307, an inspection unit 308, and an output unit 309.

[0039] The control unit 301 controls each component of the information processing device 30 in accordance with a program stored in the storage unit 302. Note that the processing of the control unit 301 may be executed by the control server 40.

[0040] The storage unit 302 stores information for each component to operate and a program to be executed by the control unit 301. The storage unit 302 stores information on a tentative movement path, which is a tentative path along which the range image sensor 20 moves in the determination phase, for example. The storage unit 302 also stores information on imaging positions (hereinafter referred to as tentative imaging positions) of N points (N is a natural number of 2 or more) on the tentative movement path, which are imaging positions at which the range image sensor 20 images the workpiece 50A. The storage unit 302 also stores specification information representing the shape and size of the workpiece 50 and the smoothness of the surface of the workpiece 50. The specification information may also be information on a three-dimensional model (CAD model) of the workpiece 50 or information on a plane model representing the range of a plane.

[0041] The information acquisition unit 303 acquires a range image of the workpiece 50 captured by the range image sensor 20 from the range image sensor 20. In addition, the information acquisition unit 303 acquires information on a tentative movement path, information on a tentative imaging position, and specification information in accordance with a user input or from the storage unit 302.

[0042] The imaging control unit 304 controls the operations of the robot 10 and the distance image sensor 20. Specifically, in the inspection phase, the imaging control unit 304 moves the distance image sensor 20 along a moving path (hereinafter referred to as the main moving path) or a provisional moving path along which the distance image sensor 20 moves. Note that the imaging control unit 304 controls the posture of the robot 10 to move the distance image sensor 20. Then, the imaging control unit 304 controls the distance image sensor 20 to image the workpiece 50 from a plurality of imaging positions corresponding to each of the provisional moving path or the main moving path. Note that the imaging control unit 304 may be able to change the position and posture of the workpiece 50.

[0043] The component unit 305 constructs integrated data of the workpiece 50 by integrating a plurality of distance images captured by the distance image sensor 20. In the present embodiment, the integrated data can be point cloud data that represents an object in a three-dimensional space by a point cloud (a set of a plurality of points), similar to the distance image as shown in FIG. 9. When the component unit 305 integrates a plurality of distance images to construct integrated data, for example, as shown in FIG. 9, the three-dimensional space including the workpiece 50 is divided into a plurality of spaces (in units of cubes of a predetermined size), and the positions of points indicating the presence of an object are averaged for each of the divided spaces in the plurality of distance images, whereby a point cloud in the integrated data can be obtained. Thereby, when noise with an average of zero is added to the point cloud coordinates, a point cloud with less noise can be obtained by integrating a plurality of distance images.

[0044] As in Step 2 of FIG. 7A, the determination unit 306 determines the imaging positions of M points (M is a natural number satisfying M < N) (hereinafter referred to as the main imaging positions), which are the positions where the distance image sensor 20 images the workpiece 50B in the inspection phase, from among the N provisional imaging positions. The determination unit 306 determines the main imaging positions of the M points based on the accuracy (measurement accuracy) of the integrated data constructed by the component unit 305. The measurement accuracy is a value representing the degree of coincidence between the workpiece 50A and the integrated data. Note that details of the method (selection method) for determining the main imaging positions will be described later.

[0045] The path generating unit 307 determines the actual movement path by connecting the actual imaging position of point M determined by the determining unit 306, as in Step 3 of FIG. 7A. In this embodiment, the path generating unit 307 determines the path that can image the workpiece 50 from the actual imaging position of point M in the shortest time (inspection takt) as the actual movement path. Any known optimization algorithm can be used to determine the path that can image the workpiece 50 from the actual imaging position of point M in the shortest time. Note that the path generating unit 307 may determine the path that can move the actual imaging position of point M in the shortest time or distance as the actual movement path, instead of the path that can image the workpiece 50 from the actual imaging position of point M in the shortest time.

[0046] The inspection unit 308 acquires, as a measurement result from the configuration unit 305, integrated data obtained by integrating M distance images of the workpiece 50B captured by the distance image sensor 20 from the main imaging position of point M. Then, the inspection unit 308 inspects the workpiece 50B based on the measurement result. For example, the inspection unit 308 inspects the workpiece 50B by comparing the shape of the workpiece 50 that can be determined from the specification information of the workpiece 50 with the shape of the workpiece 50B represented by the integrated data.

[0047] The output unit 309 outputs the inspection result (pass / fail inspection of the workpiece 50B) by the inspection unit 308. The output unit 309 may also output information on the actual imaging position of point M and information on the actual movement path.

[0048] In the embodiment, the processes of the determination phase and the inspection phase are performed by one device (information processing device 30), but the processes of each phase may be performed by different devices. For example, the process of the determination phase may be performed by a determination device (setting device), and the process of the inspection phase may be performed by an inspection device (measurement device). In this case, for example, an output unit of the determination device outputs information on the actual imaging position of point M and / or information on the actual movement path to the inspection device.

[0049] The information processing device 30 may be configured by a computer including, for example, a CPU (processor), a memory, a storage, and the like. In this case, the configuration shown in FIG. 2 is realized by loading a program stored in the storage into the memory and executing the program by the CPU. Such a computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or may be an embedded computer such as an on-board computer. Alternatively, all or a part of the configuration shown in FIG. 2 may be configured by ASIC, FPGA, or the like. Alternatively, all or a part of the configuration shown in FIG. 2 may be realized by cloud computing or distributed computing.

[0050] [Decision Phase] The process in the decision phase performed by the information processing device 30 will be described below with reference to the flowchart in Fig. 3A. Each process in the flowchart in Fig. 3A is realized by the control unit 301 executing a program stored in the storage unit 302.

[0051] In step S1001, the control unit 301 controls the information acquisition unit 303 to acquire setting information such as information on a tentative moving path, information on tentative imaging positions of N points, and specification information of the workpiece 50 from the storage unit 302. Note that the information acquisition unit 303 may acquire the setting information by a user input.

[0052] In step S1002, the control unit 301 controls the robot 10 and and controls the distance image sensor 20 to acquire N distance images capturing images of the workpiece 50A from the N tentative imaging positions. Specifically, the imaging control unit 304 controls the posture of the robot 10 so that the distance image sensor 20 moves along the tentative movement path, and controls the distance image sensor 20 to acquire a distance image every time the distance image sensor 20 reaches one of the N tentative imaging positions.

[0053] In step S1003, the control unit 301 controls the composition unit 305 and the determination unit 306 to determine M actual imaging positions (M < N) from N temporary imaging positions. In this embodiment, an example of determining M actual imaging positions using the concept of Greedy selection will be described. Greedy selection determines which one of the remaining N - K images will result in the best evaluation value when adding one more image in a situation where K images have already been selected, and the best one is added to obtain the selection result of K + 1 images. In this embodiment, the evaluation value is the measurement accuracy of the integrated data composed of the composition unit 305 from one or more distance images. The measurement accuracy of the integrated data is a value corresponding to the variation error D (variation) of the point cloud of the integrated data which is point cloud information. For example, the measurement accuracy can be the reciprocal of the variation error D. Details of the variation error D will be described later.

[0054] In this embodiment, first, the determination unit 306 selects the distance image with the highest accuracy among the N distance images. Next, the composition unit 305 composes N - 1 pieces of integrated data by integrating the selected distance image with each of the remaining N - 1 distance images. Then, the determination unit 306 selects, from the remaining N - 1 distance images, the distance image that results in the highest measurement accuracy of the integrated data when integrated with the selected distance image. As a result, two distance images are selected.

[0055] Subsequently, the composition unit 305 composes N - 2 pieces of integrated data by integrating the selected two distance images with each of the remaining N - 2 distance images. Then, the determination unit 306 selects, from the remaining N - 2 distance images, the distance image that results in the highest measurement accuracy of the integrated data when integrated with the selected two distance images. The determination unit 306 and the composition unit 305 repeat these processes until M distance images are selected. Then, the determination unit 306 determines the M temporary imaging positions corresponding to the selected M distance images as the M actual imaging positions.

[0056] By using greedy selection in this manner, once K distance images have been determined, the (K+1)th distance image can be determined, making it possible to determine the actual imaging position for point M more quickly than when the most suitable M distance images are selected using a brute force method.

[0057] In the following, an example of a method for calculating the variation error D corresponding to the measurement accuracy of the integrated data will be described with reference to Figs. 4A to 4C. Here, the determination unit 306 calculates the variation error D, which is the variation of the point cloud based on the measurement surface and the reference surface. Here, the "measurement surface" is the surface of the workpiece 50 to be measured. The "reference surface" is a fixed surface set in advance that serves as a reference when measuring the workpiece 50. The "reference surface" can be any surface, such as a base on which the workpiece 50 is placed. Also, the round points shown in Figs. 4A to 4C indicate points of the point cloud information, which is the integrated data.

[0058] The shape and position of each of the reference surface and the measurement surface may be determinable from the specification information. For example, the specification information itself may have information (plane information) on the shape and position of the reference surface and the measurement surface. In this case, the user is required to specify more information, which places a greater burden on the user, but the variation error D can be calculated with high accuracy. The specification information may have information on the shape and position of one surface and the smoothness of the other surface, or may have information on the smoothness of both surfaces. If the specification information has information on the smoothness of a surface, then by approximating each point of the integrated data, which is point cloud information, with a curved surface having that smoothness, the shape and position of that surface can be determined. Furthermore, the specification information may have information on the shape and position of each of the reference surface and the measurement surface. The edge information of the reference surface and the measurement surface may be included so that the shape and position of the edge can be determined from the specification information.

[0059] 4A is a diagram for explaining a method of calculating the variation error D when the measurement surface is a flat surface. When the measurement surface is a single surface, the determining unit 306 calculates the variance σ A 2 and the variance of the distance between the measurement surface and points located around the measurement surface σB 2 The sum of these is calculated as the variation error D. That is, the determining unit 306 calculates the variation error D based on the variation of the point cloud with respect to the reference surface and the variation of the point cloud with respect to the measurement surface.

[0060] 4B is a diagram for explaining a method of calculating the variation error D when the measurement surface is not flat (for example, curved). Here, when the measurement surface is two surfaces, the determination unit 306 calculates the variance σ A 2 and the variance σ of the distance between the first measurement surface and points located on the periphery of the first measurement surface. B 2 and the variance σ of the distance between the second measurement surface and points located on the periphery of the second measurement surface. C 2 The sum of these is calculated as the variation error D.

[0061] Note that even when only the shape and position of the reference surface can be known from the specification information (for example, when the specification information has only the shape and position of the reference surface) and the shape and position of the measurement surface cannot be known from the specification information, it is possible to calculate the variation error D. For example, the determining unit 306 may determine the variance of the distance between each point located around the reference surface and the reference surface as the variation error D.

[0062] 4C is a diagram for explaining a method of calculating the variation error D when an algorithm for calculating the dimensional measurement index from point cloud information is given. In this case, the determining unit 306 performs 3D model fitting for each of the reference surface and the measurement surface, and calculates the dimension z determined from the relationship between their relative positions. Then, the determining unit 306 calculates var[z], which is the variance of the dimension z, as the variation error D.

[0063] In the above, the measurement accuracy of the integrated data is calculated according to the variation error D, but any method may be used as long as the measurement accuracy of the integrated data can be calculated. For example, the determining unit 306 may perform a process similar to the inspection performed by the inspection unit 308, and use a value indicating the pass / fail of the inspection as the measurement accuracy. For example, the specification information may include information on a three-dimensional model of the work 50, and the determining unit 306 may determine the degree of match between the three-dimensional model represented by the specification information and the three-dimensional model represented by the integrated data constructed by the construction unit 305 as the measurement accuracy of the integrated data.

[0064] In step S1004, the control unit 301 controls the path generating unit 307 to generate a main movement path that connects the main imaging positions of the M points. Specifically, the path generating unit 307 generates a main movement path that can complete imaging at the main imaging positions of the M points in the shortest time (inspection takt). The path generating unit 307 can calculate the time required for movement between the main imaging positions of the M points based on information such as the operating speed and operating range of the robot 10. In addition, the path generating unit 307 can calculate the time required for imaging at each main imaging position based on information such as the exposure time of the range image sensor 20. Then, the path generating unit 307 can calculate the time required for completing imaging at the main imaging positions of the M points by summing up the total time required for imaging at each of the main imaging positions of the M points and the time required for movement between the main imaging positions of the M points.

[0065] Since the actual imaging position of point M is also included in the tentative movement path, the path generating unit 307 may use the tentative movement path as the actual movement path as shown in Fig. 7B. Using the tentative movement path as the actual movement path can simplify the processing in the determination phase. This is also effective in cases where the movement path doubles as part of the assembly operation of the robot 10 and the movement path cannot be changed from the predetermined path.

[0066] By determining the actual movement path and the actual imaging positions in this way, the inspection system 1 can generate integrated data with the measurement accuracy maintained even when using a small number of distance images captured from a small number of imaging positions in the inspection phase. Therefore, the number of processes is small and the workpiece 50B can be inspected with high accuracy.

[0067] [Inspection Phase] The process in the inspection phase performed by the information processing device 30 will be described below with reference to the flowchart in Fig. 3B. Each process in the flowchart in Fig. 3B is realized by the control unit 301 executing a program stored in the storage unit 302.

[0068] In step S2001, the control unit 301 controls the robot 10 and the distance image sensor 20 via the imaging control unit 304 to acquire M distance images of the workpiece 50B captured from the main imaging positions of points M. Specifically, the imaging control unit 304 controls the posture of the robot 10 so that the distance image sensor 20 moves along the main movement path, and controls the distance image sensor 20 to acquire a distance image every time the distance image sensor 20 reaches one of the main imaging positions of points M.

[0069] In step S2002, the control unit 301 controls the construction unit 305 to integrate the M distance images captured in step S2002 and generate integrated data of the workpiece 50B. As a result, the construction unit 305 acquires the integrated data as a measurement result of the workpiece 50B.

[0070] In step S2003, the inspection unit 308 is controlled to inspect the workpiece 50 based on the integrated data of the workpiece 50B.

[0071] The processing of the inspection phase may include an assembly process of the workpiece 50B. For example, the range image sensor 20 may be attached next to the gripping part of the robot 10, and imaging and inspection may be performed at the main imaging position at the position for picking up and assembling parts. This can reduce the overall time for inspection and assembly.

[0072] As described above, in this embodiment, the inspection system 1 narrows down the number of main imaging positions at which the range image sensor 20 captures images in the inspection phase based on the measurement accuracy of the integrated data. This makes it possible to inspect the workpiece 50B with high accuracy even when using a small number of range images captured by the range image sensor 20 from a small number of imaging positions in the inspection phase. This makes it possible to reduce the imaging process while maintaining the accuracy of inspection (measurement) in the inspection phase.

[0073] Furthermore, according to experiments conducted by the inventors, it was possible to significantly reduce the number of distance images taken to achieve the same variation error D, as shown in the graph of Fig. 10. Specifically, it was possible to generate integrated data with the same measurement accuracy as integrated data obtained by integrating distance images taken by the distance image sensor 20 from 20 arbitrary imaging positions, from distance images taken from 12 actual imaging positions determined as described in this embodiment.

[0074] It is also possible that the process of step S1004 is not performed in the determination phase, and the inspection system 1 uses the tentative movement path as the actual movement path in the inspection phase and captures an image of the workpiece 50B from the actual imaging position of point M.

[0075] In this embodiment, the imaging control unit 304 changes the imaging position by moving the distance image sensor 20 without moving the workpieces 50A and 50B. The imaging position does not have to be the absolute position of the distance image sensor 20, but may be the relative position of the distance image sensor 20 with respect to the workpiece 50. For this reason, the imaging control unit 304 The imaging control unit 304 may change the imaging position by moving the workpieces 50A and 50B without moving the distance image sensor 20. In other words, the actual movement path may be the path along which the workpiece 50B moves, and the tentative movement path may be the path along which the workpiece 50B moves. The imaging control unit 304 may also change the imaging position by moving the workpieces 50A and 50B as well as moving the distance image sensor 20. By moving both the distance image sensor 20 and the workpiece 50 in this way, the distance image sensor 20 can complete imaging from multiple imaging positions more quickly.

[0076] [Variation 1] In the first embodiment, the inspection system 1 determines the main imaging position of point M by using greedy selection, but it is also possible to generate a plurality of integrated data by integrating M range images in step S1003, and determine the M point corresponding to the integrated data with high measurement accuracy as the main imaging position from among them. The specific process of step S1003 according to this modification will be described below.

[0077] In step S1003, the control unit 301 controls the construction unit 305 to select M distance images from the N distance images, and construct integrated data (point cloud data) from the M distance images. Here, the construction unit 305 constructs integrated data from the M distance images for all combinations in which M distance images are selected from the N distance images. In other words, the construction unit 305: n C m = (n(n-1)···(n-m+1)) / (m(m-1)···1) pieces of integrated data. Note that the constructing unit 305 does not need to construct integrated data for all combinations, and may construct integrated data for only an arbitrary number of combinations.

[0078] Thereafter, the control unit 301 controls the determination unit 306 to determine the actual imaging position of point M from the provisional imaging positions of point N. Specifically, first, the determination unit 306 calculates the measurement accuracy of the multiple integrated data constructed by the construction unit 305. Then, the determination unit 306 determines the provisional imaging position of point M corresponding to the integrated data having the highest measurement accuracy among the multiple integrated data as the actual imaging position.

[0079] Note that in addition to the greedy selection and the decision method (combinatorial optimization method) according to this modified example, any combinatorial optimization method using a mixed integer quadratic programming problem or the like may be used.

[0080] In the following, referring to Fig. 9, an example is shown in which a mixed integer quadratic programming problem is used to reduce a problem to a {0,1} optimization problem when it is desired to find a combination solution more suitable than greedy selection. In order to concretely express the evaluation function, a method is considered in which the realized values ​​are divided into multiple spaces (voxel meshes) and the ensemble average is taken within those spaces. Fig. 9 shows a diagram in which all point clouds of N range images are arranged as they are in a three-dimensional space. Then, the point cloud contained in the kth voxel mesh in the nth range image is represented as p∈P k,n In other words, k is the index of the voxel mesh, and n is the index of N distance images. Let r be the representative coordinate of the kth voxel mesh. k Representative coordinate r k is the average value of the point cloud coordinates of the selected range image, and is expressed as the following formula (1). k can be said to be the representative coordinates of the k-th voxel mesh in the integrated data when the integrated data is constructed by integrating the selected distance images by the construction unit 305. For this reason, also in this example, it can be said that the construction unit 305 constructs multiple pieces of integrated data from multiple different combinations of the N distance images.

number

[0081] Here, x=(x1,x2,...,x N )T ∈{0,1} N is one of N range images. is a selection vector that is 1 if selected and 0 if not selected. Next, by averaging only the point groups that belong to the same distance image in advance, equation (1) can be transformed into the following equation (2).

number

[0082] And the representative coordinate r obtained above k The target coordinate value (target coordinate) is q k This target coordinate q k For example, the determination unit 306 may determine the intersection of the uv plane or smooth curved surface in FIG. 9 estimated by fitting with the voxel mesh as the target coordinate q k Alternatively, the coordinates of all N voxel meshes can be averaged to determine the target coordinate q k Using these, the representative coordinate r k and the target coordinate q k The sum of squares J of the fitting error with respect to the workpiece 50 can be expressed as the following formula (3). This sum of squares J of the fitting error can be used in the same way as the above-mentioned variation error D. In other words, the smaller the value of the sum of squares J of the fitting error, the higher the measurement accuracy, which indicates the degree of agreement with the workpiece 50.

number

[0083] In the above formula (3), if the number of elements selected from N is M (constant), the evaluation function J, which is the sum of squares of the fitting error, can be expressed as a quadratic formula as shown in the following formula (4).

number

[0084] Here, Q and b in formula (4) are numbers defined as in the following formulas (5) and (6).

number

[0085] Then, by introducing the constraint that the number of elements to be selected is a constant called M, the evaluation function J to be ultimately minimized can be expressed as the following equation (7).

number

[0086] Let J be the evaluation function x∈{0,1} N The problem of minimizing within the range is known as QUBO (Quadratic Unconstrained Binary Optimization), and a more optimal combination can be obtained by using dedicated hardware such as an Ising machine or a GPU, or a mathematical programming solver. Then, the determination unit 306 can select the range images where x=1 when the evaluation function J is minimized (when the measurement accuracy is maximized) as the M range images. Furthermore, the determination unit 306 can determine the tentative measurement positions of the M points corresponding to these M range images as the actual measurement positions of the M points.

[0087] [Variation 2] In addition, in the first embodiment, the N provisional imaging positions are assumed to be positions according to user input, but the inspection system 1 may determine the N provisional imaging positions. For example, while the determination unit 306 has determined the provisional imaging positions of p points (p < N), based on the imaging results of the workpiece 50A up to the p points, the next provisional imaging position of the (p + 1)-th point is repeatedly determined, whereby the N provisional imaging positions may be determined. For example, the determination unit 306 preferentially determines, as the provisional imaging position of the (p + 1)-th point, an imaging position within a predetermined range (for example, within a radius of 5 cm) from the provisional imaging position of the p-th point, an imaging position where no occlusion occurs in which the workpiece 50A is hidden by another object, and / or an imaging position that can image a range of the workpiece 50A that has not yet been imaged from the provisional imaging position of the p-th point.

[0088] According to this modification example, it is possible to select suitable N provisional imaging positions so as to avoid phenomena such as occlusion that cannot be known unless actually imaged.

[0089] <Second Embodiment> In the first embodiment, the inspection system 1 determines the M main imaging positions based only on the measurement accuracy (variation error D) of the integrated data in step S1003. However, the M main imaging positions may be determined based on the time required until the distance image sensor 20 completes imaging.

[0090] In the following, an example will be described in which the inspection system 1 selects the K+1th distance image in a state in which K distance images have been selected in step S1003. In this case, the configuration unit 305 first configures NK pieces of integrated data by integrating the K distance images and each of the remaining NK distance images. For each of the K+1 pieces of integrated data, the determination unit 306 calculates the measurement accuracy and the time (inspection tact) that the distance image sensor 20 can complete imaging in the shortest time at the K+1 virtual imaging position corresponding to the integrated data. Then, the determination unit 306 selects K+1 distance images corresponding to the integrated data having the largest value obtained by dividing the measurement accuracy by the inspection tact among the multiple integrated data. Note that the determination unit 306 may select the K+1 distance images using any value that increases with higher measurement accuracy and increases with shorter inspection tact, instead of the value obtained by dividing the measurement accuracy by the inspection tact.

[0091] This makes it possible to determine the main imaging position of point M that provides a good balance between the time required for imaging processing and the inspection accuracy, thereby making it possible to maintain high inspection accuracy while reducing the time required for imaging processing.

[0092] <Embodiment 3> In the first and second embodiments, one final movement route is generated from one provisional movement route in the determination phase, but one final movement route may be generated based on a plurality of provisional movement routes (three provisional movement routes, L=1 to 3, in the example of FIG. 8). Here, N provisional imaging positions are set in advance to correspond to each of the plurality of provisional movement routes. Note that N may be a different value for each provisional movement route.

[0093] Here, it is assumed that information on L tentative movement routes and tentative imaging positions corresponding to each of the L tentative movement routes is stored in the storage unit 302. In this case, in step S1001, the control unit 301 controls the information acquisition unit 303 to acquire information on the L tentative movement routes and information on the L×N tentative imaging positions corresponding thereto. In step S1002 (Step 1 in FIG. 8), the control unit 301 controls the imaging control unit 304 to capture images of the workpiece 50A from the L×N tentative imaging positions by the range image sensor 20.

[0094] In step S1003 (Step 2 in FIG. 8), the control unit 301 controls the configuration unit 305 and the determination unit 306 to determine candidates for the actual imaging position of M points for each of the L tentative movement routes by the same method as in embodiment 1. Here, the configuration unit 305 configures integrated data by integrating range images captured from the candidates for the actual imaging position of M points corresponding to each of the L tentative movement routes.

[0095] In step S1004, the control unit 301 controls the route generating unit 307 to select a provisional movement route corresponding to the integrated data with the highest measurement accuracy from among the integrated data corresponding to each of the L provisional movement routes. Then, the route generating unit 307 generates a final movement route from M points of the selected provisional movement route. Note that the route generating unit 307 may use the selected provisional movement route as the final movement route as it is (Step 3 in FIG. 8). Furthermore, the determination unit 306 determines the candidate for the final imaging position of M point corresponding to the selected provisional movement route as the final imaging position of M point.

[0096] According to the third embodiment, since it is possible to generate one optimal actual movement path from many provisional movement paths (it is possible to determine the actual imaging position of point M), it is possible to determine the actual imaging position of point M that allows inspection with higher accuracy. Therefore, it is expected that the inspection accuracy of the workpiece 50B will be further improved.

[0097] <Embodiment 4> In the first embodiment, the value M, which is the number of actual imaging positions, is a (preset) value according to a user input, but the inspection system 1 may determine the value M, which is the number of actual imaging positions. In the fourth embodiment, it is assumed that an upper limit of the time required for imaging in the inspection phase (upper tact time limit T; threshold value) is set in advance. Then, the inspection system 1 determines the number of distance images that maximizes the corresponding inspection tact time within a range that does not reach the upper tact time limit T, as the number M, which is the number of actual imaging positions.

[0098] Here, only the process of step S1003 is different from that of the first embodiment, so the process instead of step S1003 will be described.

[0099] In the fourth embodiment, in a process instead of step S1003, the control unit 301 controls the determination unit 306 to select one range image at a time from the N range images by greedy selection. Each time the determination unit 306 selects one range image, the determination unit 306 determines whether the inspection tact, which is the shortest time required to connect the virtual imaging positions corresponding to all the range images already selected in the shortest time, is equal to or less than the tact upper limit value T. If the inspection tact is equal to or less than the tact upper limit value T, the determination unit 306 selects another range image and repeats the process of comparing the inspection tact with the upper limit tact. On the other hand, if the inspection tact is greater than the tact upper limit value T, the determination unit 306 determines whether the last (here, Kth) selected range image is selected. and determines the K-1 provisional imaging positions corresponding to the 1st to K-1st selected distance images as the K-1 actual imaging positions. That is, the determination unit 306 determines this K-1 as the value M, which is the number of actual imaging positions.

[0100] According to this embodiment, even if there is a time constraint in the imaging process of the inspection phase, a suitable actual imaging position can be determined. That is, even if there is a time constraint in the imaging process of the inspection phase, it is possible to realize inspection of the workpiece 50B with high accuracy. In addition, the user does not need to worry about what value M of the actual imaging position should be set to in advance.

[0101] <Embodiment 5> In the fourth embodiment, the inspection system 1 determines the number M of actual imaging positions according to the upper tact time limit T. However, the relationship between the inspection tact time and the variation error D may be shown to the user, and the user may determine the value of the number M of actual imaging positions based on that relationship.

[0102] That is, as shown in FIG. 5A, the processes of steps S1010 to S1012 for determining the number M may be performed between steps S1002 and S1003.

[0103] In step S1010, the control unit 301 controls the determination unit 306 to calculate the respective variation error D (measurement accuracy) when the value of the number M of main imaging positions corresponds to any of the arbitrary ranges, and the value of the inspection tact corresponding to the variation error D. This can be realized, for example, by calculating the variation error D of integrated data obtained by integrating all the selected distance images each time a distance image is selected in the greedy selection, and calculating the inspection tact corresponding to the integrated data (all the selected distance images).

[0104] In step S1011, the control unit 301 displays a graph on a display unit (not shown) showing the relationship between the value of the number M of actual imaging positions (the number of images captured during inspection) and the variation error D (measurement accuracy) as shown in Fig. 6A. The control unit 301 may also display a graph on the display unit showing the relationship between the value of the number M of actual imaging positions, the variation error D (measurement accuracy), and the inspection tact time as shown in Fig. 6B.

[0105] In step S1012, the control unit 301 accepts a user input and determines a value according to the user input as the number M of main imaging positions.

[0106] According to this embodiment, a graph showing the relationship between the value of the number M of main imaging positions and the variation error D (measurement accuracy) is displayed, so that the user can understand the measurement accuracy corresponding to the number M of main imaging positions, and can easily understand the influence of the number M of main imaging positions on the inspection. Furthermore, when the graph shown in FIG. 6B is displayed, the user can determine a satisfactory balance between the inspection tact and the measurement accuracy after getting an overview of the measurement accuracy that can be achieved with a certain inspection tact. Therefore, the user can easily and appropriately determine the number M of main imaging positions.

[0107] <Embodiment 6> In embodiment 1, the inspection system 1 determined the actual imaging position of point M based on a distance image of point N from the temporary imaging position of point N, whereas in embodiment 6, the actual imaging position of point M is determined based on distance images from the temporary imaging position of point N and its surrounding positions.

[0108] In the sixth embodiment, instead of the processes of steps S1002 and S1003 in the decision phase of Fig. 3A according to the first embodiment, the processes of steps S1022 and S1023 are performed as shown in Fig. 5B. Other than that, the sixth embodiment is similar to the first embodiment, and therefore a detailed description thereof will be omitted.

[0109] In step S1022, the control unit 301 controls the imaging control unit 304 to image the workpiece 50A from the N tentative imaging positions and the peripheral positions of each tentative imaging position using the distance image sensor 20. Specifically, the imaging control unit 304 causes the distance image sensor 20 to image the workpiece 50A from the tentative imaging position and the q-point position (peripheral position of the q-point) around the tentative imaging position for each of the N tentative imaging positions. As a result, the distance image sensor 20 acquires N(q+1) distance images obtained by imaging the workpiece 50A from the N(q+1)-point position. Note that each tentative imaging position and the peripheral position corresponding to the tentative imaging position have a constant (identical) relative positional relationship. In other words, the relative positional relationship between the first tentative imaging position and the peripheral position of the q-point corresponding to the first tentative imaging position is the same as the relative positional relationship between the second tentative imaging position and the peripheral position of the q-point corresponding to the second tentative imaging position.

[0110] In step S1023, the control unit 301 controls the configuration unit 305 and the determination unit 306 to determine the actual imaging position of M point from the provisional imaging positions of N points. The configuration unit 305 sets an image set in which, for each of the N distance images, the distance image and a distance image captured from a peripheral position of q point of the provisional measurement position where the distance image was captured are combined into one set. At this time, the configuration unit 305 configures a plurality of integrated data from each of a plurality of different combinations of the N image sets (a set of a distance image captured from one imaging position and q distance images captured from peripheral positions of the imaging position). Then, the determination unit 306 selects M image sets from the N image sets based on the measurement accuracy of a plurality of integrated data obtained by integrating a plurality of image sets using any combination optimization method as described above. In this case, the determination unit 306 can determine the provisional measurement position of M point corresponding to the selected M image sets as the actual measurement position of M point.

[0111] In the following, as a specific example, an example of determining M point using greedy selection will be described, as in embodiment 1. However, it differs from embodiment 1 in that distance images captured from peripheral positions of the virtual imaging position of N point are also used.

[0112] Here, a method for selecting the (K+1)th distance image when K distance images have been selected will be described. The K distance images are distance images captured from K point among the tentative imaging positions of N points. (1) First, the configuration unit 305 configures (NK)(q+1) pieces of integrated data by integrating K image sets corresponding to the K distance images and distance images captured from the tentative imaging positions of the remaining NK points and distance images captured from the positions of the (NK)q points around the NK points. (2) Next, the determination unit 306 calculates the measurement accuracy for each of the (NK)(q+1) pieces of integrated data. (3) After that, for each tentative imaging position of the NK points, the determination unit 306 calculates the average value of the measurement accuracy of the integrated data corresponding to the tentative imaging position and the measurement accuracy of the q pieces of integrated data corresponding to the peripheral positions of the tentative imaging position as the average measurement accuracy. The average measurement accuracy for the tentative imaging position of the NK points can be regarded as the measurement accuracy of the NK pieces of integrated data obtained by integrating each of the image sets of the NK points corresponding to the tentative imaging positions of the NK points and the K image sets. (4) Finally, the determining unit 306 selects, as the (K+1)-th distance image, the distance image captured from the virtual imaging position corresponding to the largest average measurement accuracy among the average measurement accuracy for the virtual imaging positions of the NK points.

[0113] The determination unit 306 selects M distance images in this manner, and determines the temporary imaging positions of the M points corresponding to the selected M distance images as the actual imaging positions of the M points.

[0114] Even if the workpiece 50 is imaged from the same imaging position in the determination phase and the inspection phase, the position of the workpiece 50 may shift slightly between the two phases, which may result in a large difference in measurement accuracy. By determining the actual imaging position based on distance images captured from peripheral positions of the provisional imaging position as in the sixth embodiment, it is possible to reduce errors in measurement accuracy caused by the positional shift of the workpiece 50 between the determination phase and the inspection phase. Therefore, it is possible to maintain a stable and high level of measurement accuracy during the inspection phase.

[0115] [Variation 3] In the sixth embodiment, the inspection system 1 determines the actual imaging position of M point based on distance images captured by the distance image sensor 20 from the provisional imaging position of N point and imaging positions around it. In this modification, the inspection system 1 may use distance images captured individually from the provisional imaging positions of N points of each of W workpieces 50 (teaching objects) including the workpiece 50A, instead of distance images captured from the provisional imaging positions of N point and imaging positions around it. Note that each of the W workpieces 50 other than the workpiece 50A is arranged in the same position and orientation as the workpiece 50A.

[0116] That is, in the third modification, the configuration unit 305 sets an image set combining W range images obtained by individually capturing W workpieces 50 for each of the N range images. The determination unit 306 may then use any combination optimization method to select M image sets from the N image sets based on the measurement accuracy of a plurality of integrated data obtained by integrating a plurality of image sets in a plurality of different combinations. In this case, the determination unit 306 can determine the tentative measurement position of the M point corresponding to the selected M image sets as the actual measurement position of the M point.

[0117] By using distance images captured of multiple workpieces 50 (teaching objects) that are different from one another, even if there is a characteristic defect in the workpiece 50A, for example, it is possible to cancel it out by using other workpieces 50. In other words, individual variations present in the workpieces 50 can be suppressed.

[0118] The interpretation of the claims is not limited solely to the matters described in the embodiments. The interpretation of the claims also includes the scope of the claims that is described in such a way that a person skilled in the art can recognize that the problem to be solved by the invention can be solved, taking into account the common general technical knowledge at the time of filing.

[0119] (Appendix 1) A measuring device for performing three-dimensional measurement of an object (50B) using a plurality of distance images acquired at different imaging positions by a distance image sensor (20) that acquires distance images using the principle of triangulation, An information processing system (1) for determining imaging positions of M distance images to be used for measuring the object (50B), an imaging control means (304) for acquiring N distance images by imaging an object for teaching (50A) from each of N imaging positions (N>M) using the distance image sensor (20); A construction means (305) for constructing a plurality of integrated data from each of a plurality of different combinations of the N distance images; a determination means (306) for calculating an accuracy representing a degree of match between each of the plurality of integrated data and the teaching object (50A) and determining M imaging positions from among the N imaging positions based on the accuracy of each of the plurality of integrated data; An information processing system (1).

[0120] (Appendix 2) A measuring device for performing three-dimensional measurement of an object (50B) using a plurality of distance images acquired at different imaging positions by a distance image sensor (20) that acquires distance images using the principle of triangulation, An information processing method for determining imaging positions of M distance images to be used for measuring the object (50B), Using the distance image sensor (20) from each of N imaging positions (N>M), An imaging control step (S1002) for acquiring N distance images of an object for shooting (50A); A construction step (S1003) of constructing a plurality of integrated data from each of a plurality of different combinations of the N distance images; A determination step (S1003) of calculating an accuracy representing a degree of match between each of the plurality of integrated data and the teaching object, and determining M imaging positions from among the N imaging positions based on the accuracy of each of the plurality of integrated data; 13. An information processing method comprising: [Explanation of symbols]

[0121] 1: Inspection system, 10: Robot, 20: Range image sensor, 30: information processing device, 40: control server, 50: work, 60: network, 301: control unit, 302: storage unit, 303: information acquisition unit, 304: imaging control unit, 305: configuration unit, 306: Determination unit, 307: Path generation unit, 308: Inspection unit, 309: Output section

Claims

1. A measuring device for performing three-dimensional measurement of an object using a plurality of distance images acquired at different imaging positions by a distance image sensor that acquires distance images using the principle of triangulation, An information processing system for determining imaging positions of M distance images to be used for measuring the object, an imaging control means for acquiring N distance images of a teaching object from N imaging positions (N>M) using the distance image sensor; a configuration means for configuring a plurality of integrated data from each of a plurality of different combinations of the N distance images; a determination means for calculating an accuracy representing a degree of matching between each of the plurality of integrated data and the teaching object, and determining M imaging positions from among the N imaging positions based on the accuracy of each of the plurality of integrated data; An information processing system comprising:

2. each of the plurality of integrated data is point cloud data representing an object in a three-dimensional space by a point cloud; The accuracy is a value corresponding to the variation of the point cloud with respect to the teaching object.

2. The information processing system according to claim 1 .

3. The variation of the point cloud is based on the variation of each point of the integrated data relative to a preset reference surface and the variation of each point of the integrated data relative to a surface of the teaching object; 3. The information processing system according to claim 2.

4. a path generating unit that generates a movement path for changing the position of the range image sensor to the M image capturing positions so that the range image sensor can complete capturing images of the object from the M image capturing positions in a shortest time.

4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

5. the determining means determines the M imaging positions based on the accuracy and a time required for the range image sensor to capture an image.

5. The information processing system according to claim 1, wherein the information processing system is a data processing system.

6. The method further includes: acquiring a plurality of tentative movement paths, each of which includes the N image capturing positions and is a movement path of the range image sensor; For each of the plurality of provisional movement paths, The imaging control means acquires the N distance images, The constructing means constructs the plurality of integrated data, The determining means determines candidates for the M imaging positions based on accuracy of each of the plurality of integrated data; the determining means selects one tentative movement route from the plurality of tentative movement routes, and determines the M imaging position candidates corresponding to the selected tentative movement route as the M imaging positions; 6. The information processing system according to claim 1,

7. an upper limit of the time required for the range image sensor to capture an image is set in advance; M is the maximum number of range images that the range image sensor can acquire before reaching an upper limit of the time required for the range image sensor to capture an image.

7. The information processing system according to claim 1, wherein the information processing system is a data processing system.

8. a control unit that controls a display unit to display a graph showing a relationship between the number of the imaging positions determined by the determination unit and the accuracy; The M is a number according to a user input inputted after the graph is displayed.

7. The information processing system according to claim 1, wherein the information processing system is a data processing system.

9. the control means controls the display means to display a graph showing a relationship between a time required for the range image sensor to capture an image, the number of the imaging positions determined by the determination means, and the accuracy.

9. The information processing system according to claim 8.

10. The determining means is Select the range image with the highest accuracy from the N range images; selecting one distance image from the unselected distance images based on the accuracy of each integrated data obtained by integrating each unselected distance image from among the N distance images with all the selected distance images from among the N distance images, and repeating this process until the M distance images are selected; determining, from among the N imaging positions, M imaging positions corresponding to the selected M distance images; 10. The information processing system according to claim 1,

11. The imaging control means further acquires distance images by imaging the teaching object using the distance image sensor from peripheral positions that have a certain relative positional relationship with each of the N imaging positions, the constructing means constructs the plurality of integrated data by integrating N image sets, each of which is a set of a distance image captured from one of the N imaging positions and a distance image captured from a peripheral position of the one imaging position, in a plurality of different combinations.

11. The information processing system according to claim 1,

12. The imaging control means further acquires distance images by individually imaging W teaching objects including the teaching object using the distance image sensor from the N imaging positions, the constructing means constructs the plurality of integrated data by integrating N image sets, each set being a set of W distance images obtained by individually capturing images of the W teaching objects from one of the N imaging positions, in a plurality of different combinations; 12. The information processing system according to claim 1,

13. the information processing system includes the measurement device, the measuring device acquires integrated data obtained by integrating the M distance images captured by the distance image sensor from the M imaging positions of the object as a measurement result of the object.

13. The information processing system according to claim 1,

14. A measuring device for performing three-dimensional measurement of an object using a plurality of distance images acquired at different imaging positions by a distance image sensor that acquires distance images using the principle of triangulation, An information processing method for determining imaging positions of M distance images to be used for measuring the object, comprising: an imaging control step of acquiring N distance images of a teaching object from N imaging positions (N>M) using the distance image sensor; a constructing step of constructing a plurality of integrated data from each of a plurality of different combinations of the N distance images; a determining step of calculating an accuracy representing a degree of matching between each of the plurality of integrated data and the teaching object, and determining M imaging positions from among the N imaging positions based on the accuracy of each of the plurality of integrated data; 13. An information processing method comprising:

15. A program for causing a computer to execute each step of the information processing method according to claim 14.

Images