Coordination support system and coordination support method

Abstract

To automatically determine a plurality of positions and postures of a sensor for detecting a positional displacement of a component of each facility in a manufacturing line, and reduce the number of steps of adjusting a positional displacement.SOLUTION: In an adjustment support system having an operation unit and a storage unit, the storage unit holds sensor information including information on the characteristics of a sensor for imaging a target and imaging target information including information on the size, the shape, and the arrangement of the imaging target of the sensor. The operation unit generates a plurality of candidates of imaging positions and imaging postures of the imaging target of the sensor, determines whether the sensor can detect a displacement of plural directions of the imaging target from an image taken by the sensor on the basis of the sensor information and the imaging target information for each candidate of the imaging position and the imaging posture, and determines at least one imaging position and imaging posture for the sensor to actually take the imaging target from the plurality of imaging positions and imaging postures on the basis of the result of determination.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology for supporting adjustment of equipment in a manufacturing line. [Background technology]

[0002] In the manufacturing industry, there is a need for short-term modifications to equipment on production lines in order to quickly respond to changes in the production environment, such as epidemics and natural disasters. Since the components of each piece of equipment have an installation position error (misalignment) on the actual machine compared to the ideal value in the design, it is necessary to adjust the misalignment each time the equipment is modified. In order to measure the misalignment on the actual machine using a sensor such as a camera, it is necessary to search for a sensor position and orientation that can detect the misalignment with the desired accuracy, and reducing the labor required for the search is a challenge.

[0003] A method for automatically searching for a sensor position and orientation is described in Japanese Patent Laid-Open No. 2016-99257 (Patent Document 1). This publication states that "the system includes a calculation means for calculating hidden areas in which the target object is occluded when the target object is viewed from a plurality of viewpoints based on two-dimensional information or three-dimensional information acquired by a sensor means at a first position and orientation, and a determination means for determining a second position and orientation, different from the first position and orientation, as a position and orientation of a sensor means capable of acquiring three-dimensional information for measuring the position and orientation of the target object based on the hidden areas calculated by the calculation means." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-99257 A Summary of the Invention [Problem to be solved by the invention]

[0005] The method of Patent Document 1 determines the sensor position and orientation based on the calculation result of the hidden area where the target object is occluded, and does not calculate the magnitude of the positional deviation of the target object as viewed from the sensor position and orientation. Therefore, for example, if the magnitude of the positional deviation of the target object acquired when viewed from the determined sensor position and orientation is below the threshold detectable by the sensor, the positional deviation cannot be detected.

[0006] Furthermore, the method of Patent Document 1 determines a single position and orientation for measuring three-dimensional information of a target object. However, a three-dimensional distance sensor such as a stereo camera acquires images using multiple cameras installed in relatively close positions, and calculates the distance to the target based on the difference between the images. In this case, the detection accuracy is low because the difference in the depth direction is difficult to appear on the image. Currently, the detection accuracy in the depth direction is on the order of millimeters, and it is difficult to detect a minute positional deviation of sub-millimeters or less from a single position and orientation. Therefore, in order to detect a minute positional deviation of sub-millimeters or less, it is necessary to determine multiple positions and orientations that are relatively far from each other.

[0007] Therefore, an object of the present invention is to provide a technique for automatically determining the positions and orientations of multiple sensors capable of detecting positional deviations, thereby reducing the number of steps required for searching for the sensor positions and orientations for adjusting positional deviations. [Means for solving the problem]

[0008] In order to solve at least one of the above problems, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problems, and an example thereof is an adjustment support system having a calculation unit and a storage unit, the storage unit holds sensor information including information on the characteristics of a sensor that captures an object, and photographing object information including information on the dimensions, shape, and arrangement of the object photographed by the sensor, the calculation unit generates a plurality of candidates for the photographing position and the photographing posture of the object photographed by the sensor, and calculates the photographing position and the photographing posture of the object based on the sensor information and the photographing object information for each of the plurality of candidates. , including directions other than the depth direction of the image captured by the sensorThe method is characterized in that it is determined whether or not positional deviations in multiple directions can be detected from the image captured by the sensor, and based on the result of the determination, one or more shooting positions and shooting postures for the sensor to actually capture the subject are determined from multiple candidates for the shooting positions and shooting postures. Effect of the Invention

[0009] According to one aspect of the present invention, the positions and orientations of a plurality of sensors capable of detecting a positional deviation can be automatically determined, and the number of steps required for adjusting the positional deviation can be reduced.

[0010] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of an overall configuration of an adjustment support system according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart showing an example of a flow of a process executed by the adjustment support system according to the first embodiment of the present invention. [Diagram 3] 11 is a flowchart showing an example of a flow of a process executed by a shooting position candidate generating unit according to the first embodiment of the present invention. [Figure 4] 11 is a flowchart showing an example of a flow of a process executed by a photographing position determining unit according to the first embodiment of the present invention. [Diagram 5] FIG. 4 is an explanatory diagram showing an example of shooting position information displayed on a display unit according to the first embodiment of the present invention. [Figure 6] FIG. 4 is an explanatory diagram illustrating an example of image-taking position information stored in an image-taking position information storage unit according to the first embodiment of the present invention. [Figure 7] FIG. 11 is a block diagram showing an example of the overall configuration of an adjustment support system according to a second embodiment of the present invention. [Figure 8] 11 is a flowchart showing an example of a flow of processes executed by the adjustment support system according to the second embodiment of the present invention. [Figure 9]11 is a flowchart showing an example of a flow of processing executed by a shooting operation generation unit according to a second embodiment of the present invention. [Figure 10] FIG. 11 is a block diagram showing an example of the overall configuration of an adjustment support system according to a third embodiment of the present invention. [Figure 11] 11 is a flowchart showing an example of a flow of processes executed by the adjustment support system according to the third embodiment of the present invention. [Figure 12] 13 is a flowchart showing an example of a flow of processing executed by an error estimating unit according to the third embodiment of the present invention. [Figure 13] 13 is a flowchart showing an example of a flow of processing executed by an operation plan correction unit according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an adjustment support system and an apparatus according to the present invention will be described based on embodiments with reference to the drawings. EXAMPLES

[0013] FIG. 1 is a block diagram showing an example of the overall configuration of an adjustment support system according to a first embodiment of the present invention.

[0014] As shown in FIG. 1, the adjustment support system 100 includes an input unit 101, an equipment information memory unit 102, a sensor information memory unit 103, a shooting target information memory unit 104, a shooting position information memory unit 105, a shooting position candidate generation unit 106, a shooting position determination unit 107, an output unit 108, and a display unit 109.

[0015] Input information is input to the input unit 101. The equipment information storage unit 102 stores equipment information including information on external characteristics such as dimensions, shape, and arrangement of the equipment. The sensor information storage unit 103 stores sensor information on sensor characteristics such as the number of pixels of the sensor, viewing angle, and shooting distance. The shooting target information storage unit 104 stores shooting target information including information on external characteristics such as dimensions, shape, and arrangement of the shooting target. The shooting position information storage unit 105 stores information such as the position and orientation of the sensor.

[0016] As described later, the sensor may be installed on the hand of the robot. In this case, the adjustment support system 100 may further hold information on the characteristics of the robot (e.g., information indicating the positions and postures that the hand of the robot can take). Such information may be included in the sensor information, may be included in the equipment information, or may be held independently of them.

[0017] The imaging position candidate generation unit 106 generates multiple candidates for the sensor position and orientation. The imaging position determination unit 107 determines multiple combinations of sensor positions and orientations that can detect a position shift of the imaging target from the multiple imaging position candidates. The output unit 108 outputs the determined sensor positions and orientations. The display unit 109 outputs the determined sensor positions and orientations.

[0018] The adjustment support system 100 may be realized by, for example, a computer having a calculation unit 112 and a storage unit 113. In this case, the calculation unit 112 is a processor that executes a program stored in the storage unit 113, and the shooting position candidate generation unit 106 and the shooting position determination unit 107 are realized by the calculation unit 112 executing the program. That is, in the following description, the processes executed by the shooting position candidate generation unit 106 and the shooting position determination unit 107 are realized by, for example, the calculation unit 112 executing the program.

[0019] The storage unit 113 includes a main storage device such as a DRAM and an auxiliary storage device such as a hard disk drive or SSD, and stores programs executed by the calculation unit 112 and data used in the processing of the calculation unit 112. The facility information storage unit 102, the sensor information storage unit 103, the photographing target information storage unit 104, and the photographing position information storage unit 105 are storage areas within the storage unit 113.

[0020] The input unit 101 may include input devices such as a keyboard, a pointing device, and various sensors. The type of sensor is not limited, and may be, for example, an optical camera or a three-dimensional scanner such as a so-called LIDAR. Alternatively, the input unit 101 may include a communication device (not shown) to which information is input via a network (not shown). The output unit 108 outputs information obtained as a result of the processing of the calculation unit 112. For example, the output unit 108 may store the obtained information as output information in the storage unit 113, or may output the information via a communication device. The display unit 109 displays the information obtained as a result of the processing of the calculation unit 112. For example, the display unit 109 may include a display device that displays characters, drawings, etc.

[0021] 2 is a flowchart showing an example of the flow of processing executed by the adjustment support system 100 according to the first embodiment of the present invention. The flow of processing in the adjustment support system 100 will be described below with reference to FIGS.

[0022] <Shooting position candidate generation unit> In the first procedure shown in Fig. 2, when the adjustment support system 100 starts processing, the imaging position candidate generating unit 106 receives the equipment information, imaging target information, and sensor information as inputs and generates a plurality of sensor position candidates for detecting positional deviation of the imaging target (step 110). Details of this processing will be described with reference to Fig. 3.

[0023] FIG. 3 is a flowchart showing an example of the flow of processing executed by the shooting position candidate generating unit 106 according to the first embodiment of the present invention.

[0024] In step 120, the imaging position candidate generating unit 106 arranges a virtual polyhedron with the imaging target at its center. The imaging position candidate generating unit 106 arranges the polyhedron so that the distance between the center of each face of the polyhedron and the imaging target coincides with the imaging distance stored in the sensor information storage unit 103. When the volume of the imaging target is large and the surface of the imaging target does not fit in the captured image, a point on the surface of the imaging target may be set as the center of the polyhedron instead of the center of the imaging target. In addition, a polyhedron is used in which the direction of the line connecting the center of each face of the polyhedron and the imaging target coincides with the direction of the normal of each face. For example, a regular polyhedron is known as a polyhedron having such properties, but it does not have to be a regular polyhedron as long as it has similar properties. In addition, the imaging target does not necessarily have to be located at the center of the polyhedron as long as it is contained within the polyhedron.

[0025] The method of arranging the polyhedron as described above is one example of a method of generating candidates for the position and orientation of the sensor for photographing the subject from multiple directions, and candidates for the position and orientation of the sensor may be generated so as to photograph the subject from multiple directions by a method other than the above. For example, candidates for the position and orientation of the sensor may be generated using only some of the faces constituting the polyhedron.

[0026] In step 121, the photographing position candidate generating unit 106 acquires the center position of each face of the polyhedron and the normal direction of each face. Then, the photographing position candidate generating unit 106 sets the acquired center position as the sensor position, the normal direction as the sensor orientation, and sets the sensor position and orientation corresponding to each face as photographing position candidates. Note that, among the sensor orientations, the rotation direction around the normal direction of each face of the polyhedron as the rotation axis may use an arbitrarily determined value.

[0027] In step 122, the photographing position candidate generating unit 106 excludes from the photographing position candidates acquired in step 121 unrealizable positions and orientations. An unrealizable position and orientation is, for example, a position and orientation that interferes with equipment. In addition, when the sensor is installed on a robot or the like, the photographing position candidate generating unit 106 excludes from the photographing position candidates any orientation that the robot cannot achieve.

[0028] In step 123, the photographing position candidate generating unit 106 excludes from the photographing position candidates those positions and orientations in which it is difficult to detect the photographing target. The positions and orientations in which it is difficult to detect the photographing target are, for example, positions and orientations in which occlusion occurs due to an object existing between the sensor and the photographing target. The photographing position candidate generating unit 106 also excludes from the photographing position candidates those positions and orientations in which the contour of the photographing target becomes unclear due to the influence of a shadow generated by lighting, etc.

[0029] Here, steps 122 and 123 are performed in order to reduce the processing time of the photographing position determination unit 107 in step 111 by excluding in advance photographing position candidates that are not suitable as photographing positions.

[0030] In step 124, the photographing position candidate generating unit 106 transfers the remaining photographing position candidates that have not been excluded to the photographing position determining unit 107, and the process ends.

[0031] By the above process, it is possible to automatically generate candidates for the sensor position and orientation for capturing an image of a target at a desired shooting distance.

[0032] <Shooting position determination section> 2, the photographing position determination unit 107 determines the positions and orientations of a plurality of sensors for detecting a positional deviation of a photographing target (step 111). Details of this process will be described with reference to FIG.

[0033] FIG. 4 is a flowchart showing an example of the flow of processing executed by the photographing position determining unit 107 according to the first embodiment of the present invention.

[0034] In step 130, the photographing position determination unit 107 moves the photographing target by small distances in multiple directions in the virtual space based on the photographing target information stored in the photographing target information storage unit 104. The small distances are, for example, distances corresponding to the adjustment accuracy required for adjusting the positional deviation, and may be determined in advance.

[0035] In step 131, the photographing position determination unit 107 acquires an image of the subject photographed by the sensor at each of the photographing position candidates generated in step 110. The image is acquired in a virtual space based on the sensor information stored in the sensor information storage unit 103 and the photographing subject information stored in the photographing subject information storage unit 104. In other words, the photographed image is not acquired by the sensor photographing the subject in the actual space, but by calculating an image that would be obtained when photographing is performed from the sensor at the position and orientation according to each photographing position candidate based on the sensor information and the photographing subject information. The acquisition of an image of the subject by such calculation is also referred to as photographing in a virtual space.

[0036] Shooting in the virtual space is performed before and after the subject is moved a small distance in step 130. The shooting position determination unit 107 sets the image before the movement as a reference image and the image after the movement as a shifted image, and obtains a difference image between the reference image and the shifted image. The difference image is an image obtained, for example, when background difference processing is applied.

[0037] In step 132, the photographing position determination unit 107 obtains the amount of misalignment detection from the difference image obtained in step 131, and if the amount of misalignment detection is equal to or greater than a threshold, it is determined that misalignment in the movement direction is detectable at that photographing position. Here, the amount of misalignment detection is information that indicates the magnitude of misalignment as seen from a sensor placed at a certain position and in a certain orientation, such as the line width or pixel angle of the difference image. In addition, the threshold used for the determination is, for example, the minimum detectable pixel angle determined by the sensor specifications.

[0038] In step 133, the photographing position determination unit 107 performs the judgment in step 132 for all combinations of photographing position candidates and moving directions, and then determines the photographing position candidate with the most moving directions in which a position shift can be detected as the first photographing position.

[0039] In step 134, the photographing position determination unit 107 determines, as the second photographing position, the photographing position candidate with the largest amount of detected positional deviation among the photographing position candidates capable of detecting positional deviation in the moving direction in which the first photographing position cannot be detected.

[0040] In step 135, the photographing position determination unit 107 judges whether or not the amount of detected positional deviation is equal to or greater than a threshold value for all of the moving directions generated in step 130, using the first photographing position and the second photographing position. If the judgment result is YES, the process proceeds to step 137, and if the judgment result is NO, the process proceeds to step 136.

[0041] In step 136, the photographing position determination unit 107 adds, to the photographing positions, the photographing position candidates capable of detecting positional deviations in the movement directions that cannot be detected from all the photographing positions determined up to step 135. The photographing position determination unit 107 repeats this process until it becomes possible to detect positional deviations in all the movement directions generated in step 130.

[0042] In step 137, information on the determined photographing position, a photographed image of the subject at the photographing position, a detected amount of positional deviation, and the like are output to the display unit 109.

[0043] FIG. 5 is an explanatory diagram showing an example of the shooting position information displayed on the display unit 109 according to the first embodiment of the present invention.

[0044] For example, the display unit 109 may output information as shown in FIG.

[0045] Coordinates 140 and 141 are coordinates representing a first shooting position and a second shooting position, respectively. In this description, the shooting positions are expressed using x, y, and z that represent positions in a Cartesian coordinate system, and a, b, and c that represent attitudes by the amount of rotation of the coordinate axes, but other coordinate systems such as a cylindrical coordinate system and other attitude expressions such as quaternions may also be used.

[0046] Image 142 is an image that represents the positional relationship between the sensor, the subject, and the equipment at the first shooting position, based on the subject information stored in shooting subject information storage unit 104 and the equipment information stored in equipment information storage unit 102. Similarly, image 143 is an image that represents the positional relationship between the sensor, the subject, and the equipment at the second shooting position, based on the subject information stored in shooting subject information storage unit 104 and the equipment information stored in equipment information storage unit 102.

[0047] Images 144 and 145 are the reference image of the subject viewed from the first shooting position and the reference image of the subject viewed from the second shooting position, respectively, acquired in step 131 .

[0048] Table 146 is a table listing information such as the amount of positional deviation detected and whether or not it is detectable at the first shooting position, obtained in step 132. Similarly, table 147 is a table listing information such as the amount of positional deviation detected and whether or not it is detectable at the second shooting position, obtained in step 132.

[0049] If there is a problem with the photographing position, the user of the adjustment support system 100 can correct the numerical values ​​of the photographing position coordinates 140 and 141 by directly inputting them by operating a button 148 via the input unit 101. The photographing position determination unit 107 can execute the processes of steps 131 and 132 again based on the corrected photographing position, and check the amount of positional deviation detection and whether or not a positional deviation can be detected.

[0050] If there is no problem with the photographing position, when the user operates the button 149, the output unit 108 outputs the photographing position information to the photographing position information storage unit 105, and the process ends.

[0051] FIG. 6 is an explanatory diagram showing an example of the image-taking position information stored in the image-taking position information storage unit 105 according to the first embodiment of the present invention.

[0052] The table 150 has a photographing position number column 151 and a position and orientation column 152 .

[0053] The photographing position number field 151 stores a number for identifying the photographing position.

[0054] The position and orientation field 152 stores information indicating the coordinates of the determined image capturing position and the orientation of the sensor at that image capturing position.

[0055] In this description, the photographing position number and the position and orientation are described as photographing position information, but necessary information may be added or deleted, and the present invention does not limit the information to be described.

[0056] As described above, according to this embodiment, the positions and orientations of a plurality of sensors capable of detecting a positional deviation can be automatically determined, and the number of steps required for adjusting the positional deviation can be reduced. EXAMPLES

[0057] Next, a second embodiment of the present invention will be described. In this embodiment, a system is described that generates a robot photography operation program based on photography position information when detecting a positional deviation using a sensor installed at the hand of the robot. Except for the differences described below, each part of the system of the second embodiment has the same function as each part of the first embodiment shown in Figures 1 to 6 with the same reference numerals, and therefore the description thereof will be omitted.

[0058] FIG. 7 is a block diagram showing an example of the overall configuration of an adjustment support system according to the second embodiment of the present invention.

[0059] Of the adjustment support system 200 of the second embodiment shown in Fig. 7, the description of the parts that are the same as those of the adjustment support system 100 shown in Fig. 1 will be omitted. The difference from Fig. 1 is that the adjustment support system 200 includes a calculation unit 212 and a storage unit 211. The calculation unit 212 includes a shooting operation generation unit 201 in addition to the shooting position candidate generation unit 106 and the shooting position determination unit 107 shown in Fig. 1. The storage unit 211 includes a shooting operation program storage unit 202 in addition to the respective storage units 102 to 105 shown in Fig. 1.

[0060] The photographing action generation unit 201 generates an action program for the robot to move to each photographing position and take a photograph based on the photographing position information stored in the photographing position information storage unit 105. This is realized, for example, by the calculation unit 212 executing the program stored in the storage unit 211.

[0061] The photographing operation program storage unit 202 stores the robot operation program generated by the photographing operation generation unit 201.

[0062] 8 is a flowchart showing an example of the flow of processing executed by the adjustment support system 200 according to the second embodiment of the present invention. Hereinafter, the flow of processing in the adjustment support system 200 will be described with reference to FIGS.

[0063] The first and second procedures shown in FIG. 8 are not different from the first and second procedures (step 110 and step 111) shown in FIG. 2, and therefore will not be described.

[0064] In the third procedure shown in Fig. 8, the photographing operation generation unit 201 generates an operation program for moving a sensor installed at the robot's hand to each photographing position and taking photographs (step 210). Details of this process will be described with reference to Fig. 9.

[0065] FIG. 9 is a flowchart showing an example of the flow of processing executed by the shooting operation generation unit 201 according to the second embodiment of the present invention.

[0066] In step 220, the photographing operation generation unit 201 generates a plurality of photographing sequence candidates by rearranging the photographing position number column 151 in FIG.

[0067] In step 221, the photographing operation generation unit 201 converts the sensor position and orientation described in the position and orientation field 152 in FIG. 6 into a position and orientation in the robot coordinate system.

[0068] In step 222, the photographing motion generation unit 201 generates a motion trajectory of the robot that moves to the photographing position according to the photographing sequence candidate generated in step 220. A trajectory planning method such as RRT (Rapidly Exploring Random Tree) is used to generate the motion trajectory.

[0069] In step 223, the photographing action generation unit 201 determines one photographing action from each of the photographing action candidates. As a criterion for the determination, for example, a criterion for selecting the action trajectory with the shortest action time from among the action trajectories generated in step 222 may be used, but other criteria may also be used.

[0070] In step 224, the photographing action generation unit 201 outputs a motion trajectory corresponding to the determined photographing order as a motion program of the robot, and stores it in the photographing action program storage unit 202, and the process ends.

[0071] As described above, according to this embodiment, after automatically determining the positions and orientations of a plurality of sensors capable of detecting a positional deviation, an operation program for the robot to perform a photographing operation can be automatically generated, and the labor required for adjusting the positional deviation can be reduced. EXAMPLES

[0072] Next, a third embodiment of the present invention will be described. In this embodiment, a system is described in which a sensor installed at the tip of a robot detects a positional deviation and corrects a motion plan for the robot based on an estimated amount of positional deviation. Except for the differences described below, each part of the system of the third embodiment has the same function as each part with the same reference numeral in the first or second embodiment shown in Figs. 1 to 9, and therefore the description thereof will be omitted.

[0073] FIG. 10 is a block diagram showing an example of the overall configuration of an adjustment support system according to the third embodiment of the present invention.

[0074] Of the adjustment support system 300 of the third embodiment shown in Fig. 10, the description will be omitted for the parts that are the same as those of the adjustment support system 100 shown in Fig. 1 and the adjustment support system 200 shown in Fig. 7. As a difference from Fig. 7, the adjustment support system 300 includes a calculation unit 342 and a storage unit 341. The calculation unit 342 includes an error estimating unit 302 and an operation plan correcting unit 303 in addition to the shooting position candidate generating unit 106, the shooting position determining unit 107, and the shooting operation generating unit 201 shown in Fig. 7. The storage unit 341 includes an operation plan storage unit 301 and an operation plan revision proposal storage unit 304 in addition to the respective storage units 102 to 105 and 202 shown in Fig. 7.

[0075] The motion plan storage unit 301 stores a motion plan that defines a series of robot motions, such as picking or welding, in the form of a motion program or the like.

[0076] The error estimation unit 302 estimates the amount of positional deviation of the subject based on an image captured according to the imaging operation program stored in the imaging operation program storage unit 202 .

[0077] The motion plan correction unit 303 corrects the motion plan of the robot stored in the motion plan storage unit 301 based on the positional deviation amount of the imaging target estimated by the error estimating unit 302 .

[0078] The processing of the error estimating unit 302 and the motion plan correcting unit 303 is realized, for example, by the calculation unit 342 executing a program stored in the storage unit 341 .

[0079] The motion plan revision proposal storage unit 304 stores the motion plan of the robot revised by the motion plan revision unit 303 in the form of a motion program or the like.

[0080] 11 is a flowchart showing an example of the flow of processing executed by the adjustment support system 300 according to the third embodiment of the present invention. Hereinafter, the flow of processing in the adjustment support system 300 will be described with reference to FIGS.

[0081] The first and second steps shown in Fig. 11 are not different from the first step (step 110) and second step (step 111) shown in Fig. 2, and therefore their explanation will be omitted. Also, the third step shown in Fig. 11 is not different from the third step (step 210) shown in Fig. 8, and therefore its explanation will be omitted.

[0082] <Error estimation part> 11, the error estimating unit 302 captures images of the subject from a plurality of shooting positions according to the shooting operation program, and estimates the positional deviation amount of the subject (step 310). Details of this process will be described with reference to FIG.

[0083] FIG. 12 is a flowchart showing an example of the flow of processing executed by the error estimating unit 302 according to the third embodiment of the present invention.

[0084] In step 320, the error estimation unit 302 executes the photographing operation program stored in the photographing operation program storage unit 202 to obtain a photographed image of the photographing target. In this way, an image of the photographing target photographed by the sensor in the actual space is obtained.

[0085] In step 321, the error estimation unit 302 compares the captured image acquired in step 320 with the reference image acquired in step 131 to estimate the amount of positional deviation of the subject.

[0086] The amount of positional deviation can be estimated, for example, by extracting a feature of the photographed object from an image and measuring the difference between the feature in the reference image and the photographed image. For example, the pixel position where the characteristic shape of the photographed object appears, or the contour line of the photographed object, may be used as the feature. Alternatively, a known feature such as a Scale-Invariant Feature Transform (SIFT) feature or a Speeded-Up Robust Features (SURF) feature may be used. Furthermore, the error estimation unit 302 may generate various positional deviations in advance in a virtual space to obtain deviation images, and may use the deviation images as learning data to extract the feature and estimate the amount of positional deviation using a technique such as a neural network or machine learning.

[0087] In step 322, the error estimating unit 302 transfers the amount of positional deviation estimated in step 321 to the motion plan correcting unit 303, and the process ends.

[0088] By the above process, the amount of positional deviation of the subject can be automatically estimated from the captured image.

[0089] <Motion plan correction section> In the fifth procedure shown in Fig. 11, the operation plan correction unit 303 captures images of the subject from a plurality of shooting positions according to the shooting operation program, and estimates the positional deviation amount of the subject (step 311). Details of this process will be described with reference to Fig. 13.

[0090] FIG. 13 is a flowchart showing an example of the flow of processing executed by the operation plan correction unit 303 according to the third embodiment of the present invention.

[0091] In step 330, the operation plan correction unit 303 reflects the amount of positional deviation of the imaging target estimated by the error estimating unit 302 in the imaging target information stored in the imaging target information storage unit 104, and places the imaging target in the virtual space. Here, reflecting the amount of positional deviation in the imaging target information refers to a process of adding 0.2 mm to the x coordinate and adding 0.5° to the a coordinate of the imaging target information when the amount of positional deviation is 0.2 mm in the x direction and 0.5° in the a direction, for example.

[0092] In step 331, the motion plan correction unit 303 corrects the motion plan in the virtual space in which the positional deviation is reflected. For example, when the motion plan stored in the motion plan storage unit 301 is a picking motion of the robot and the subject to be photographed is a part to be picked, the motion plan correction unit 303 substitutes the amount of positional deviation estimated by the error estimation unit 302 as a correction value into the robot coordinates representing the picking position in the motion program, and generates a motion trajectory of the robot for picking the part in a state in which the positional deviation of the part to be picked is corrected in the virtual space by the process of step 330. Here, a known trajectory planning method such as RRT may be used to generate the motion trajectory.

[0093] In step 332, the motion plan modification unit 303 outputs the modified motion plan to the motion plan modification proposal storage unit, and ends the process.

[0094] As described above, according to this embodiment, the sensor installed in the robot can automatically detect and estimate positional deviations and correct the operation program, thereby reducing the number of steps required for adjusting positional deviations.

[0095] Furthermore, the system according to the embodiment of the present invention may be configured as follows.

[0096] (1) An adjustment support system (e.g., adjustment support system 100, 200, or 300) having a calculation unit (e.g., calculation unit 112, 212, or 342) and a memory unit (e.g., memory unit 113, 211, or 331), in which the memory unit holds sensor information (e.g., sensor information stored in sensor information memory unit 103) including information on characteristics of a sensor that captures an image of an object, and photographing target information (e.g., photographing target information stored in photographing target information memory unit 104) including information on dimensions, shape, and arrangement of the object to be photographed by the sensor, and the calculation unit generates a plurality of candidates for the photographing position and the photographing posture of the object to be photographed by the sensor (e.g., step 110), and for each of the plurality of candidates for the photographing position and the photographing posture, determines whether or not positional deviations in a plurality of directions of the object to be photographed can be detected from an image photographed by the sensor based on the sensor information and the photographing target information, and based on the result of the determination, determines one or more photographing positions and photographing postures for the sensor to actually photograph the object from the plurality of candidates for the photographing position and the photographing posture (e.g., step 111).

[0097] This makes it possible to automatically determine the positions and orientations of multiple sensors that can detect positional deviations, thereby reducing the number of steps required for adjusting position deviations.

[0098] (2) In the above (1), the calculation unit generates multiple candidates for the shooting positions and shooting directions by taking the centers of each face of a polyhedron containing the object to be photographed as candidates for the shooting positions and taking the postures for photographing the normal directions of each face as candidates for the shooting postures (e.g., step 121).

[0099] This makes it possible to generate suitable candidates for the sensor position and orientation so that position shifts in more directions can be detected.

[0100] (3) In (1) above, the calculation unit calculates, for each of a plurality of candidates for the shooting position and shooting posture, images obtained when the sensor captures images before and after the shooting subject is moved a predetermined distance in each of a plurality of directions, based on the sensor information and the shooting subject information, and determines that a positional shift in that direction can be detected if the line width of the difference image between the images before and after the movement of the predetermined distance exceeds a predetermined threshold (e.g., steps 130 to 132).

[0101] This makes it possible to determine the sensor position and orientation that allows detection of positional deviation.

[0102] (4) In (1) above, the calculation unit identifies, for each of a plurality of candidates for the shooting position and shooting posture, the number of directions of positional deviation that can be detected based on the images captured by the sensor, among the positional deviations of a predetermined distance in multiple directions of the subject to be photographed, and determines one or more shooting positions and shooting postures for the sensor to actually photograph the subject based on the number of detectable directions of positional deviation (e.g., step 134).

[0103] This makes it possible to determine the sensor position and orientation that can detect positional deviations in more directions.

[0104] (5) In the above (4), the calculation unit determines the one or more shooting positions and shooting postures for the sensor to actually capture an image of the subject so that all positional deviations of a predetermined distance in multiple directions can be detected (e.g., steps 134 to 136).

[0105] This makes it possible to determine the sensor position and orientation that can detect positional deviations in all conceivable directions.

[0106] (6) In the above (4), the calculation unit determines, from among the multiple candidates for the shooting position and shooting posture, the candidate having the largest number of detectable directions of positional deviation as one of the one or more shooting positions and shooting postures for the sensor to actually capture the image of the subject (e.g., step 134).

[0107] This makes it possible to determine the sensor position and orientation that can detect positional deviations in more directions.

[0108] (7) In the above (6), the calculation unit identifies a direction that is not detectable based on an image captured at the shooting position and shooting posture determined as the shooting position and shooting posture from among multiple candidates for the shooting position and shooting posture, and adds the candidate shooting position and shooting posture in which a positional deviation in the identified direction can be detected as one of the shooting positions and shooting postures for the sensor to actually capture the subject (e.g., steps 134 to 136).

[0109] This makes it possible to determine the sensor position and orientation that can detect positional deviations in all conceivable directions.

[0110] (8) In (1) above, the calculation unit outputs information indicating one or more shooting positions and shooting postures for the sensor to actually capture an image of the subject (e.g., step 137, FIG. 5), and when information for correcting at least one of the one or more shooting positions and shooting postures for the sensor to actually capture an image of the subject is input, the calculation unit corrects the one or more shooting positions and shooting postures for the sensor to actually capture an image of the subject in accordance with the input information.

[0111] This allows the user's knowledge to be reflected in the detection of positional deviation.

[0112] (9) In the above (1), the sensor is installed on a robot, and the calculation unit creates a robot program for sequentially moving the sensor according to one or more shooting positions and shooting postures so that the sensor actually captures an image of a subject, and stores the program in a memory unit (e.g., step 210).

[0113] This makes it possible to automatically determine the positions and orientations of multiple sensors capable of detecting positional deviation, and then automatically generate an operating program for the robot to perform photographing operations, thereby reducing the amount of work required for adjusting positional deviations.

[0114] (10) In (9) above, the calculation unit determines an order in which the sensor should be moved according to one or more shooting positions and shooting postures for the sensor to actually capture an image of the subject so as to minimize the operation time of the robot (e.g., step 220), and creates a robot program according to the determined order (e.g., steps 221 to 224).

[0115] This can improve the efficiency of the positional deviation adjustment process.

[0116] (11) In the above (1), the memory unit further holds a motion plan of the robot for work in equipment including the subject to be photographed (e.g., a motion plan stored in the motion plan memory unit 301), and the calculation unit estimates the amount of positional deviation of the subject to be photographed by comparing an image actually captured by the sensor at one or more shooting positions and shooting postures for the sensor to actually capture the subject to be photographed with a result calculated based on sensor information and subject to be photographed information, and modifies the motion plan of the robot based on the estimated amount of positional deviation (e.g., step 311), and stores the modified motion plan of the robot in the memory unit (e.g., a motion plan stored in the motion plan modification proposal memory unit 304).

[0117] This makes it possible to automatically detect and estimate positional deviations and correct the operation program using sensors installed in the robot, thereby reducing the number of steps required for adjusting positional deviations.

[0118] (12) In the above (1), the memory unit further retains equipment information (e.g., equipment information stored in the equipment information memory unit 102) including information on the dimensions, shape, and layout of the equipment including the subject to be photographed, and the calculation unit excludes, from multiple candidate shooting positions and shooting postures based on the equipment information, shooting positions and shooting postures that cannot photograph the subject due to interference with the equipment (e.g., step 122).

[0119] This eliminates positions and postures that cannot actually be adopted from the candidates, improving work efficiency.

[0120] (13) In the above (1), the sensor is installed at the robot's hand, the memory unit further stores information indicating the characteristics of the robot, and the calculation unit, based on the information indicating the characteristics of the robot, excludes from a plurality of candidate shooting positions and shooting postures that cannot be realized by the robot (e.g., step 122).

[0121] This eliminates positions and postures that cannot actually be adopted from the candidates, improving work efficiency.

[0122] (14) In the above (1), the sensor is a camera, and the object to be photographed is a positioning pin for installing a work object on the equipment.

[0123] This makes it possible to reduce the number of steps required to adjust positional deviations when modifying equipment in a manufacturing line.

[0124] The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment has been described in detail for better understanding of the present invention, and is not necessarily limited to those having all the configurations described. Also, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0125] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function can be stored in a storage device such as a non-volatile semiconductor memory, a hard disk drive, or an SSD (Solid State Drive), or in a computer-readable non-transitory data storage medium such as an IC card, an SD card, or a DVD.

[0126] In addition, the control lines and information lines are shown as those considered necessary for the explanation, and not all control lines and information lines in the product are shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0127] 100, 200, 300... adjustment support system, 101... input unit, 102... equipment information storage unit, 103... sensor information storage unit, 104... shooting target information storage unit, 105... shooting position information storage unit, 106... shooting position candidate generation unit, 107... shooting position determination unit, 108... output unit, 109... display unit, 113, 211, 341... storage unit, 112, 212, 342... calculation unit

Claims

1. An adjustment support system having a calculation unit and a storage unit, The storage unit holds sensor information including information regarding characteristics of a sensor that captures an image of an object, and photographing object information including information regarding dimensions, shapes, and arrangement of the object photographed by the sensor; The calculation unit is generating a plurality of candidates for a photographing position and a photographing posture of the photographing object by the sensor; an adjustment support system which determines, for each of a plurality of candidates for the shooting position and shooting posture, based on the sensor information and the shooting subject information, whether or not a positional shift of the subject in a plurality of directions, including directions other than the depth direction of the image captured by the sensor, can be detected from the image captured by the sensor, and determines, based on a result of the determination, from the plurality of candidates for the shooting position and shooting posture, one or more shooting positions and shooting postures at which the sensor will actually capture the subject.

2. The adjustment support system according to claim 1, The calculation unit generates multiple candidates for the shooting positions and shooting directions by setting the centers of each face of a polyhedron that contains the subject to be photographed as candidates for the shooting positions, and setting the posture for photographing the normal direction of each face as candidates for the shooting posture.

3. The adjustment support system according to claim 1, The calculation unit calculates, for each of a plurality of candidates for the shooting position and shooting posture, images obtained when the sensor captures images before and after the subject is moved a predetermined distance in each of the plurality of directions, based on the sensor information and the subject information, and determines that a positional misalignment in that direction can be detected if a line width of a difference image between the images before and after the movement of the predetermined distance exceeds a predetermined threshold.

4. The adjustment support system according to claim 1, The calculation unit identifies, for each of a plurality of candidates for the shooting position and shooting posture, the number of directions of positional deviation that can be detected based on the images captured by the sensor, among the positional deviations of a predetermined distance in a plurality of directions of the subject to be photographed, and determines the one or more shooting positions and shooting postures for the sensor to actually photograph the subject based on the number of detectable directions of positional deviation.

5. The adjustment support system according to claim 4, The calculation unit determines the one or more shooting positions and shooting postures for the sensor to actually photograph the subject so that all positional deviations of a specified distance in the multiple directions can be detected.

6. The adjustment support system according to claim 4, the calculation unit determines, from among the multiple candidates for the shooting position and shooting posture, the candidate having the largest number of directions of detectable positional deviation as one of the one or more shooting positions and shooting postures for the sensor to actually photograph the subject.

7. The adjustment support system according to claim 6, the calculation unit identifies a direction that is not detectable based on an image captured at the shooting position and shooting posture determined as the shooting position and shooting posture from among a plurality of candidates for the shooting position and shooting posture, and adds the candidate shooting position and shooting posture in which a positional deviation in the identified direction can be detected as one of the shooting positions and shooting postures for the sensor to actually photograph the subject.

8. The adjustment support system according to claim 1, The calculation unit is outputting information indicating the one or more photographing positions and photographing postures for the sensor to actually photograph the photographing target; When information is input to correct at least one of the one or more shooting positions and shooting postures for the sensor to actually photograph the subject, the adjustment support system corrects the one or more shooting positions and shooting postures for the sensor to actually photograph the subject in accordance with the input information.

9. The adjustment support system according to claim 1, The sensor is installed on a robot, The calculation unit creates a robot program for sequentially moving the sensor according to the one or more shooting positions and shooting postures so that the sensor actually photographs the subject, and stores the program in the memory unit.

10. The adjustment support system according to claim 9, The calculation unit determines an order in which the sensor is moved according to the one or more shooting positions and shooting postures for the sensor to actually photograph the subject so as to minimize the operation time of the robot, and creates the robot program according to the determined order.

11. The adjustment support system according to claim 1, The storage unit further stores an operation plan for a robot for a task in a facility including the subject to be photographed, The calculation unit is an image actually captured by the sensor at the one or more shooting positions and shooting attitudes for the sensor to actually capture the image of the subject, and a result of calculating an image obtained when the sensor captures the image at the one or more shooting positions and shooting attitudes for the sensor to actually capture the image of the subject based on the sensor information and the subject information, thereby estimating a positional deviation amount of the subject; correcting a motion plan for the robot based on the estimated positional deviation amount; An adjustment support system characterized by storing the modified motion plan of the robot in the memory unit.

12. The adjustment support system according to claim 1, The storage unit further holds equipment information including information on dimensions, shapes, and arrangement of the equipment including the subject to be photographed, The calculation unit excludes, from a plurality of candidate shooting positions and shooting postures based on the equipment information, shooting positions and shooting postures that cannot photograph the subject due to interference with the equipment.

13. The adjustment support system according to claim 1, The sensor is installed at a tip of a robot, The storage unit further stores information indicating characteristics of the robot, An adjustment support system characterized in that the calculation unit excludes shooting positions and shooting postures that cannot be realized by the robot from multiple candidate shooting positions and shooting postures based on information indicating characteristics of the robot.

14. The adjustment support system according to claim 1, An adjustment support system characterized in that the sensor is a camera and the object to be photographed is a positioning pin for installing a work object on equipment.

15. An adjustment support method executed by an adjustment support system having a calculation unit and a storage unit, The storage unit holds sensor information including information regarding characteristics of a sensor that captures an image of an object, and photographing object information including information regarding dimensions, shapes, and arrangement of the object photographed by the sensor; The adjustment assistance method includes: a step of generating a plurality of candidates for a photographing position and a photographing posture of the object to be photographed by the sensor, the step being performed by the calculation unit; a step of determining, for each of a plurality of candidates for the shooting position and shooting posture, based on the sensor information and the shooting subject information, whether or not a positional shift of the subject in a plurality of directions, including directions other than the depth direction of the image captured by the sensor, can be detected based on the image captured by the sensor, and, based on a result of the determination, determining one or more shooting positions and shooting postures for the sensor to actually capture the subject.

Images