Assistance device, assistance system, assistance method, and program

Abstract

An assistance device (10) assists a worker to adjust a reference position in which a marker to be identified should initially be located, in a step in which the marker is identified by using a camera to image a workpiece conveyed by a conveying device having a drive unit, and the marker is positioned by further conveying the workpiece by means of the conveying device. An image acquiring unit (121) acquires a preparatory image obtained by imaging the workpiece at a preparatory position while being conveyed by the conveying device. A display data generating unit (160) generates display data for displaying, together with the preparatory image, at least one of a movement direction of the marker of the workpiece in the preparatory image corresponding to an operating direction of a drive unit immediately before the workpiece reaches the preparatory position, and a region within the preparatory image corresponding to a candidate position, which is a candidate of the reference position at which the operating direction of the drive unit is maintained before and after the marker is specified.

Description

Support device, support system, support method and program

[0001] The present disclosure relates to an assistance device, an assistance system, an assistance method, and a program.

[0002] In factory automation (FA), a process of transporting workpieces using a transport device such as a robot arm is sometimes carried out. When the drive unit of the transport device has a gear-based power transmission mechanism, such as a reducer and a transmission, backlash can impede power transmission when the drive unit reverses its operating direction. Backlash is a gap between gears, and when the rotation direction of a gear reverses, the drive gear spins freely by an amount corresponding to the gap. Therefore, a reversal of the drive unit's operating direction results in a decrease in the accuracy of workpiece positioning. Therefore, a method for controlling a robot arm so that the drive unit's operating direction remains constant without reversing when transporting a workpiece has been proposed (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2017-226029

[0004] In the technology of Patent Document 1, the drive direction of the joints of the robot arm is controlled to be constant when the robot arm moves to an assembly teaching point, and the robot arm operates from the assembly teaching point to assemble the workpiece to the object to be assembled. However, in cases where precise workpiece transport is required, such as when assembling a workpiece on the order of micrometers, there are limitations to motor control, so an additional process of positioning the workpiece using a high-magnification camera is required. For example, when correcting errors smaller than the workpiece gripping error to achieve high-precision final positioning, the gripping error may be recognized and corrected using a first camera, and then a marker attached to the workpiece may be recognized and aligned using a second camera with a higher magnification.

[0005] In this case, the addition of a process for alignment using a high-magnification camera makes it more difficult to set the movement direction of the drive unit so that it is maintained until the end. This is because the relative position and orientation between the workpieces, which allows for clear recognition of the markers attached to the workpieces, are limited, thereby restricting the movement path. It is desirable to set the movement path, including the initial position and via points of the workpiece, so that the movement direction of the drive unit is maintained until the end under these restrictions, but such setting work is complicated and difficult. As a result, if the movement path is not set including an appropriate initial position and via points, the accuracy of workpiece transport may be reduced.

[0006] The present disclosure has been made in light of the above-mentioned circumstances, and aims to improve the accuracy of work transport by setting an appropriate initial position for the work in the alignment process using a camera.

[0007] In order to achieve the above-mentioned object, the assistance device disclosed herein is an assistance device that, in a process of identifying feature points that define at least one of the position and posture of a workpiece by photographing the workpiece transported by a transport device having a drive unit with a camera, and aligning the feature points by further transporting the workpiece with the transport device, assists in adjusting a reference position where the identified feature points should initially be located.The assistance device includes: an image acquisition means that acquires a preparatory image of an object corresponding to the workpiece, photographed by the camera, at a preparatory position while being transported by the transport device; an information acquisition means that acquires information indicating the relationship between the operation of the drive unit and the movement of the object in the image photographed by the camera; a history data acquisition means that acquires history data indicating the operation history of the drive unit; and a display data generation means that generates, based on the information and history data, display data for displaying, together with the preparatory image, at least one of: a movement direction of the feature point of the object in the preparatory image that corresponds to the operation direction of the drive unit just before the object reaches the preparatory position; and an area in the preparatory image that corresponds to a candidate position that is a candidate for the reference position and where the operation direction of the drive unit of the transport device is maintained before and after the feature points are identified.

[0008] According to the present disclosure, the initial position of the workpiece in the process of aligning using a camera can be set appropriately, thereby improving the accuracy of workpiece transport.

[0009] FIG. 1 is a diagram showing the configuration of the support system according to the first embodiment. FIG. 1 is a diagram showing a state in which assembly according to the first embodiment is completed. FIG. 2 is a diagram showing the drive unit of the positioning stage according to the first embodiment. FIG. 3 is a diagram showing the drive unit of the transport device according to the first embodiment. FIG. 4 is a diagram showing the imaging range of the camera according to the first embodiment. FIG. 1 is a diagram showing the transport path of the work to be assembled according to the first embodiment. FIG. 2 is a diagram showing the transport path of the work to be assembled according to the first embodiment. FIG. 3 is a diagram showing the transport path of the work to be assembled according to the first embodiment. FIG. 4 is a diagram showing the hardware configuration of the support device according to the first embodiment. FIG. 5 is a diagram showing the functional configuration of the support device according to the first embodiment. FIG. 6 is a diagram showing an example of display of an operation guide according to the first embodiment. FIG. 7 is a diagram for explaining the margin according to the first embodiment. FIG. 8 is a diagram for explaining the calculation of the area according to the first embodiment. FIG. 9 is a diagram for explaining the calculation of the area according to the first embodiment. FIG. 10 is a diagram showing an example of display of the area according to the first embodiment.

[0010] Hereinafter, a support system according to an embodiment of the present disclosure will be described in detail with reference to the drawings. In the description, an XYZ coordinate system with three mutually orthogonal axes will be used as appropriate. In this coordinate system, the Z axis is parallel to the vertical line. In addition, a θ axis, which is an angle in a polar coordinate system on the XY plane and defines an angle equivalent to an azimuth angle, will be used as appropriate.

[0011] Embodiment 1. <Configuration of Support System 100> The support system 100 according to this embodiment is a system that supports a worker in adjusting the position of a workpiece to be photographed in order to perform a process of assembling two plate-shaped workpieces with high precision. As shown in FIG. 1 , the support system 100 includes a support device 10 that supports positioning work, cameras 20-22, a workpiece to be assembled 30 and a workpiece to be assembled 40 that are the targets of image capture by the cameras 20-22, a transport device 50 that transports the workpiece to be assembled 30 and the workpiece to be assembled 40, and a positioning stage 51 on which the workpiece to be assembled 40 is placed. The support device 10 is connected to the cameras 20-22 via communication lines and receives images captured by the cameras 20-22. While FIG. 1 shows one transport device 50 as a representative example, multiple transport devices 50 may be used.

[0012] The positioning performed in the example of Fig. 1 is a process performed, for example, on a semiconductor manufacturing line. The workpiece 30 to be assembled and the workpiece 40 to be assembled are each provided with alignment markers. In this process, the workpiece 30 to be assembled is transported from a supply position indicated by a dashed line to a reference position indicated by a solid line using images captured by a camera 20. Thereafter, as shown in Fig. 2, the workpiece 30 to be assembled is assembled to the workpiece 40 to be assembled using images captured by cameras 21 and 22 to align the workpieces so that their markers overlap.

[0013] The workpiece 40 is placed on a positioning stage 51 that is capable of translational and rotational movement. FIG. 3 shows an example of a drive unit for the positioning stage 51. The X-axis drive unit 511 and the Y-axis drive unit 512 in FIG. 3 are drive units that are capable of translational movement and include a ball screw or a linear motor. The θ-axis drive unit 513 is a drive unit that rotates and includes a stepping motor. These three axes are connected in series and can translate and rotate the stage fixed to the θ-axis drive unit 513.

[0014] The workpiece 30 to be assembled is grasped and transported by a robot hand. The distance traveled by the workpiece 30 from the supply position to the assembly position is long compared to the precision required for final assembly. For this reason, correction operations to achieve the final assembly precision are generally not performed on the workpiece 30 to be assembled. On the other hand, the positioning stage 51 has a short maximum movement distance, but is capable of high-precision positioning. For this reason, in the final assembly stage, the positioning stage 51 performs high-precision position correction based on the relative distance between markers obtained from images captured by the cameras 21 and 22. This allows the two workpieces to be assembled with high precision at a position where the markers overlap.

[0015] However, for ease of understanding, the following description will focus on an example in which a conveying device 50, which is a robot hand that conveys a workpiece 30 to be assembled, conveys the workpiece 30 from a supply position via a reference position to a final assembly position using five axes as shown in FIG. 4. In the example of FIG. 4, an X-axis drive unit 501 and a Y-axis drive unit 502 are drive units capable of translation, and a θ-axis drive unit 503 is a drive unit that rotates. Furthermore, a Z-axis drive unit 504 is a drive unit capable of translation, and a robot hand 505 grips or releases the workpiece 30 to be assembled. These drive units are assumed to be operable without interfering with the cameras 20 to 22.

[0016] 1, markers 31 and 32 are attached to the workpiece 30 to be assembled. Markers 41 and 42 are attached to the workpiece 40 to be assembled. The assembly positions are adjusted so that marker 31 overlaps with marker 41 and marker 32 overlaps with marker 42. Camera 21 photographs markers 31 and 41, and camera 22 photographs markers 32 and 42.

[0017] 5 shows the imaging ranges of cameras 21 and 22, which capture images looking vertically downward. The hatched area 211 is the imaging range of camera 21 and includes marker 31 of workpiece to be assembled 30 at the reference position and marker 41 of workpiece to be assembled 40. The hatched area 221 is the imaging range of camera 22 and includes marker 32 of workpiece to be assembled 30 at the reference position and marker 42 of workpiece to be assembled 40.

[0018] The camera 20 is installed alongside the cameras 21 and 22 directly above the positioning stage 51, and has a wide viewing angle that enables it to capture the entire workpiece 40. The camera 20 is used to correct any gripping error that occurs when the workpiece 30 is supplied, and to roughly position it to the reference position. If the supply position of the workpiece 30 is outside the range of the camera 20, a camera (not shown) may be further installed to capture an image of the workpiece 30 at the supply position in order to correct the gripping error.

[0019] <Flow of Assembly Work> Next, the flow of the assembly work will be described in detail. The assembly work includes, in this order, rough positioning of the workpiece 40 to be assembled, rough positioning of the workpiece 30 to be assembled, and final positioning using markers.

[0020] In the rough positioning of the workpiece 40, the positioning stage 51 is moved to an initial position, and the workpiece 40 is placed on the positioning stage 51. The positioning of the workpiece 40 is performed, for example, from a supply unit for the workpiece 40 via a robot hand having a drive unit. Then, for example, an image of the workpiece 40 is obtained by photographing it with the wide-field camera 20, and image recognition is performed to calculate a supply error correction amount for correcting an error in the position and orientation of the workpiece 40 during supply. When the positioning stage 51 is moved so that the marker falls within the field of view of the narrow-field cameras 21 and 22 used for final positioning, correction using the calculated supply error correction amount is added, thereby performing positioning including error correction.

[0021] Positioning including error correction is achieved, for example, as follows. First, in a preparation phase before processing a large number of workpieces, the positioning stage 51 is moved to an initial position, and then a reference workpiece 40 is placed within the field of view of the camera 20. The position and orientation of the workpiece 40 obtained by image recognition are saved as the reference position and orientation. The positioning stage 51 is then moved so that the workpiece 40 fits within the field of view of the cameras 21 and 22, and the position of the positioning stage 51 is saved as the stage reference position. Then, in an actual operation phase in which a large number of workpieces are processed, the inverse matrix of the movement and rotation matrix from the reference position and orientation to the position and orientation of the workpiece 40 placed on the positioning stage 51 and image-recognized is calculated as a supply error correction amount based on a coordinate system fixed to the reference position and orientation. This movement and rotation matrix corresponds to the supply error of the workpiece 40 to be processed in the actual operation phase, and the inverse matrix is ​​a correction amount for correcting the position and orientation of the workpiece 40 to the reference position and orientation. Next, when the positioning stage 51 is moved to the stage reference position, the workpiece 40 moves to a position that differs by the amount of the supply error relative to the reference workpiece 40. Therefore, based on a coordinate system fixed to the position and posture of the workpiece 40, the supply error correction amount is applied to the position and posture to correct it, thereby calculating the position and posture at the stage reference position including the correction for the supply error and performing positioning.

[0022] There are other commonly known calculation methods for correcting the position and orientation by image recognition, but their description will be omitted. Furthermore, there are commonly known calibration methods for converting the pixel values ​​of the camera image and the movement amount of the drive unit, so their description will be omitted. Furthermore, the position where image recognition is performed may not be the initial position of the positioning stage 51, but may be set as a different position to which the positioning stage 51 is moved from the initial position.

[0023] In the coarse positioning of the workpiece 30 to be assembled, the workpiece 30 to be assembled is transported from a supply position to a reference position. First, a robot hand having a drive unit moves to the supply position of the workpiece 30 to grip the workpiece 30 to be assembled. The robot hand is, for example, a suction hand or a clamping hand. Then, image recognition is performed using the wide-field camera 20, and a gripping error correction amount for correcting gripping errors is calculated. Then, when the robot hand is moved so that the markers fit within the fields of view of the cameras 21 and 22, correction is made using the calculated gripping error correction amount, thereby performing positioning including error correction. The error correction method is the same as that for the supply error of the workpiece 40 to be assembled, and therefore its description will be omitted.

[0024] In final positioning using markers, the workpiece 30 to be assembled is transported from a reference position to an assembly position. First, the cameras 21 and 22 recognize the markers, and the amount of movement of the drive unit responsible for final positioning to a position where the markers overlap is calculated, and the drive unit is operated by that amount of movement. Once the movement to the position where the markers overlap is complete, an assembly operation including lowering the robot hand and releasing the grip is performed to complete the assembly work. If the calculated movement amounts from the multiple cameras 21 and 22 differ, for example, an average value is used as the movement amount.

[0025] To improve assembly accuracy, it is desirable to prevent the effects of backlash from occurring when starting this final positioning. In other words, the operator adjusts the reference position of each drive unit that performs final positioning so that the direction of movement up to the reference position is the same as the direction of movement when performing final positioning.

[0026] <Influence of Backlash> The following describes the trajectory in which the influence of backlash occurs during final positioning. FIG. 6 shows an example of a trajectory in which the influence of backlash does not occur, and FIG. 7 shows an example of a trajectory in which the influence of backlash occurs. In FIGS. 6 and 7, the workpiece 30 to be assembled, indicated by the solid line, is in the supply position, and the workpiece 30 to be assembled, indicated by the dashed line, is in the reference position. The workpiece 40 to be assembled in FIGS. 6 and 7 is in a position after completion of rough positioning, and the assembly work is performed at this position. The dashed arrow indicates the trajectory during rough positioning of the workpiece 30 to be assembled, and the solid arrow indicates the trajectory during final positioning of the workpiece 30 to be assembled. In FIGS. 6 and 7, the right direction is the positive X direction, the upward direction is the positive Y direction, and the counterclockwise direction is the positive θ direction. The directions of the coordinate axes are the same as the operating directions of each drive unit.

[0027] In the example of Fig. 6, the direction of movement of all drive units during final positioning is the same as the direction during rough positioning, so the effects of backlash do not occur. On the other hand, in the example of Fig. 7, the directions of movement on the Y-axis and θ-axis during final positioning are the same as those during rough positioning, but the direction of movement on the X-axis is reversed. Therefore, the effects of backlash occur in the X-axis drive unit 501.

[0028] 8 and 9 show modified examples in which final positioning is performed using a positioning stage 51 on which a workpiece 40 is placed. The workpiece 40 is shown by a solid line in the supply position, the workpiece 40 is shown by a dashed line in the reference position, and the workpiece 40 is shown by a thick solid line in the assembly position. In the example of FIG. 8, the direction of movement for rough positioning and the direction of movement for final positioning are the same for all drive units, so no backlash effects occur. On the other hand, in the example of FIG. 9, the direction of movement on the X-axis is reversed, so the effect of backlash occurs. As can be seen from FIGS. 6 to 9, adjusting the direction of movement of the workpiece to the reference position and the reference position is important to avoid the effect of backlash.

[0029] <Hardware Configuration of the Support Device 100> Next, the support device 10 will be described in more detail. The support device 10 supports the worker in making adjustments by displaying appropriate guides when adjusting the reference position to prevent the effects of backlash from occurring. The support device 10 is configured with hardware elements that function as a computer. In detail, as shown in FIG. 10 , the support device 10 has a processor 11, a main memory unit 12, an auxiliary memory unit 13, an input unit 14, an output unit 15, and a communication unit 16. The main memory unit 12, the auxiliary memory unit 13, the input unit 14, the output unit 15, and the communication unit 16 are all connected to the processor 11 via an internal bus 17.

[0030] The processor 11 includes a CPU (Central Processing Unit) as a processing circuit. The processor 11 executes a program P1 stored in the auxiliary storage unit 13 to realize various functions and execute the processes described below.

[0031] The main memory unit 12 includes a RAM (Random Access Memory). A program P1 is loaded into the main memory unit 12 from the auxiliary memory unit 13. The main memory unit 12 is used as a working area for the processor 11.

[0032] The auxiliary storage unit 13 includes a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) and an HDD (Hard Disk Drive). In addition to the program P1, the auxiliary storage unit 13 stores various data used in the processing of the processor 11. The auxiliary storage unit 13 supplies the processor 11 with data used by the processor 11 in accordance with instructions from the processor 11. The auxiliary storage unit 13 also stores data supplied from the processor 11.

[0033] The input unit 14 includes input devices such as hardware switches, input keys, a keyboard, and a pointing device. The input unit 14 acquires information input by a user of the assistance device 10 and notifies the processor 11 of the acquired information.

[0034] The output unit 15 includes output devices such as a light emitting diode (LED), a liquid crystal display (LCD), and a speaker, and presents various information to the user in accordance with instructions from the processor 11.

[0035] The communication unit 16 includes a communication interface circuit for communicating with an external device. The communication unit 16 receives a signal from the outside and outputs data indicated by this signal to the processor 11. The communication unit 16 also transmits a signal indicating the data output from the processor 11 to the external device.

[0036] <Functional Configuration of the Support Device 10> The above-described hardware configurations work together to enable the support device 10 to perform various functions. In detail, as shown in Fig. 11, the support device 10 has, as its functions, a control unit 110 that controls the components of the support device 10, an acquisition unit 120 that acquires information and data, a supply position movement unit 130 that moves the conveying device 50 to a supply position for the workpiece, a supply error correction unit 140 that corrects supply errors, a reference position setting unit 150 that sets a reference position, a display data generation unit 160 that generates display data, and a display unit 170 that displays support information to the worker based on the display data. The control unit 110 is mainly realized by the processor 11.

[0037] The acquisition unit 120 is mainly realized by the processor 11, the input unit 14, and the communication unit 16. The acquisition unit 120 may read information and data from a recording medium such as a memory card. The acquisition unit 120 has an image acquisition unit 121 that acquires images taken by the cameras 20 to 22, an information acquisition unit 122 that acquires information about the transport device 50 and information about the cameras 20 to 22, and a history data acquisition unit 123 that acquires history data indicating the operation history of the transport device 50.

[0038] The image acquisition unit 121 sequentially acquires images periodically captured by the cameras 20 to 22 from the cameras 20 to 22. However, the image acquisition unit 121 may request the cameras 20 to 22 to provide images as needed. As described below, in order for an operator to adjust the reference position, the image acquisition unit 121 simply acquires an image captured near the position that is to become the reference position. The image acquisition unit 121 corresponds to an example of an image acquisition means that acquires a preparatory image captured by a camera of an object at a preparatory position while being transported by a transport device. Here, the preparatory position refers to a position where an object corresponding to a reference workpiece is temporarily placed during the preparatory phase in order to adjust the reference position where the workpiece should be placed during the actual operation phase. In addition to the position of the object, the preparatory position is also used below as appropriate to represent the position of the marker of the object at the preparatory position. Similarly, the reference position is also used below as appropriate to represent the position of the workpiece and the position of the marker of the workpiece at the reference position.

[0039] The information acquisition unit 122 acquires various pieces of information necessary for positioning and various pieces of information necessary for adjustments by the operator from at least one of the operator, the transport device 50, the positioning stage 51, and the controllers that control the transport device 50 and the positioning stage. The information acquired by the information acquisition unit 122 includes information indicating the structure of the transport device 50, information indicating the number, configuration, and specifications of the drive units possessed by the transport device 50, information indicating the workpiece supply position, and an operation program for executing positioning. The workpiece supply position is the position at which the workpiece to be assembled 30 is gripped, and the position of the positioning stage 51 when the workpiece to be assembled 40 is placed on the positioning stage 51. In other words, the workpiece supply position corresponds to the position of each drive unit when the transport device 50 and the positioning stage, which have drive units, are able to move the workpiece.

[0040] The history data acquisition unit 123 acquires history from the transport device 50 and the positioning stage 51 indicating how the transport device 50 and the positioning stage 51 operated in three-dimensional space. For example, the history data acquisition unit 123 acquires history data indicating the movement direction and movement amount of each of the X-axis drive units 511, 501, the Y-axis drive units 512, 502, the θ-axis drive units 513, 503, the Z-axis drive unit 504, and the robot hand 505 in association with past time. The history data acquisition unit 123 corresponds to an example of a history data acquisition means that acquires history data indicating the operation history of the drive units.

[0041] The information acquired by the acquisition unit 120 may be handled based on a common world coordinate system, or may be handled based on a different coordinate system, such as the coordinate system of the camera image. However, if there is information expressed in different coordinate systems, the acquisition unit 120 also acquires information for alternately converting the coordinate systems. The information acquisition unit 122 corresponds to an example of an information acquisition means that acquires information indicating the relationship between the operation of the drive unit and the movement of the object in the image captured by the camera.

[0042] The supply position moving unit 130 is realized mainly by cooperation between the processor 11 and the communication unit 16. The supply position moving unit 130 moves the conveying device 50 and the positioning stage 51 to a supply position for the workpiece based on the information and data acquired by the acquisition unit 120. The supply position of the conveying device 50 is the gripping position of the workpiece 30 to be gripped, and the supply position of the positioning stage 51 is the initial position for receiving the supply of the workpiece 40 to be assembled. These supply positions may be temporary positions input in advance by the operator, or may be set based on design information or a positioning simulation of the support system 100.

[0043] The supply error correction unit 140 is realized mainly by cooperation between the processor 11 and the communication unit 16. The supply error correction unit 140 corrects supply errors and performs rough positioning. The supply errors corrected by the supply error correction unit 140 include gripping errors of the workpiece 30 to be assembled and supply errors of the workpiece 40 to be assembled. Specifically, if the supply position is outside the camera's image capture range, the supply error correction unit 140 operates the conveyance device 50 to bring the supply position within the image capture range. The supply error correction unit 140 then recognizes the workpiece based on the image captured by the camera 20, calculates a supply error correction amount, and moves the workpiece to a reference position corrected by the calculated supply error correction amount. The supply error correction method is as described above. The operating positions of each drive unit of the conveyance device 50 and the positioning stage 51 at the time the workpiece is received at the supply position are stored. The operating positions of each drive unit may be tentative positions input in advance by the operator, or may be set based on design information or simulations of the assistance system 100.

[0044] The reference position setting unit 150 is mainly realized by cooperation between the processor 11 and the communication unit 16. The reference position setting unit 150 is used by an operator to adjust a reference position for final positioning using markers and set a reference position that is not affected by backlash. If the markers 31 and 32 of the workpiece 30 to be assembled and the markers 41 and 42 of the workpiece 40 to be assembled are not within the imaging range of the cameras 21 and 22, the reference position setting unit 150 moves the workpiece so that they are within the imaging range. The position to which the workpiece is moved may be a temporary position input in advance by the operator, or may be set based on design information or a simulation of the support system 100. When the operator adjusts the reference position to set the reference position, the reference position setting unit 150 calls the display data generation unit 160 to perform its function.

[0045] The display data generating unit 160 is mainly realized by the processor 11. The display data generating unit 160 generates display data including a guide for the worker to adjust the reference position, and causes the display unit 170 to display the contents of the display data. In detail, the display data generating unit 160 calculates the movement direction and movement amount of each drive unit when a reference workpiece used to adjust the reference position moves from the supply position to the current preparation position, and generates display data for presenting the calculated movement direction and movement amount to the worker as an operation guide, as shown in Fig. 12. Fig. 12 includes an image captured by the camera 21, and the workpiece 40 to be assembled is represented by a thick line, and the workpiece 30 to be assembled is represented by a thin line.

[0046] The upper right corner of this image shows the movement direction and movement amount of each drive unit calculated based on the captured image and the drive unit configuration information. The arrows indicate the movement direction. In the example of Figure 12, the arrows indicate that the drive unit moved -100 mm in the X-axis direction, +500 mm in the Y-axis direction, and 5 degrees counterclockwise. As long as the drive unit moves in the direction indicated by the arrow during final positioning, the effects of backlash will not occur. If the drive unit moves in the opposite direction to the arrow during final positioning, the displayed value also indicates the margin to the reference position where the effects of backlash will not occur. That is, as shown by the dashed line in Figure 13, when the marker moves from the supply position to the assembly position via the preparation position, the movement direction in the X-axis direction is reversed. However, if the reference position is set on the opposite side of the preparation position relative to the supply position, the movement direction in the X-axis direction will be maintained when passing through the reference position. If the reference position is set in the opposite direction to the preparation position relative to the preparation position, beyond the margin, the effects of backlash will not occur. By visually checking the operation guide as shown in FIG. 12, the operator can easily adjust the reference position to avoid the occurrence of backlash.

[0047] The display data generating unit 160 also generates display data for displaying an area corresponding to the reference position of the marker where the influence of backlash does not occur, superimposed on the captured image, as shown by the dashed line in Fig. 14. If the center point of the marker 31 of the workpiece 30 to be assembled is within this area, the influence of backlash does not occur.

[0048] When displaying the region, the display data generation unit 160 distinguishes between movable markers that move within the captured image when a drive unit responsible for final positioning is operated, and fixed markers that do not move. The markers may be distinguished by the worker specifying a marker from within the captured image, or if the two markers have different shapes, colors, or patterns, the marker type may be specified in advance. Since the robot hand 505 gripping the workpiece 30 to be assembled performs final positioning, the marker 31 of the workpiece 30 to be assembled is determined to be a movable marker, and the marker 41 of the workpiece 40 to be assembled is determined to be a fixed marker.

[0049] The display data generation unit 160 has a region calculation unit 161 that calculates a region based on the position and orientation of the moving marker, the position and orientation of the fixed marker, and the amount of movement of the drive unit immediately before the workpiece reaches the preparation position for adjustment. An example of calculating a region free from the influence of backlash by the region calculation unit 161 will be described below.

[0050] The calculation of the area differs depending on whether the direction of movement from the movable marker to the fixed marker, i.e., the direction in which the workpiece should be transported via the reference position when the preparation position is the reference position, corresponds to the immediately preceding operating direction of each drive unit, resulting in no backlash, or whether these directions do not correspond and backlash occurs. When these directions correspond, the area calculation unit 161 first calculates a rotation matrix M that causes the orientation of the marker 31 on the workpiece 30 to be assembled to be the same as the orientation of the marker 41 on the workpiece 40, as shown in FIG. 15 , and then calculates the position of the marker 31 when only the orientation is changed without translation. Then, as shown in FIG. 16 , the area calculation unit 161 sets a translation-enabled area starting from the fixed marker and extending in the direction of the movable marker. The translation-enabled area extends beyond the movable marker by the amount of movement displayed in the operation guide. In the example of Figure 16, if the amount of parallel movement from the fixed marker to the moving marker whose posture has been changed is +75 mm in the X-axis direction and -50 mm in the Y-axis direction, the moving marker has a size of +175 mm in the X-axis direction and -550 mm in the Y-axis direction from the fixed marker.

[0051] The area shown in FIG. 14 is calculated by changing the orientation by multiplying the translational area by the inverse matrix of the calculated rotation matrix M. Note that, because the center of rotation when rotating the orientation is different from the position of the marker 31, the calculated area may deviate from the marker. It is not necessary to actually rotate the workpiece when calculating the area; the area can be calculated from the image captured by the camera and the calibration information of the drive unit. This calculation example is merely an example, and a similar area may be calculated by combining both translation and rotation.

[0052] If the direction of movement from the moving marker to the fixed marker does not correspond to the direction of movement of each drive unit immediately before, the method of calculating the area differs depending on whether the amount of movement from the moving marker to the fixed marker, i.e., the distance between the markers, is greater than the amount of movement displayed in the operation guide. Hereinafter, the amount of movement displayed in the operation guide may be referred to as the margin.

[0053] When the distance between the markers is greater than the margin, moving the moving marker toward the fixed marker reverses the direction in which the effects of backlash do not occur, as shown in Fig. 13, and therefore the region is calculated in the same manner as in the example of Fig. 14. For example, in the example of Fig. 17, if the distance between the markers after aligning their orientations is +75 mm in the X-axis direction and -50 mm in the Y-axis direction, a translation-enabled region is calculated that has a size of +55 mm in the X-axis direction and -550 mm in the Y-axis direction, starting from the fixed marker.

[0054] When the distance between the markers is smaller than the margin, a translational region is calculated from the fixed marker in the opposite direction to the moving marker. In the example of Figure 18, if the distance between the markers after alignment is +75 mm in the X-axis direction and -50 mm in the Y-axis direction, a translational region is calculated that has a size of -125 mm in the X-axis direction and -550 mm in the Y-axis direction, starting from the fixed marker.

[0055] Whether the inter-marker distance is greater than or less than the margin, the calculated translational movement region is multiplied by the inverse matrix of the rotation matrix M to calculate the region free from the influence of backlash. The display of the region is not limited to the display of a rectangular frame that should contain the center point of the marker, as shown in Figures 14 and 16 to 18, but may also display a reference point for calculating the position and orientation of the marker and the region that should contain the reference point, or may display a region that should contain the entire marker taking into account the size of the marker. The region is a candidate position that is a candidate for the reference position, and corresponds to an example of a region in the preparatory image that corresponds to a candidate position where the operating direction of the drive unit of the conveyance device is maintained before and after the marker is photographed and identified.

[0056] The display data generator 160 also generates display data for superimposing and displaying information indicating the orientation of the fixed marker relative to the moving marker and the tilt of the region on the captured image of the camera. Because the tilt of the region is not quantitatively displayed in the captured image, setting the tilt can be a tedious task for the operator when adjusting the reference position. For example, while a small amount of change in the orientation of the moving marker is preferable, if the margin of error in the rotational direction is small, backlash may occur if an error occurs during final positioning. Therefore, the display data generator 160 calculates the amount of orientation change from the current orientation of the moving marker in the captured image to the orientation of the fixed marker, and generates display data for display as shown in FIG. 19 . This makes it easier for the operator to set a reference position with a sufficient margin of error.

[0057] The display data generating unit 160 also generates display data for superimposing auxiliary information on the captured image to assist the worker in his or her work. The auxiliary information is, for example, information that suggests expanding the calculated area by setting a waypoint between the preparation position and the assembly position instead of the assembly position as the target position for moving the workpiece from the current preparation position when the calculated area is narrow. The information used to determine whether the area is narrow includes the command unit amount of the drive unit, the field of view of the captured image, and the size of the workpiece and marker. For example, the area is determined to be narrow when the size of the area is less than several tens of times the command unit of the drive unit, when the number of pixels of the area calculated from the field of view is less than several tens, or when the size of the area is equal to or smaller than the size of the workpiece or marker to be recognized. The threshold value used for this determination may be set in advance by the worker.

[0058] The auxiliary information may also include a suggestion to reduce the attitude change if the amount of attitude change at final positioning is large and most of the area falls outside the imaging range. As shown in FIG. 17 , the auxiliary information may also include a notification that the movement direction will be reversed if the movement direction of the marker differs from the direction in which the drive unit operates, and a warning that, because backlash will occur if the current preparation position is set as the reference position as is, reconfirmation should be performed after adjustment to ensure that there is no backlash. In the example of FIG. 18 , the auxiliary information may also include a notification that the approach direction will be reversed. Furthermore, if the marker on the workpiece 40 to be assembled or the workpiece 30 to be assembled is hidden and image recognition is impossible, the auxiliary information may also include a suggestion to adjust the position so that the marker can be recognized.

[0059] Returning to FIG. 11 , the display unit 170 is mainly realized by the output unit 15. By displaying based on the display data on the display unit 170, the operator can follow the guide to recognize the marker and easily adjust the reference position so that the influence of backlash does not occur. The operator sets the adjusted reference position in the reference position setting unit 150. After the reference position is set, for example, in an automated process, each of the workpieces that are sequentially supplied is photographed at the reference position by the cameras 21 and 22, and then final positioning is performed.

[0060] <Assistance Processing> Next, the assistance processing executed by the assistance device 10 will be described with reference to Fig. 20. This assistance processing corresponds to an example of an assistance method executed by the assistance device.

[0061] In the support process, the support device 10 transports the reference workpiece 30 to the preparation position and causes the cameras 21 and 22 to photograph it (step S1). Note that when transporting it to the preparation position, a supply error is corrected.

[0062] Next, the acquisition unit 120 acquires various information and data (step S2). Specifically, the acquisition unit 120 acquires an image of the workpiece at the preparation position, acquires information for converting the coordinate system, and acquires the operation history of the conveyance device 50.

[0063] Next, the display data generating unit 160 causes the display unit 170 to display an operation guide as shown in FIG. 12 (step S3).

[0064] Next, the display data generator 160 determines whether the first movement direction of the marker identified by photographing the workpiece moved to the preparation position is equal to the second movement direction to which the marker should move next (step S4). If it is determined that the first movement direction is equal to the second movement direction (step S4; Yes), the display data generator 160 calculates an area by defining a range from the fixed marker as the starting point toward the moving marker (step S5), as shown in FIG. 16. The range definition is performed for each of the X and Y axes. Here, if the orientation of the marker can be identified, step S5 may include calculating the rotation matrix M as described above and performing an operation using the inverse matrix of the rotation matrix M. Alternatively, if a marker that does not represent orientation is used, the operation related to the rotation matrix M may be omitted.

[0065] If it is determined that the first movement direction is not equal to the second movement direction (step S4: No), the display data generating unit 160 determines whether the movement amount of the marker of the workpiece to be moved next from the preparation position is greater than the margin corresponding to the movement amount of the marker identified by photographing the workpiece moved to the preparation position (step S6).If it is determined that the movement amount is greater than the margin (step S6; Yes), the display data generating unit 160 executes step S5.

[0066] On the other hand, if it is determined that the amount of movement is not greater than the margin (step S6; No), the display data generation unit 160 calculates the area by defining a range starting from the fixed marker in the opposite direction to the moving marker, as shown in Figure 18 (step S7).

[0067] The display data generating unit 160 corresponds to an example of a display data generating means that calculates an area based on the current position of a marker on a workpiece at the preparation position, the target position for aligning the marker, and a first movement direction and a first movement amount of the marker immediately before the workpiece reaches the preparation position. Here, the current position and the target position are positions on the X-axis and the Y-axis, respectively. Furthermore, the X-axis and the Y-axis correspond to examples of movement axes of the marker corresponding to the operation of the drive unit, and the movement direction and movement amount from the moving marker to the fixed marker correspond to examples of a second movement direction and second movement amount. Furthermore, the display data generation unit 160 is an example of a display data generation means that calculates an area by defining a range from the target position as a starting point to an end point that is away by the first movement amount in the direction opposite to the current position when the second movement direction is equal to the first movement direction, and when the second movement direction is different from the first movement direction and the second movement amount is greater than the first movement amount, and that calculates an area by defining a range from the target position as a starting point to an end point that is away by the first movement amount in the direction opposite to the current position when the second movement direction is different from the first movement direction and the second movement amount is smaller than the first movement amount.

[0068] Following steps S5 and S7, the display data generator 160 displays the calculated area on the display unit 170 (step S8). Next, the display data generator 160 displays the posture change amount and auxiliary information on the display unit 170 as needed (step S9). Then, the assistance device 10 accepts the reference position set by the worker based on the assistance information provided in steps S3, S8, and S9 (step S10). Thereafter, the assistance process ends.

[0069] <Effects> As described above, support information including operation guides, areas, posture change amounts, and auxiliary information is provided to the worker along with the captured image. This makes it easier to adjust the capture position of the object to be captured while maintaining the operating direction of the drive unit in a fixed direction. Furthermore, it is possible to appropriately set the reference position, which is the initial position of the workpiece in the alignment process using a camera, and improve the accuracy of workpiece transport.

[0070] When an operator adjusts the reference position while visually viewing the captured image without receiving assistance information, the operator must make the adjustment while taking into consideration various information, such as the operating direction of each drive unit, the coordinate system for image recognition, and the relationship between the amount of correction in image recognition and the amount of movement required to prevent the effects of backlash. However, such adjustment work is difficult and prone to errors. Furthermore, when corrections are made consecutively through multiple image recognitions, the position of the subject to be photographed in the second and subsequent image recognitions is corrected based on the previous image recognitions. Therefore, the photographing position of the subject to be adjusted must be set, including the amount of correction, making the adjustment difficult.

[0071] In contrast, with the assistance device 10 according to the present embodiment, the worker can intuitively understand the conditions under which the direction of movement of the drive unit is maintained and the influence of backlash does not occur while visually checking the captured image, and can therefore easily adjust the reference position where the object to be captured is located while being transported to the target position.

[0072] Second Embodiment Next, a second embodiment will be described, focusing on the differences from the first embodiment described above. Note that the same reference numerals are used for configurations that are the same as or equivalent to those in the first embodiment described above. This embodiment differs from the first embodiment described above in that a display including an expected position error for the workpiece is provided.

[0073] During operation after adjustment work is completed, various positioning errors occur, affecting operation. Adjustment work is usually performed by simulating the supply or operation of one workpiece. In contrast, during actual operation, in which multiple workpieces are processed sequentially, errors occur in the workpiece position. Adjustment work must be performed with a margin of error to deal with such position errors. Therefore, by displaying the expected position errors, such as gripping errors and correction errors due to alignment, the efficiency of adjustment work can be improved.

[0074] The information acquisition unit 122 according to this embodiment acquires error information indicating an expected error. The information acquisition unit 122 may acquire a value input by an operator as the expected error, or may acquire the expected error by calculating a statistical value of errors measured when an actual operation is performed. The information acquisition unit 122 corresponds to an example of an information acquisition means for acquiring error information indicating an error in the position of an object photographed by a camera.

[0075] 21, the display data generating unit 160 generates display data for displaying the range of positional error that may occur at the center point of the marker 31 of the workpiece 30. This range is calculated by calculating the position and orientation of the marker when the expected error is added to the marker 31 being recognized in the image, based on the acquired expected error, the position and orientation of the moving marker, and the calibration information of the captured image. The display data generating unit 160 corresponds to an example of a display data generating means for generating display data for displaying the error range corresponding to the error together with the preparatory image.

[0076] By making adjustments so that the range of influence of the displayed error falls within the area indicated by the dashed line, the operator can adjust the reference position so that the effects of backlash do not occur even if the position of the workpiece fluctuates within the expected error range.

[0077] Furthermore, the display data generating unit 160 may change the display of the area itself to take into account the influence of the error, rather than displaying the range of influence of the error separately from the calculated area. Specifically, the calculated area may be reduced by an amount corresponding to the expected error. By performing adjustment work so that the center point of the marker is within the reduced area, the operator can adjust the reference position so that the influence of backlash does not occur even if the position of the workpiece fluctuates within the range of the expected error. The display data generating unit 160 corresponds to an example of a display data generating means that generates display data for reducing and displaying the area in accordance with the error.

[0078] In addition, as shown in Figure 21, when part of the error range is outside the calculated area, the display data generation unit 160 may provide the worker with a notification in the auxiliary information that there is a possibility of backlash effects occurring.

[0079] As described above, by generating display data taking into account expected errors, it becomes easier to adjust the reference position to take into account errors that may occur during operation. The above-mentioned range of position error corresponds to an example of an error range.

[0080] Embodiment 3. Next, Embodiment 3 will be described, focusing on the differences from the above-described Embodiment 1. Note that the same reference numerals are used for configurations that are the same as or equivalent to those in the above-described Embodiment 1. This embodiment differs from the above-described Embodiment 1 in that markers photographed during multiple test runs are superimposed on the photographed images.

[0081] Since variations occur due to various errors during actual operation, adjustment work may involve multiple test runs to check whether there are any problems with the adjustment. In such cases, by displaying the adjustment results from multiple test runs together, it becomes easier to check variations and work efficiency can be improved.

[0082] In this embodiment, the worker inputs instructions to execute multiple test runs into the assistance device 10. In accordance with these instructions, the assistance device 10 executes multiple test runs and records the positions and orientations of the markers photographed during the runs. Then, the display data generator 160 generates display data for displaying the markers photographed during the multiple test runs superimposed on the photographed images, as shown in FIG.

[0083] The operator checks the variation from the displayed position and orientation of the markers and adjusts the parameters, including the gain of the drive unit, and the reference position as necessary to prevent the effects of backlash. Specifically, if the range of variation is large, the parameters of the drive unit and the jig used to supply the workpiece are adjusted to suppress the variation. If any of the captured markers falls outside the calculated area, the reference position is corrected. Examples of such adjustment methods may be displayed as auxiliary information.

[0084] The display data generating unit 160 may display the area in which the marker appears, instead of displaying all of the positions and orientations of the markers in multiple test runs as marker shapes. For example, it may display a convex polygon that includes all of the marker shapes in all trials.

[0085] As described above, by collectively displaying the positions and orientations of the markers photographed during multiple test runs, the operator can easily grasp variations during operation and easily adjust the reference position. The display data generation unit 160 corresponds to an example of a display data generation means that generates display data for displaying the results of sequentially photographing multiple objects together with a preparation image.

[0086] Fourth Embodiment Next, a fourth embodiment will be described, focusing on the differences from the first embodiment described above. Note that the same reference numerals are used for configurations that are the same as or equivalent to those in the first embodiment described above. This embodiment differs from the first embodiment described above in that the auxiliary information provides a suggestion to correct the installation orientation of the workpiece at the assembly position.

[0087] When multiple markers are positioned simultaneously, there may be no reference position that does not cause the effects of backlash depending on the installation position of the workpiece. For example, as shown in Figure 23, if there are markers at the upper left and lower right of the workpiece at the assembly position, in order to move the workpiece 30 to be assembled without hiding the markers of the workpiece 40 to be assembled, it is necessary to transport the workpiece 30 from the upper right or lower left. However, if the supply position of the workpiece 30 to be assembled is the lower right, it is necessary to set a via point to transport the workpiece 30 without causing the effects of backlash. Adding via points is not desirable because it increases the operating time per product during operation.

[0088] Here, if the workpiece 40 is rotated by 90 degrees, it can be transported from the lower right without passing through any intermediate points, as in the first embodiment.

[0089] The display data generating unit 160 according to the present embodiment generates display data for suggesting to the worker a change in the installation direction of the workpiece when it determines that the direction of movement from the supply position differs from the direction of movement at the time of final positioning while keeping all markers within the imaging range. At this time, the display data generating unit 160 may suggest a change in the posture so that the marker of the workpiece 40 to be assembled is set in the direction of movement indicated in the operation guide.

[0090] As described above, if a suggestion to change the workpiece installation position is made, the worker can efficiently adjust an appropriate reference position based on the new workpiece installation position without adjusting the reference position or adding a waypoint even though an appropriate reference position does not exist. The suggestion may be to change only the position of the workpiece, only the orientation, or both. The display data generation unit 160 corresponds to an example of a display data generation means that generates display data to suggest changing at least one of the position and orientation of the workpiece, which are the target positions for alignment, when the position of a feature point of an object in the preparation position is the reference position and the operating direction of the drive unit of the conveyance device is different before and after the feature point is identified.

[0091] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments.

[0092] For example, in the above embodiments, the objects photographed by the cameras 21 and 22 were portions of a workpiece or markers. However, this is not limited to this. They may be the entire workpiece or a point identified from an image of the workpiece. The workpiece does not need to have a marker attached. Furthermore, if the position of the workpiece can be effectively controlled by photographing a marker on a stage corresponding to the workpiece's position, the workpiece itself need not be photographed. It is sufficient that a feature point that defines at least one of the workpiece's position and orientation is identified from the photographed image. In the above embodiments, the position and orientation of a horizontal, flat workpiece on the XY plane are calculated by image recognition of two markers corresponding to the feature points. However, three or more markers may be identified to calculate the orientation including the elevation angle of the workpiece. If the workpiece is spherical, the center point of the projection plane of the photographed workpiece may be used as the feature point that defines the position of the workpiece, and orientation may not be considered. If the workpiece's rotation axis is fixed, a feature point that defines only the orientation may be identified, and position may not be considered. The support device 10 is an example of a support device that identifies feature points that determine at least one of the position and posture of the work by photographing the work transported by a transport device having a drive unit with a camera, and supports the adjustment of the reference position where the identified feature points should initially be located in the process of aligning the feature points by further transporting the work by the transport device.

[0093] In the above embodiment, the width and height directions of the captured image correspond to the operating directions of the X-axis drive unit 511 and the Y-axis drive unit 512, and these operating directions correspond to the movement direction of the object being captured in the image. However, this is not limiting, and the operating direction of the drive units and the movement direction of the object being captured in the image may be different. When these operating directions and movement directions are different, it is sufficient if the movement direction corresponding to the operating direction is displayed together with the captured image.

[0094] Furthermore, part of the display content of the display data may be omitted. For example, only the operation guide as shown in FIG. 12 may be displayed without displaying the area as shown in FIG. 14. Furthermore, in the operation guide, only the direction may be displayed, and the amount of movement may be omitted. Furthermore, even if only the area as shown in FIG. 14 is displayed without displaying the operation guide, it can contribute to improving the efficiency of the reference position adjustment work by the operator.

[0095] The directions included in the operation guide are shown in the coordinate system of the captured image so that the worker can intuitively understand them, so it is desirable that the direction of movement of the object being photographed in the image coincides with the operating direction of the drive unit, or the direction of movement of the object being photographed corresponds to the operating direction of the drive unit.

[0096] The amount of movement included in the operation guide may be changed to a movement amount of the object captured in the image corresponding to the amount of movement, or the movement amount may be displayed together with the amount of movement. Specifically, instead of or together with the amount of movement in millimeters, the number of pixels corresponding to the amount of movement may be displayed as the movement amount. The display data generation unit 160 corresponds to an example of a display data generation means that generates, based on the information acquired by the information acquisition means and the history data acquired by the history data acquisition means, display data for displaying, together with the preparatory image, at least one of the movement direction of the feature point of the object in the preparatory image corresponding to the movement direction of the drive unit just before the object reaches the preparatory position, the movement amount of the drive unit just before the object reaches the preparatory position, and the movement amount of the feature point in the preparatory image corresponding to the movement amount.

[0097] In addition, the display data generation unit 160 corresponds to an example of a display data generation means that generates display data for displaying, together with the preparation image, at least one of the movement direction of the feature point of the object in the preparation image, which corresponds to the operating direction of the drive unit just before the object to be photographed reaches the preparation position, and an area in the preparation image that corresponds to a candidate position that is a candidate for the reference position and where the operating direction of the drive unit of the conveying device is maintained before and after the feature point is identified, based on the information acquired by the information acquisition means and the history data acquired by the history data acquisition means.

[0098] Furthermore, although the example in which the position and orientation of the workpiece or marker are obtained by image recognition has been described, the present invention is not limited to this. For example, when the position of a spherical or cylindrical workpiece is obtained by image recognition, it is not necessary to recognize the orientation.

[0099] Furthermore, although the above description has focused on an example in which the reference position is adjusted when the workpiece is transported from the supply position to the target position, which is the assembly position, via the reference position, the present invention is not limited to this. For example, when the object to be photographed is transported from a first way point to a second way point to a third way point, the second way point may correspond to the reference position, and the third way point may correspond to the target position.

[0100] Furthermore, the support device 10 may be configured without the display unit 170, and the support device 10 may output display data to an external display device.

[0101] In addition, the support device 10 may be configured by omitting the supply position moving unit 130, the supply error correction unit 140, and the reference position setting unit 150, and the support device 10 may support the worker by linking with an external device that has functions equivalent to those of these components.

[0102] The functions of the support device 10 according to the above-described embodiment can be realized by dedicated hardware or by an ordinary computer system.

[0103] For example, by storing and distributing program P1 on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), or an MO (Magneto-Optical disk), and installing program P1 on a computer, a device that executes the above-mentioned processing can be configured.

[0104] Furthermore, the program P1 may be stored in a disk device of a server device on a communication network such as the Internet, and may be downloaded to a computer by superimposing it on a carrier wave, for example.

[0105] The above process can also be achieved by starting and executing the program P1 while transferring it via a network such as the Internet.

[0106] Furthermore, the above-described processing can also be achieved by executing all or part of program P1 on a server device, and executing program P1 while the computer sends and receives information about the processing via a communications network.

[0107] In addition, when the above-mentioned functions are realized by an operating system (OS) or by the OS working together with an application, only the parts other than the OS may be stored on a medium and distributed, or may be downloaded to a computer.

[0108] Furthermore, the means for realizing the functions of the assistance device 10 is not limited to software, and some or all of the functions may be realized by dedicated hardware or circuits.

[0109] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure.

[0110] The present disclosure is suitable for tasks related to positioning.

[0111] 10 Support device, 11 Processor, 12 Main memory unit, 13 Auxiliary memory unit, 14 Input unit, 15 Output unit, 16 Communication unit, 17 Internal bus, 20 to 22 Camera, 30 Work to be assembled, 31, 32, 41, 42 Marker, 40 Work to be assembled, 50 Conveyance device, 51 Positioning stage, 100 Support system, 110 Control unit, 120 Acquisition unit, 121 Image acquisition unit, 122 Information acquisition unit, 123 History data acquisition unit, 130 Supply position movement unit, 140 Supply error correction unit, 150 Reference position setting unit, 160 Display data generation unit, 161 Area calculation unit, 170 Display unit, 211, 221 Area, 501, 511 X-axis drive unit, 502, 512 Y-axis drive unit, 503, 513 θ-axis drive unit, 504 Z-axis drive unit, 505 robot hand, P1 program.

Claims

1. In a process of aligning a feature point that defines at least one of the position and orientation of a workpiece by photographing the workpiece conveyed by a conveying device having a driving unit with a camera and further conveying the workpiece by the conveying device, a support device for assisting in adjusting a reference position where the identified feature point should be initially located, the support device comprising: an image acquisition unit that acquires a preparation image obtained by photographing an object corresponding to the workpiece and located at a preparation position during conveyance by the conveying device with the camera; an information acquisition unit that acquires information indicating a relationship between an operation of the driving unit and movement of the object in an image photographed by the camera; a history data acquisition unit that acquires history data indicating an operation history of the driving unit; and a display data generation unit that generates display data for displaying at least one of a moving direction of the feature point of the object in the preparation image corresponding to an operation direction of the driving unit immediately before the object reaches the preparation position based on the information and the history data, and a region in the preparation image corresponding to a candidate position that is a candidate for the reference position and where the operation direction of the driving unit of the conveying device is maintained before and after the feature point is identified, together with the preparation image.

2. The support device according to claim 1, wherein the display data generation unit generates the display data for displaying at least one of a moving direction of the feature point of the object in the preparation image corresponding to an operation direction of the driving unit immediately before the object reaches the preparation position, an operation amount of the driving unit immediately before the object reaches the preparation position, and a moving amount of the feature point in the preparation image corresponding to the operation amount, together with the preparation image, based on the information and the history data.

3. The support device according to claim 1 or 2, wherein the display data generation unit calculates the region based on a current position of the feature point of the object at the preparation position, a target position for alignment of the feature point, a first moving direction and a first moving amount of the feature point immediately before the object reaches the preparation position.

4. The current position and the target position are positions on the movement axis of the feature point corresponding to the operation of the drive unit. The first movement direction is a direction on the movement axis. The first movement amount is a movement amount on the movement axis. The display data generation means calculates the area by defining a range from the target position as a starting point to an end point that is separated from the starting point by the first movement amount in the direction of the current position when the second movement direction from the current position to the target position is equal to the first movement direction, and when the second movement direction is different from the first movement direction and the second movement amount from the current position to the target position is greater than the first movement amount. The display data generation means calculates the area by defining a range from the target position as a starting point to an end point that is separated from the starting point by the first movement amount in the direction opposite to the current position when the second movement direction is different from the first movement direction and the second movement amount is smaller than the first movement amount. The support device according to claim 3.

5. The information acquisition means acquires error information indicating an error in the position of the object photographed by the camera. The display data generation means generates the display data for displaying an error range corresponding to the error together with the preparation image. The support device according to any one of claims 1 to 4.

6. The information acquisition means acquires error information indicating an error in the position of the object photographed by the camera. The display data generation means generates the display data for reducing and displaying the area in correspondence with the error. The support device according to any one of claims 1 to 4.

7. The display data generation means generates the display data for sequentially photographing a plurality of the objects and displaying the results together with the preparation image. The support device according to any one of claims 1 to 6.

8. When the position of the feature point of the object at the preparation position is the reference position, the display data generation means generates the display data for proposing to change at least one of the position and the posture that are the target of the workpiece alignment when the operation direction of the drive unit of the transfer device is different before and after the feature point is specified. The support device according to any one of claims 1 to 7.

9. The support device according to any one of claims 1 to 8, wherein the feature point is a marker attached to the workpiece.

10. A support system comprising: a camera; a transport device having a drive unit for transporting a workpiece; and a step of identifying a feature point that defines at least one of the position and orientation of the workpiece by photographing the workpiece transported by the transport device with the camera, and aligning the feature point by further transporting the workpiece with the transport device, and the support device according to any one of claims 1 to 9 for assisting in adjusting a reference position where the identified feature point should be initially located.

11. A support method for assisting in adjusting a reference position where a feature point to be identified should be initially located in a step of identifying a feature point that defines at least one of the position and orientation of a workpiece by photographing the workpiece transported by a transport device having a drive unit with a camera, and aligning the feature point by further transporting the workpiece with the transport device, the method comprising: an image acquisition means acquiring a preparation image of an object corresponding to the workpiece and located at a preparation position during transportation by the transport device, photographed by the camera; an information acquisition means acquiring information indicating a relationship between an operation of the drive unit and a movement of the object in an image photographed by the camera; a history data acquisition means acquiring history data indicating an operation history of the drive unit; and a display data generation means generating display data for displaying at least one of a moving direction of the feature point of the object in the preparation image corresponding to an operation direction of the drive unit immediately before the object reaches the preparation position, and a region in the preparation image corresponding to a candidate position that is a candidate for the reference position and where the operation direction of the drive unit of the transport device is maintained before and after the feature point is identified, together with the preparation image, based on the information and the history data.

12. In a process of aligning feature points by identifying at least one of the position and orientation of a workpiece by photographing the workpiece conveyed by a conveying device having a driving unit and further conveying the workpiece by the conveying device, a support device for assisting adjustment of a reference position where the identified feature points should be initially located, an image acquisition unit that acquires a preparation image obtained by photographing an object corresponding to the workpiece and located at a preparation position during conveyance by the conveying device with the camera, an information acquisition unit that acquires information indicating the relationship between the operation of the driving unit and the movement of the object in the image photographed by the camera, a history data acquisition unit that acquires history data indicating the operation history of the driving unit, and display data generation means for generating display data for displaying at least one of the moving direction of the feature points of the object in the preparation image corresponding to the operation direction of the driving unit immediately before the object reaches the preparation position, and a region in the preparation image corresponding to a candidate position that is a candidate for the reference position and where the operation direction of the driving unit of the conveying device is maintained before and after the feature points are identified, together with the preparation image. A program for causing the above to function.

Images