Collaborative motion assist device
Through the use of collaborative action assist devices, operators can understand the settings and execution of robot movements in real time through the head-mounted display and camera, solving the problem that operators in the prior art have difficulty in intuitively grasping robot movements, and improving operation efficiency.
Patent Information
- Application Number
- CN201910702015.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-03
- Filing Date
- 2019-07-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-07-31
AI Technical Summary
During the robot teaching and collaborative operation, it is difficult for operators to intuitively grasp the settings and actual execution of the robot's movements, resulting in a decrease in operation efficiency.
Provide a collaborative action assist device, displaying relevant data through a head-mounted display, allowing the operator to visually recognize the robot's actions, and detect the position of the robot and the operator through a camera, and combine the signal of the input device to display the form and control of the robot's actions in real time. parameter.
The operator can intuitively understand the settings and execution of the robot's actions, improve operation efficiency, and reduce dependence on the teaching device.
Smart Images

Figure CN110788836B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to, for example, a collaborative operation assisting device that assists collaborative operation with a robot. Background Art
[0002] When an operator teaches a robot to perform an action, or when such a robot works in collaboration with an operator, the operator sometimes manually causes the robot to perform a lead-through action to change the position or posture of the robot, wherein the robot has a movable part such as a robotic arm, and the movable part is driven by a servo motor.
[0003] When teaching a robot an action through a leading action, the operator generally operates an input device called a teach pendant in order to confirm the settings related to the robot's action or to input the value of a control parameter for setting the robot's action. However, in order to operate the teach pendant, the operator needs to temporarily interrupt the ongoing operation, so the operation efficiency is reduced. Therefore, the following technology is proposed: extracting words that indicate the robot's action from the operator's voice and generating data for instructing the robot to act (for example, refer to Japanese Patent Publication No. 2016-289543).
[0004] However, in the above-mentioned technology, it is difficult to intuitively understand how the robot actually moves relative to the control parameters set by the operator. For example, even if a predetermined viscosity value is set in a predetermined direction based on a tool mounted on the robot, it is difficult for the operator to intuitively understand which direction the predetermined direction is in the actual space and to what extent the robot's movement changes according to the set viscosity value. Summary of the invention
[0005] In one aspect, an object of the present invention is to provide a collaborative motion assisting device that allows an operator to easily and intuitively grasp settings related to the motion of a robot.
[0006] According to one embodiment, a collaborative action assisting device is provided. The collaborative action assisting device comprises: a storage unit storing display data for representing the robot's action in each form when the robot and an operator perform collaborative action; a display unit worn by an operator so that the operator can visually recognize the robot through a display area of the display unit; a photographing unit photographing the robot or the operator to generate an image representing the robot or the operator; a display position setting unit detecting, based on the image generated by the photographing unit, the position of a part of the robot associated with the form of the robot's action specified via an input device on a display area when the operator visually recognizes the robot through the display area; a display data selecting unit selecting display data corresponding to the specified form from the display data stored in the storage unit in accordance with the specified form of the robot's action; and a display control unit causing the display area of the display unit to display the selected display data in a predetermined positional relationship with the position of the part of the robot on the display area.
[0007] According to one aspect, the operator can easily and intuitively grasp the settings related to the operation of the robot. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 This is a schematic structural diagram of a collaborative action assisting device according to one embodiment.
[0009] Figure 2 This is a functional block diagram of a processor related to collaborative action auxiliary processing.
[0010] Figure 3A An example of a form that indicates leading action.
[0011] Figure 3B Another example of a form that indicates leading action.
[0012] Figure 4A An example of display data displayed on the head mounted display is shown with respect to each axis guide.
[0013] Figure 4B Regarding the orthogonal guide, an example of display data displayed on the head mounted display is shown.
[0014] Figure 4C An example of display data displayed on the head mounted display when the values of the control parameters of the servo motors of the respective axes are different during the respective axis driving is shown.
[0015] Figure 5 It is an action flow chart of collaborative action assisted processing. DETAILED DESCRIPTION
[0016] The following describes a collaborative action assisting device with reference to the accompanying drawings. The collaborative action assisting device enables the operator to intuitively grasp the settings related to the actions of the robot by displaying display data on a transmissive display worn by the operator, wherein the display data is associated with a part of the robot associated with a specified action form of the robot performing collaborative actions with the operator, and indicates the actions of the robot under the specified action form.
[0017] Figure 1 1 is a schematic structural diagram of a collaborative action assisting device. The collaborative action assisting device 1 includes an input device 2, a head-mounted display 3, a camera 4, a wireless communication interface 5, a communication interface 6, a memory 7, and a processor 8. Among them, the wireless communication interface 5, the communication interface 6, the memory 7, and the processor 8 are installed in, for example, a control device 12 of a robot 11 that is a teaching object or works in collaboration with an operator. In addition, the wireless communication interface 5, the communication interface 6, and the memory 7 are connected to the processor 8 via a signal line so as to be able to communicate with each other. In addition, the input device 2, the head-mounted display 3, and the camera 4 are connected to the processor 8 via the wireless communication interface 5 so as to be able to communicate with each other through wireless communication that complies with a predetermined wireless communication standard, such as Bluetooth (registered trademark), for example.
[0018] In this example, the robot 11 is a multi-joint robot having three axes 11a to 11c, and a tool 11d is installed at the front end of the robot 11. Each axis 11a to 11c of the robot 11 is driven by a servo motor (not shown), thereby changing the posture of the robot 11. In addition, the robot 11 that cooperates with the operator is not limited to this example, as long as it is a robot having at least one axis driven by a servo motor.
[0019] The input device 2 is an input device that can be carried by the operator, such as a so-called teaching pendant, and is used to input setting information related to the action of the robot 11. To this end, the input device 2 includes, for example: a plurality of operation buttons; a control circuit that generates an operation signal corresponding to the pressed operation button among the plurality of operation buttons; and a wireless communication circuit that generates a wireless signal including the operation signal generated by the control circuit and transmits the wireless signal to the control device 12. In addition, the input device 2 generates an operation signal corresponding to the operation of the operator, and transmits the wireless signal including the operation signal to the control device 12. The operation signal is used to indicate whether the robot 11 is to perform a leading action, the form of the leading action, and the values of various control parameters for controlling the servo motors driving each axis in the leading action of the robot 11.
[0020] The head mounted display 3 is an example of a display unit, and when worn on the operator's head, displays display data for the robot 11 received from the control device 12, which indicates the movement of the robot 11 in the leading movement when the robot 11 performs the leading movement, in an overlapped manner with the operator's field of view. To this end, the head mounted display 3 includes, for example: a transmissive display composed of a liquid crystal display; a wearing component configured so that when the head mounted display 3 is worn on the operator's head, the transmissive display overlaps the operator's field of view; a wireless communication circuit that wirelessly communicates with the control device 12; and a control circuit that causes the transmissive display to display the display data included in the wireless signal received from the control device 12, or transmits an image received from the camera 4 to the control device 12 via the wireless communication circuit.
[0021] The camera 4 is an example of a photographing unit, and is mounted on the head-mounted display 3 so that, for example, the operator's field of view can be photographed when the operator wears the head-mounted display 3. For example, the camera 4 is mounted on the side or above the transmissive display of the head-mounted display 3 so that the camera 4 faces the front direction of the operator wearing the head-mounted display 3. In addition, the camera 4 is connected to the control circuit of the head-mounted display 3 so as to be communicable. During the period when the collaborative action assisting device 1 performs the collaborative action assisting process, the camera 4 photographs the photographing range including the field of view of the operator wearing the head-mounted display 3 in each predetermined photographing cycle to generate an image, and outputs the generated image to the control circuit of the head-mounted display 3.
[0022] The wireless communication interface 5 includes a circuit for executing processing related to the transmission and reception of wireless signals conforming to a predetermined wireless communication standard. In addition, the wireless communication interface 5 extracts information such as operation signals or images contained in the wireless signals received from the input device 2 and the head-mounted display 3, and transmits them to the processor 8. In addition, the wireless communication interface 5 generates a wireless signal including data to be displayed on the head-mounted display 3 obtained from the processor 8, and transmits the generated wireless signal to the head-mounted display 3.
[0023] The communication interface 6 includes, for example, a communication interface for connecting the control device 12 to the communication line 13 and a circuit for executing processing related to the transmission and reception of signals via the communication line 13. In addition, the communication interface 6 outputs, for example, a torque command value for the servo motor of the robot 11 obtained from the processor 8 to the robot 11 via the communication line 13. In addition, the communication interface 6 receives information indicating the operating status of the servo motor, such as the value of the feedback current of the servo motor and the encoder signal from the encoder indicating the rotation amount of each shaft 11a to 11c driven by the servo motor, from the robot 11 via the communication line 13, and transmits the information to the processor 8.
[0024] The memory 7 is an example of a storage unit, and includes, for example, a readable and writable semiconductor memory and a read-only semiconductor memory. In addition, the memory 7 may include a storage medium such as a semiconductor memory card, a hard disk, or an optical storage medium, and a device for accessing the storage medium.
[0025] The memory 7 stores various programs for controlling the robot 11 executed by the processor 8 and computer programs for cooperative action assisting processing. In addition, the memory 7 stores display data indicating the movement of the robot 11 for each form of the leading action displayed on the head mounted display 3 in association with the corresponding form of the leading action. Furthermore, the memory 7 stores information indicating the movement status of the servo motor obtained from the robot 11 during the movement of the robot 11, various data generated during the cooperative action assisting processing, and images obtained from the camera 4.
[0026] The processor 8 is an example of a control unit, and includes, for example, a central processing unit (CPU) and its peripheral circuits. In addition, the processor 8 may also include a processor for numerical calculation. In addition, the processor 8 controls the entire collaborative action assisting device 1. In addition, the processor 8 controls the action of the robot 11 and performs collaborative action assisting processing.
[0027] Figure 2 1 is a functional block diagram of the processor 8 related to the collaborative action assisting process. The processor 8 includes a pilot control unit 21, a display data selection unit 22, a display position setting unit 23, and a display control unit 24. These units of the processor 8 are, for example, functional modules implemented by a computer program executed on the processor 8. Alternatively, these units may be implemented as a dedicated arithmetic circuit installed as part of the processor 8.
[0028] The pilot control unit 21 is an example of an action control unit, and controls the action of the robot 11 according to whether the robot 11 performs a pilot action, the form of the pilot action, or the value of a control parameter specified by an operation signal included in a wireless signal received from the input device 2 via the wireless communication interface 5. Here, the pilot action is described.
[0029] Figure 3A and Figure 3B An example of the form of leading action is shown respectively. Figure 3A In the action form shown, the robot 11 moves in response to the external force applied to the axes 11a to 11c of the joints of the robot 11 so that the axes 11a to 11c of the joints follow. Figure 3A As shown by arrow 301, in response to the operator 300 pressing the front end of the robot 11, the axes 11a to 11c of the joints of the robot 11 rotate as shown by arrows 302 to 304. Figure 3A The action form shown is called axis leading.
[0030] On the other hand, Figure 3B In the action form shown, the robot 11 moves in response to the external force applied to any position of the robot 11 so that the orthogonal coordinate system set about the position follows. Figure 3B As shown, the operator 300 applies force to the tool 11d installed at the front end of the robot 11, thereby rotating one of the axes 11a to 11c of each joint, such as the axis 11c on which the tool 11d is installed, so that the orthogonal coordinate system 311 set for the tool 11d changes in accordance with the direction of the force. Figure 3B The form of action shown is called orthogonal leading.
[0031] In the case of each axis piloting, any one axis of the robot 11 may be fixed. In the case of orthogonal piloting, any one axial motion in an orthogonal coordinate system or any one circumferential motion in an orthogonal coordinate system may be fixed.
[0032] Furthermore, in the leading action, in order to set the action of the robot 11 via the input device 2, the virtual viscosity, elasticity, inertia or rotation speed of each axis 11a~11c of the robot 11 when the robot 11 moves can be set as the control parameters of the servo motor.
[0033] Therefore, the pilot control unit 21 determines the fixed axis according to the form of the pilot action specified by the operation signal and the designation of the fixed axis. In addition, if the operator applies an external force to the robot 11, the pilot control unit 21 rotates the axis other than the fixed axis according to the form of the pilot action specified by the external force. At this time, the pilot control unit 21 determines the torque command value for the servo motor driving the axis according to the value of the virtual viscosity, elasticity or inertia of each axis contained in the operation signal (when the viscosity value, etc. is not included in the operation signal, it is a value set in the past or a default value). At this time, the pilot control unit 21 determines the torque command value by, for example, referring to a table representing the relationship between the combination of the values of virtual viscosity, elasticity or inertia and the torque command value. Such a table is, for example, pre-stored in the memory 7. Alternatively, the pilot control unit 21 can also calculate the torque command value according to a formula representing the relationship between the combination of the values of virtual viscosity, elasticity or inertia and the torque command value. Then, the pilot control unit 21 outputs the torque command value to the robot 11 via the communication interface 6 and the communication line 13, so that the servo motors used to drive the axes other than the fixed axis generate torques corresponding to the values of virtual viscosity, elasticity or inertia. In addition, the servo motors can be controlled by proportional-integral-differential (PID) according to the actual torque generated by the external force applied to the robot 11 by the operator and the torque specified by the torque command value. In addition, the pilot control unit 21 outputs the control value of the servo motor representing the set speed (for example, the duty cycle of pulse width modulation) to the robot 11 via the communication interface 6 and the communication line 13.
[0034] The pilot control unit 21 can obtain the rotation amount of each axis by counting the number of times the encoder signal is received from the encoder installed on each axis or the servo motor driving each axis. The pilot control unit 21 can obtain the posture of the robot 11 at the time point of starting the action and the rotation amount of each axis at the time point of interest.
[0035] The display data selection unit 22 selects display data to be displayed on the head mounted display 3 from the display data of each action form stored in the memory 7, corresponding to the form of the piloting action specified by the operation signal. To this end, the display data selection unit 22 selects the display data corresponding to the form of the piloting action specified by the operation signal, for example, by referring to a table indicating the relationship between the form of the piloting action and the display data. For example, in the piloting of each axis, display data indicating a symbol (e.g., an arc-shaped arrow) indicating the rotation direction of each axis is selected, and on the other hand, in the orthogonal piloting, display data indicating a symbol for indicating an orthogonal coordinate system (e.g., a combination of a straight arrow indicating each axis of the orthogonal coordinate system and an arc-shaped arrow centered on the straight arrow of each axis) is selected. In addition, the display data selection unit 22 reads the selected display data from the memory 7.
[0036] In addition, the display data selection unit 22 may also change the display data in accordance with the value of the control parameter (e.g., viscosity value, elasticity value, inertia value, rotation speed, etc.) included in the operation signal. For example, the larger the value of the set virtual viscosity, elasticity, or inertia, the higher the brightness of the arrow represented by the display data. Alternatively, the display data selection unit 22 may change the color, thickness, length, or transparency of the arrow represented by the display data in accordance with the value of the set virtual viscosity, elasticity, or inertia. In addition, when the control parameters are set to different values for each axis of the robot 11, the display data may be changed for each axis in accordance with the value of the control parameter set for that axis.
[0037] Furthermore, when any one of the axes 11 a to 11 c included in the robot 11 is fixed, the display data selection unit 22 may read the display data indicating the fixed axis from the memory 7 .
[0038] The display data selection unit 22 transfers the selected display data read from the memory 7 to the display control unit 24 .
[0039] The display position setting unit 23 sets the position for displaying the display data selected by the display data selection unit 22 in the display area of the transmissive display of the head mounted display 3. For example, in the case of each axis guiding, it is preferable to display a mark indicating the rotation direction of the axis around the axis of the robot 11 that is not fixed. In addition, in the orthogonal guiding, it is preferable to display a mark indicating the orthogonal coordinate system at the position of the designated orthogonal coordinate system. In this way, it is preferable to display the display data indicating the action of the robot 11 in association with the part of the robot 11 associated with the form of the designated guiding action so that the action position of the robot 11 can be known in the guiding action. Therefore, when the cooperative action assisting process is started, the display position setting unit 23 detects the part of the robot 11 associated with the form of the designated guiding action in the obtained image every time an image is obtained from the camera 4 after the start. For example, when the form of the designated guiding action is each axis guiding, the display position setting unit 23 detects each axis of the robot 11 on the image. In addition, when the form of the designated guiding action is orthogonal guiding, the position of the designated orthogonal coordinate system of the robot 11 is detected, for example, the tool 11d attached to the front end of the robot 11 is detected. To this end, the display position setting unit 23, for example, performs template matching with the image using a template representing each part of the robot 11 that becomes the detection object, thereby detecting the area on the image where the part that becomes the detection object is reflected. In addition, such a template is stored in the memory 7 in advance, for example. At this time, the display position setting unit 23 can use different multiple templates for template matching corresponding to the posture of the robot 11 seen from the operator. In this case, the display position setting unit 23 can use the posture of the part shown by the template that is most consistent with the image among the multiple templates related to the part as the posture of the part seen from the operator. Alternatively, the display position setting unit 23 can input the image to a recognizer that has been learned in advance to detect the part of the robot 11 that becomes the detection object, thereby detecting the area on the image where the part of the robot 11 that becomes the detection object is reflected. The display position setting unit 23 can use a recognizer such as a deep neural network or AdaBoost as the above-mentioned recognizer.
[0040] In the present embodiment, the camera 4 is mounted on the head-mounted display 3 so that the visual field of the operator wearing the head-mounted display 3 is included in the photographing range of the camera 4. Therefore, when the operator sees a predetermined object through the transmissive display of the head-mounted display 3, the position where the predetermined object is shown on the image obtained by the camera 4 corresponds to the position where the predetermined object overlaps on the display area of the transmissive display. Therefore, the display position setting unit 23 can obtain, for each part of the robot 11 that becomes the detection target, the position on the display area corresponding to the center of gravity of the area where the part is shown on the image obtained by the camera 4, as the position of the part on the display area, by referring to the table indicating the correspondence between the position on the image obtained by the camera 4 and the position on the display area of the transmissive display of the head-mounted display 3. For example, such a table is pre-stored in the memory 7, or, instead of such a table, the display position setting unit 23 may input the coordinates of the center of gravity of the area where the part to be detected is projected into a position transformation formula representing the correspondence between the position on the image obtained by the camera 4 and the position on the display area of the transmissive display of the head-mounted display 3, thereby calculating the position of the part on the display area.
[0041] The display position setting unit 23 performs the above-mentioned processing and updates the positions of the parts associated with the designated form of the guiding action on the display area every time an image is obtained from the camera 4. Thus, even if the operator moves or changes the direction of his face during the guiding action, the display position setting unit 23 can make the head mounted display 3 display the display data for representing the robot's action in the form of the guiding action designated for the parts so that the parts associated with the designated form of the guiding action for the robot 11 are in a predetermined positional relationship.
[0042] The display position setting unit 23 notifies the display control unit 24 of the positions of various parts of the robot 11 associated with the designated guiding action form every time an image is obtained from the camera 4 .
[0043] The display control unit 24 causes the head mounted display 3 to display display data indicating an action related to the part in the form of the designated guiding action in such a manner that the parts associated with the robot 11 are in a predetermined positional relationship. For example, the display control unit 24 sets the position of the display data corresponding to the part in the display area in such a manner that the positions of the parts associated with the designated guiding action in the display area are in a predetermined positional relationship. Then, the display control unit 24 transmits a signal including the display data corresponding to the part and its display position to the head mounted display 3 via the wireless communication interface 5 for each part associated with the designated guiding action.
[0044] Figure 4A An example of display data displayed on the head mounted display 3 is shown for each axis guide. Figure 4A In the example shown, in the display area 400 of the head mounted display 3, the axes 11a to 11c of the robot 11 are in the operating positions, so arc-shaped arrows 401 to 403 are displayed for each axis 11a to 11c with the axis as the center to indicate the rotation direction of the axis.
[0045] Furthermore, when any one of the axes 11a to 11c is fixed, the display data indicating the motion of the robot 11 may not be displayed for the fixed axis. For example, when the axis 11a is fixed, the arc-shaped arrow 401 indicating the periphery of the axis 11a may not be displayed. Thus, the operator can intuitively understand the fixed axis and the unfixed axis in the piloting motion.
[0046] Alternatively, the display control unit 24 may also display display data indicating that no action is performed for the fixed axis. For example, when the axis 11a is fixed, the display control unit 24 may display an X-shaped mark at a position on the display area corresponding to the axis 11a. In this case, when the absolute value of the actual torque of the servo motor of the fixed axis obtained via the communication interface 6 is greater than a predetermined value indicating that an external force is applied, the display control unit 24 may display display data indicating that no action is performed. Thus, the display control unit 24 suppresses the number of display data overlapping with the operator's field of view so that the operator does not feel complicated, and when the operator moves the robot 11, the operator can intuitively grasp the fixed axis.
[0047] Figure 4B Regarding the orthogonal guide, an example of display data displayed on the head mounted display 3 is shown. Figure 4B In the example shown, in the display area 400 of the head-mounted display 3, an orthogonal coordinate system is set at the position of the tool 11d mounted on the robot 11, and therefore, an orthogonal coordinate system 410 and arc-shaped arrows 411 to 413 are displayed around the axes of the orthogonal coordinate system with the position on the display area corresponding to the tool 11d as the origin.
[0048] In this example, the display control unit 24 may not display the arrow indicating the axis of the fixed orthogonal coordinate system. Alternatively, the display control unit 24 may make the brightness of the fixed orthogonal coordinate system axis lower than the brightness of the unfixed orthogonal coordinate system axis. Alternatively, the display control unit 24 may make the transparency of the fixed orthogonal coordinate system axis higher than the transparency of the unfixed orthogonal coordinate system axis.
[0049] Figure 4CThe following is an example of display data displayed on the head mounted display 3 when the values of control parameters of the servo motors such as the virtual viscosity, elasticity, or inertia of each axis are different in the driving of each axis. Figure 4C In the example of , in the display area 400 of the head mounted display 3, each axis 11a to 11c of the robot 11 becomes an action position, so for each axis 11a to 11c, an arc-shaped arrow 421 to 423 for indicating the rotation direction of the axis is displayed with the axis as the center. In addition, in this example, the thickness of the arrow is changed according to the value of the set virtual viscosity, elasticity or inertia. For example, when the viscosity value set for the axis 11b is larger than the viscosity value set for the axis 11a and the viscosity value set for the axis 11c, the arrow 422 displayed around the axis 11b is displayed thicker than the arrow 421 displayed around the axis 11a and the arrow 423 displayed around the axis 11c. As a result, the operator can intuitively grasp that the rotation action of the axis 11b is softer than the rotation action of the axis 11a and the axis 11c. In this way, by changing the displayed display data according to the value of the control parameter of the servo motor set for each axis, the operator can intuitively grasp the set control parameter.
[0050] Figure 5 2 is an operation flow chart of the collaborative action support process. The processor 8 executes the collaborative action support process according to the following operation flow every time it receives an operation signal from the input device 2, for example.
[0051] The guiding control unit 21 controls the guiding action of the robot 11 according to the operation signal from the input device 2 (step S101). In addition, the display data selection unit 22 selects display data to be displayed on the head mounted display 3 from the display data stored in the memory 7 according to the form of the guiding action specified by the operation signal (step S102).
[0052] The display position setting unit 23 detects the part of the robot 11 associated with the form of the designated guiding action in the image obtained from the camera 4 (step S103). Then, the display position setting unit 23 determines the position of the part in the display area of the head mounted display 3 based on the position of the detected part of the robot 11 on the image (step S104).
[0053] The display control unit 24 causes the head mounted display 3 to display the selected display data with respect to the portion of the robot 11 associated with the designated leading action form so that the position of the portion on the display area of the head mounted display 3 has a predetermined positional relationship (step S105 ).
[0054] After step S105, the processor 8 determines whether an operation signal indicating the end of the leading action is received from the input device 2 (step S106). If the operation signal indicating the end of the leading action is not received (step S106: No), the processor 8 repeats the processing of step S103 and thereafter. On the other hand, if the operation signal indicating the end of the leading action is received (step S106: No), the processor 8 ends the collaborative action auxiliary processing.
[0055] As described above, when the robot is made to perform a leading action, the collaborative action assisting device associates the robot portion associated with the form of the designated leading action, and causes the head mounted display worn by the operator to display display data for indicating the robot's action in the form of the designated leading action. Therefore, the collaborative action assisting device can easily enable the operator to intuitively grasp the settings related to the action of the robot performing the leading action.
[0056] According to a modified example, the display control unit 24 may use, as display data, an image of a portion of the robot 11 associated with the form of the designated leading action, projected from the image obtained from the camera 4, such as a contour image of the portion. In this case, the display control unit 24 clips the following area from the image, the area projecting the portion associated with the form of the designated leading action detected by the display position setting unit 23. The display control unit 24 performs edge detection processing using an edge detection filter such as a Sobel filter on the clipped area, and detects edge pixels having an edge strength greater than a predetermined value. Then, the display control unit 24 detects the leftmost edge pixel and the rightmost edge pixel in each pixel string in the horizontal direction in the clipped area as contour pixels for representing the contour of the portion associated with the form of the designated leading action. Then, the display control unit 24 generates a contour image in which each contour pixel has a pixel value different from the values of other pixels as display data.
[0057] In this case, the display control unit 24 may change the size of the contour image in accordance with the value of the control parameter of the servo motor of each axis of the robot 11. For example, the display control unit 24 may enlarge the contour image in such a manner that the contour image becomes larger as the virtual viscosity value or elasticity value becomes larger. Thus, the operator can visually know the value of the control parameter set for the portion of the robot 11 associated with the form of the designated guiding action through the size of the portion, and can intuitively grasp the set control parameter.
[0058] According to another modified example, the pilot control unit 21 of the processor 8 can change the value of the control parameter set for the portion associated with the display data by causing the operator to point with a finger at a position overlapping with the display data displayed on the head mounted display 3 for a predetermined period of time (for example, 1 to 2 seconds). Figure 4A The tip of the arrow 411 shown in FIG. 1 is turned rightward, thereby increasing the virtual viscosity value of the shaft 11a by a predetermined value. On the other hand, the operator indicates with a finger during a predetermined period of time. Figure 4A The front end of the arrow 411 shown turns left, thereby reducing the virtual viscosity value of the shaft 11a by a predetermined value.
[0059] For this purpose, the display data is set to indicate a range (for example, Figure 4A The leading control unit 21 detects the operator's finger in the image obtained by the camera 4. For example, the leading control unit 21 detects the finger by inputting the image to a pre-learned recognizer, thereby detecting the area where the finger is reflected on the image, and taking the center of gravity of the area as the position of the finger. The leading control unit 21 can use a recognizer such as a deep neural network or AdaBoost as the above-mentioned recognizer. In addition, as described with respect to the display position setting unit 23, the position on the image obtained by the camera 4 corresponds one-to-one to the position on the display area of the head-mounted display 3. Therefore, the leading control unit 21 performs the same processing as the display position setting unit 23 to obtain the position on the display area corresponding to the position of the finger detected on the image. If the position on the display area is included in the range of changing the control parameter, the value of the control parameter is changed by the amount set for the range.
[0060] According to this modification, the operator can change the control parameter without operating the input device 2, so the operation efficiency is improved.
[0061] According to another modification, the display control unit 24 may cause the head mounted display 3 to display the movement trajectory of the part of the robot 11 associated with the designated form of the guiding action. In this case, the display control unit 24 acquires the position of the part of the robot 11 associated with the designated form of the guiding action on the display area from the display position setting unit 23 each time an image is acquired from the camera 4 while the rotation amount of the axis calculated based on the coded signal related to any axis of the robot 11 acquired from the robot 11 via the communication interface 6 is changing, that is, while the posture of the robot 11 is changing. Then, the display control unit 24 uses a line connecting the acquired positions in a time series order as a movement trajectory, and causes the head mounted display 3 to display the movement trajectory.
[0062] Furthermore, a sensor such as a gyro sensor that can measure the amount of change in the orientation of the operator's face may be installed in the head mounted display 3 so that even if the operator changes the orientation of his face during a series of operations, the correct movement trajectory is displayed. In this case, the head mounted display 3 measures the amount of change in the orientation of the operator's face by the sensor at each predetermined cycle, and transmits a wireless signal containing the amount to the control device 12. The display control unit 24 calculates the displacement amount of the orientation of the face from the last image acquisition to the latest image acquisition based on the amount of change in the orientation of the operator's face included in the wireless signal received via the wireless communication interface 5 each time an image is acquired from the camera 4. For example, if the amount of change in the orientation of the operator's face is an angular acceleration, the display control unit 24 can calculate the displacement amount of the orientation of the operator's face by performing a second-order integration of the angular acceleration. Then, the display control unit 24 moves the position of the part of the robot 11 associated with the form of the designated guiding action on the display area when each image is acquired by an amount corresponding to the displacement amount of the orientation of the operator's face, with the time when the latest image is acquired as a reference.
[0063] According to another modification, the camera 4 may be installed, for example, facing downward on the ceiling of the room where the robot 11 is installed, so that the relative position of the operator with respect to the robot 11 can be detected. In addition, a sensor such as a gyro sensor may be installed on the head mounted display 3 so that the amount of change in the orientation of the operator's face can be measured. In addition, the position of the robot 11 in the real space is stored in advance in the memory 7. In this case, for example, when the operator operates the input device 2 to input the robot 11 to start the guiding operation, the operator looks at a predetermined position of the robot 11 (for example, the tool 11d installed on the robot 11) for a certain period of time (for example, 3 to 5 seconds) for initialization. On the other hand, when the guiding control unit 21 of the processor 8 receives an operation signal indicating the start of the guiding operation of the robot 11 from the input device 2, it controls the servo motor so that the robot 11 becomes a predetermined posture set in advance. In addition, when the display position setting unit 23 of the processor 8 receives an operation signal indicating the start of the guiding operation of the robot 11 from the input device 2, it detects the operator in the image captured by the camera 4 during the certain period of time. For example, the display position setting unit 23 detects the center of gravity of the area where the operator appears on the image by inputting an image to a pre-learned recognizer to detect the operator. The display position setting unit 23 can use a recognizer such as a deep neural network or AdaBoost as the above-mentioned recognizer. The position on the image corresponds one-to-one to the direction relative to the camera 4. In addition, the distance from the camera 4 to the ground where the robot 11 and the operator are located is also known, so the display position setting unit 23 can determine the position of the operator in the real space (hereinafter referred to as the initial position) based on the position of the center of gravity of the area where the operator appears on the image. Therefore, the display position setting unit 23 can determine the relative positional relationship between the robot 11 and the operator. In addition, the posture of the robot 11 during the period when the operator is looking at the predetermined position of the robot 11 is also known, so the line of sight direction from the operator to the predetermined position of the robot 11 when the operator is at the initial position is also known. Therefore, the display position setting unit 23 sets the intersection of the line of sight direction and the display area of the head-mounted display as the reference point.
[0064] After initialization, the head mounted display 3 measures the amount of change in the orientation of the operator's face through the sensor at each predetermined cycle, and transmits a wireless signal containing the amount to the control device 12. The display position setting unit 23 of the processor 8 calculates the displacement of the orientation of the face from the time of looking at the reference position based on the amount of change in the orientation of the operator's face contained in the wireless signal received via the wireless communication interface 5. For example, if the amount of change in the orientation of the operator's face is angular acceleration, the display position setting unit 23 can calculate the displacement of the orientation of the operator's face by performing a second-order integration of the angular acceleration. In addition, the display position setting unit 23 can detect the movement direction and movement amount of the operator from the initial position by detecting the position of the operator from the image obtained by the camera 4 in the same way as described above. In addition, the display position setting unit 23 can obtain the change of the robot 11 relative to the predetermined posture based on the rotation amount of each axis. Therefore, the display position setting unit 23 can calculate the angle of the direction from the operator to each action position of the robot 11 relative to the front direction of the operator's face based on the operator's moving direction and moving amount from the initial position, the displacement of the operator's face orientation relative to the operator's face orientation when the operator is looking at the reference position of the robot 11 at the initial position, and the posture change amount of the robot 11 relative to the predetermined posture. Then, the display position setting unit 23 can determine the position of each action position on the display area of the head mounted display 3 based on the angle.
[0065] Furthermore, according to other modified examples, the display data selection unit 22 may change the selected display data in accordance with the action state of the robot 11 caused by the external force applied to the robot 11 by the operator. For example, the display data selection unit 22 may change the color or brightness of the arrow shown in the display data related to the axis in accordance with the actual torque of the servo motor of each axis 11a to 11c of the robot 11 received from the robot 11 via the communication interface 6. Alternatively, the display data selection unit 22 may calculate the remaining rotation amount that can be rotated for each of the axes 11a to 11c of the robot 11 based on the rotation amount of the axis and the rotation range of the axis calculated based on the encoder signal of the axis received from the robot 11 via the communication interface 6, and the length of the arrow shown in the display data is shortened as the remaining rotation amount is smaller. Alternatively, the display data selection unit 22 may flash the arrow of the display data related to the axis for which the absolute value of the actual torque of the servo motor received from the robot 11 via the communication interface 6 is greater than a predetermined value. This allows the operator to more easily understand the operating state of the robot 11 .
[0066] Furthermore, according to other variations, the input device 2 may include a voice input unit such as a microphone, so that the operator can change the value of the control parameter by voice. In addition, the input device 2 sends a wireless signal including a voice signal to the control device 12, and the voice signal represents the voice of the operator obtained by the voice input unit. Alternatively, the voice input unit may be installed on the head-mounted display 3, and a wireless signal including a voice signal may be sent from the head-mounted display 3 to the control device 12, and the voice signal represents the voice of the operator obtained by the voice input unit. The pilot control unit 21 of the processor 8 of the control device 12 may perform a predetermined voice recognition process such as a word recognition method using a hidden Markov model on the voice signal included in the wireless signal received via the wireless communication interface 5, thereby recognizing the voice of the operator, and changing the value of the control parameter or changing the form of the designated pilot action according to the recognition result. As a result, the trouble of the operator in operating the input device 2 can be reduced, and thus the operation efficiency can be improved.
[0067] All examples and specific terms listed here are examples and terms for the purpose of teaching to help readers understand the concepts given by the inventors to promote the present invention and the technology, and should be interpreted as not limited to the structure of any example of this specification related to the advantages and disadvantages of the present invention, such specific examples and conditions. The embodiments of the present invention are described in detail, but it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.
Claims
1. A collaborative action assisting device, It is characterized in that have: a storage unit storing display data for representing the movement of the robot in each form when the robot having a plurality of axes rotatable therefrom performs a cooperative movement with an operator; a display unit, which is worn by the operator and the operator can visually recognize the robot via a display area of the display unit; an imaging unit configured to capture an image of the robot or the operator and generate an image representing the robot or the operator; a display position setting unit for detecting, based on the image, a position of each of the plurality of axes of the robot associated with a form of movement of the robot specified via an input device, on the display area when the operator visually recognizes the robot via the display area; A display data selection unit for selecting, in accordance with a form of movement of the robot specified via an input device, display data corresponding to the specified form from among the display data stored in the storage unit; A display control unit displays the selected display data in the display area of the display unit in a manner that has a predetermined positional relationship with the position of each axis of the robot on the display area, and makes the display data different for the axis that rotates and the axis that is fixed in the form of a specified movement of the robot among the multiple axes.
2. The collaborative action assisting device according to claim 1, It is characterized in that When the specified action form of the robot is an action form in which one of the plurality of axes rotates in response to the external force applied to the robot, the display data is a symbol indicating the rotation of the rotating axis. The display position setting unit detects the position of the axis of rotation among the plurality of axes on the display area, The display control unit displays a mark indicating the rotation of the axis around the axis to be rotated among the plurality of axes on the display area.
3. The collaborative action assisting device according to claim 1, It is characterized in that When the specified action of the robot is an action in which the position or orientation of a coordinate system set on one of the plurality of axes of the robot changes in response to an external force applied to the robot, the display data is a symbol representing the coordinate system. The display position setting unit detects the position of the axis for setting the coordinate system on the display area. The display control unit displays a symbol indicating the coordinate system in the display area so as to overlap with a position of an axis for setting the coordinate system.
4. The collaborative action assisting device according to claim 1, It is characterized in that The plurality of axes of the robot are driven by servo motors. The display data selection unit changes the selected display data in accordance with a value of a control parameter for controlling the servomotor set via the input device.
5. The collaborative action assisting device according to any one of claims 1 to 4, It is characterized in that The imaging unit is mounted on the display unit in a manner capable of capturing the visual field of the operator. The display position setting unit detects an area on the image that represents each of the multiple axes of the robot, and detects the position of each of the multiple axes of the robot on the display area according to the positional relationship between the position of the area on the image and the corresponding position on the display area.
6. The collaborative action assisting device according to any one of claims 1 to 4, It is characterized in that The display data selection unit changes the selected display data in accordance with an operation state of the robot caused by an external force applied to the robot.
7. The collaborative action assisting device according to any one of claims 1 to 3, It is characterized in that The plurality of axes of the robot are driven by servo motors. The imaging unit is mounted on the display unit in a manner capable of capturing the visual field of the operator. The collaborative action assisting device also includes a motion control unit, which detects the operator's finger on the image, and when the position of the operator's finger on the image corresponding to the position on the display area overlaps with the display data displayed in the display area, changes the value of a control parameter associated with the display data for controlling the servo motor.
8. The collaborative action assisting device according to any one of claims 1 to 3, It is characterized in that The display control unit displays, in the display area, a movement trajectory of a portion of the robot associated with the designated form of the movement of the robot.
9. The collaborative action assisting device according to any one of claims 1 to 3, It is characterized in that The plurality of axes of the robot are driven by servo motors. The above-mentioned collaborative action assisting device further comprises: a voice input unit that generates a voice signal representing the voice of the operator; and An action control unit recognizes the operator's voice included in the voice signal, and changes the value of a control parameter for controlling the servomotor or changes the form of the action of the robot according to the recognized operator's voice.
