Motion teaching system and motion teaching method

Abstract

This invention provides a motion teaching system that allows for easy instruction of robot arm movements. [Solution] A motion teaching system for teaching the movements of a robot arm, comprising a robot hand that can be connected to and detached from the robot arm, the robot hand comprising an action part that acts on a workpiece, and a camera part whose position relative to the action part is fixed and which captures images, the camera part captures a first teaching image while the robot hand is moved after being detached from the robot arm and grasped by a person, and stores an image based on the first teaching image in a storage unit.

Description

[Technical Field]

[0001] This disclosure relates to a motion teaching system and a motion teaching method for teaching the movements of a robot arm. [Background technology]

[0002] Traditionally, one method of teaching robots is direct teaching. In direct teaching, teaching is performed by directly moving the robot's arm with a human hand. As a conventional technology related to direct teaching, a direct teaching mode switching method that automatically switches between direct teaching mode and normal mode is known (see Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-254361 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] This disclosure provides a motion teaching system and a motion teaching method that can easily teach the movements of a robot arm. [Means for solving the problem]

[0005] One aspect of the present disclosure is a motion teaching system for teaching the operation of a robot arm, comprising a robot hand that is connectable to and detachable from the robot arm, wherein the robot hand comprises an action part that acts on a workpiece, and a camera part fixed in position relative to the action part and capturing images, wherein the camera part captures a first teaching image while the robot hand is moved after being detached from the robot arm and held by a person, and stores an image based on the first teaching image in a storage unit.

[0006] One aspect of the present disclosure is an operation teaching method for teaching the operation of a robot arm to which a robot hand can be connected. The robot hand includes an acting part that acts on a workpiece, and a camera part that is fixed in position with respect to the acting part and captures an image. The operation teaching method includes capturing a first teaching image during the movement of the robot hand held by a person, and storing an image based on the first teaching image in a storage unit.

Effects of the Invention

[0007] According to the present disclosure, the operation of the robot arm can be easily taught.

Brief Description of the Drawings

[0008] [Figure 1] Block diagram showing a configuration example of an operation teaching system in the first embodiment [Figure 2] External perspective view showing an example of the configuration and installation environment of a robot device [Figure 3] External perspective view showing a configuration example of a hand part [Figure 4A] Diagram for explaining the operation of the hand part during teaching processing [Figure 4B] Diagram for explaining the operation of the hand part during playback processing [Figure 5] Diagram for explaining the operation control of an arm part [Figure 6] Flowchart showing an operation example of an operation teaching system during teaching processing [Figure 7] Flowchart showing an operation example of an operation teaching system during playback processing [Figure 8] Diagram for explaining a method of deriving certainty using a learned model [Figure 9] Examples of an image with features and an image without features [Figure 10] Flowchart showing an operation example of an operation teaching system during thinning processing [Figure 11A] Diagram for supplementing thinning processing (Part 1) [Figure 11B]Diagram to supplement the thinning process (Part 2) [Figure 12A] This figure shows the result of playback processing without masking. [Figure 12B] This figure shows the result of playback processing when masking is applied. [Figure 13] A flowchart illustrating an example of operation during mask processing in the motion teaching system. [Figure 14A] Figure showing an example of a segmented image display. [Figure 14B] This figure shows an example of a motion training image with a masked area. [Figure 15] Diagram showing that the workpiece WK is outside the camera's field of view. [Figure 16] A diagram showing that the working part of the hand unit is located within the imaging range of the camera unit. [Figure 17] Flowchart showing an example of the operation when processing a work absence notification. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below, with reference to the drawings as appropriate. However, unnecessary details may be omitted. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding by those skilled in the art. The accompanying drawings and the following explanation are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0010] (The circumstances leading to the acquisition of the embodiments of this disclosure) When introducing a robotic device, it is necessary to perform teaching work to instruct the robot arm on how to move. Patent Document 1 describes direct teaching as being performed.

[0011] In Patent Document 1, the movement of the robot arm is recorded by direct teaching, which requires a person to grasp and move the robot arm while it is in motion. Therefore, a person needs to apply a certain amount of force to move the robot arm, making it difficult to perform precise positioning.

[0012] Another possible method for teaching the robot arm's movements is for a person to operate a dedicated terminal to record the robot hand's position (coordinates) and angle in three-dimensional space. In this case, it may be necessary to specify angles such as roll, pitch, and yaw, which may be difficult for beginners to understand intuitively.

[0013] In the following embodiments, a motion teaching system and motion teaching method that can easily teach the movements of a robot arm will be described.

[0014] (First Embodiment) <Configuration of the motion teaching system> Figure 1 shows an example configuration of the motion teaching system 5 in the first embodiment. The motion teaching system 5 comprises a robot device 10 and an information processing device 40. The robot device 10 and the information processing device 40 may be connected in a communicative manner and communicate and acquire data and information from each other. Alternatively, the robot device 10 and the information processing device 40 do not have to be connected in a communicative manner. In this case, the information processing device 40 may acquire information held by the robot device 10, for example via a storage unit, and perform various processes based on the acquired information.

[0015] The robot device 10 is a picking robot that picks up (grabs) workpieces (objects). The robot device 10 includes an arm section 20 and a hand section 30. The hand section 30 can be connected to (attached to, mounted on) and detached from (removed from) the arm section 20.

[0016] The arm section 20 includes a connection section 210, a movable section 220, an motion control section 230, a playback section 240, a storage section 250, and a communication section 260. The arm section 20 corresponds to a robot arm.

[0017] The connecting part 210 has a connecting mechanism that can be connected to the connecting part 310 of the hand part 30, and is freely connectable to the connecting part 310 of the hand part 30. The movable part 220 is a movable part of the arm part 20, such as the base and each joint part.

[0018] The playback unit 240 receives an operation start instruction to begin the operation of the arm unit 20 while the hand unit 30 is connected to the arm unit 20. In response to the operation start instruction, the playback unit 240 plays back an operation teaching image that teaches the operation of the arm unit 20. The operation teaching image is captured by the hand unit 30 after it has been detached from the arm unit 20. The playback unit 240 also instructs the camera unit 330 of the hand unit 30 to begin capturing an image (actual operation image) of the hand unit 30 while it is connected to the arm unit 20.

[0019] Furthermore, the process of capturing motion instruction images by the hand unit 30 is also referred to as the teaching process, and the process of playing back motion instruction images by the arm unit 20 is also referred to as the playback process. Therefore, the capture of motion instruction images occurs during the teaching process, and the capture of actual motion images occurs during the playback process.

[0020] The motion control unit 230 controls the movement of the movable part 220. In this case, the motion control unit 230 controls the movable part 220 so that the actual motion image matches the motion teaching image. In other words, the motion control unit 230 controls the movement of the movable part 220 of the arm unit 20 so that the camera unit 330 captures the same multiple images as these images while the hand unit 30 is connected to the arm unit 20, based on a series of consecutive images included in the motion teaching image. This ensures that the images obtained during teaching and playback are the same, causing the hand unit 30 to move in the same way. As a result, the relative positional relationship between the hand unit 30 gripping the workpiece WK and the workpiece WK is reproduced by the images in the actual motion image following the images in the motion teaching image.

[0021] The motion control unit 230 and the regeneration unit 240 may each be implemented as functions of a processor. The processor may be configured using, for example, a Central Processing Unit (CPU), a Digital Signal Processor (DSP), or a Graphics Processing Unit (GPU). The processor may also be configured using various integrated circuits (for example, a Large Scale Integration (LSI) or a Field Programmable Gate Array (FPGA)). The processor implements various functions by executing programs held in the memory unit 250. The processor may comprehensively control each part of the robot device 10 and perform various processes, and may perform processes other than those performed by the motion control unit 230 and the regeneration unit 240.

[0022] The storage unit 250 includes, for example, Random Access Memory (RAM) or Read Only Memory (ROM). The storage unit 250 may also include, for example, a Hard Disk Drive (HDD), Solid State Drive (SSD), optical disc, SD card, etc. The storage unit 250 may also be an external storage medium and may be detachable from the arm unit 20. The storage unit 250 stores various data, information, or programs.

[0023] The memory unit 250 also stores images. These images may include action instruction images. Action instruction images include videos, still images, images consisting of a series of still images, etc. These images may also include action record images. Action record images include videos, still images, images consisting of a series of still images, etc.

[0024] The communication unit 260 communicates various data or information according to a wired or wireless communication method. The communication method used by the communication unit 260 may include a Local Area Network (LAN), Wide Area Network (WAN), mobile phone network, or power line communication. The communication unit 260 may be connected to the hand unit 30 in a communication manner and may also be connected to the information processing device 40 or other external devices in a communication manner.

[0025] The hand unit 30 includes a connecting unit 310, an operating unit 320, a camera unit 330, a notification unit 340, and a communication unit 350. The hand unit 30 corresponds to a robot arm.

[0026] The connecting portion 310 has a connecting mechanism that can be connected to the connecting portion 210 of the arm portion 20, and is freely connectable to the connecting portion 210. The operating portion 320 is the part that acts on the workpiece WK to be gripped. The action on the workpiece WK may include, for example, gripping the workpiece WK, contacting the workpiece WK, pushing the workpiece WK, pulling the workpiece WK, or other actions.

[0027] The camera unit 330 has an image sensor and acquires images by capturing images of a subject. These images include both video and still images. The camera unit 330 is fixed in relative position to the action unit 320. The camera unit 330 may capture images of the hand unit 30 in motion while it is being held by a person as motion teaching images, and may store these motion teaching images in the storage unit 250 of the arm unit 20 via the communication unit 350.

[0028] For example, when a person grasps and moves the hand unit 30, the hand unit 30 approaches (approaches) the workpiece WK and grasps it is captured as a motion teaching image. The motion teaching image serves as a model for when the hand unit 30 is connected to the arm unit 20 after the motion teaching image is captured, and the arm unit 20 moves the hand unit 30. Therefore, the arm unit 20 can operate optimally by operating based on the motion teaching image.

[0029] The camera unit 330 can detect whether the connection unit 310 and the connection unit 210 are connected. When the camera unit 330 detects that the connection unit 310 and the connection unit 210 have been disconnected (i.e., removed), it may start capturing motion teaching images. When the camera unit 330 detects that the connection unit 310 and the connection unit 210 have been connected (i.e., attached), it may stop capturing motion teaching images. The camera unit 330 may also start and stop capturing motion teaching images regardless of the connection state of the connection unit 310 and the connection unit 210. For example, if the hand unit 30 is equipped with a recording button, the camera unit 330 may start and stop recording in response to the pressing of the recording button. In this case, the camera unit 330 may switch between starting and stopping recording each time the recording button is pressed.

[0030] The image captured by the camera unit 330 may be recognized (detected) by image recognition, allowing for the recognition of the workpiece WK and the background other than the workpiece WK. Furthermore, the distance from the camera unit 330 to the workpiece WK captured in the image may be recognized (detected) by image recognition. This image recognition may be performed by one of the illustrated or not illustrated control or processing units within the motion teaching system 5.

[0031] The notification unit 340 notifies various data and information. The notification unit 340 may, for example, display information, output information as sound, vibrate according to the information, transmit information to an external device, or notify information by other means. Information may be displayed on a display, or it may be displayed by lighting or flashing a Light Emitting Diode (LED), etc. The notification unit 340 may notify information when a predetermined event occurs. For example, the notification unit 340 may notify warning information if there are few features in the image within the motion teaching operation, making it difficult to track the object in the image. For example, the notification unit 340 may notify warning information if the workpiece is not visible in at least one image within the motion teaching image.

[0032] The communication unit 350 communicates various data or information according to a wired or wireless communication method. The communication method used by the communication unit 350 may include LAN, WAN, mobile phone network, or power line communication. The communication unit 350 may be connected to the arm unit 20 in a communicative manner and may also be connected to the information processing device 40 or other external devices in a communicative manner.

[0033] The information processing device 40 performs various processes based on the operation instruction image. The information processing device 40 may be, for example, a personal computer (PC), a mobile terminal, a tablet terminal, a server device, etc. The information processing device 40 includes a processing unit 410, a storage unit 420, a communication unit 430, an input unit 440, and a display unit 450.

[0034] The processing unit 410 may be implemented as a function of a processor. The processor may be configured using, for example, a CPU, DSP, or GPU. The processor may also be configured using various integrated circuits (e.g., LSI or FPGA). The processor implements various functions by executing a program held in the memory unit 420. The processing unit 410 may comprehensively control each part of the information processing device 40 and perform various processes.

[0035] The memory unit 420 includes RAM or ROM, and may include an HDD, SSD, optical disc, or SD card. The memory unit 420 may also be an external storage medium and may be detachable from the information processing device 40. The memory unit 420 stores various data, information, programs, or learning models.

[0036] The communication unit 430 communicates various data or information according to a wired or wireless communication method. The communication method used by the communication unit 430 may include LAN, WAN, mobile phone network, or power line communication.

[0037] The input unit 440 may include various buttons, keys, keyboards, touch panels, microphones, sensors, or other input devices. The input unit 440 accepts input of various data or information. The input unit 440 is operated by a user. The user is, for example, a worker HM or administrator who performs teaching, playback, or processing motion instruction images.

[0038] The display unit 450 is, for example, a liquid crystal display or an organic EL display. The display unit 450 displays various data or information. The display by the display unit 450 may be confirmed by, for example, the user.

[0039] Figure 2 is an external perspective view showing an example of the configuration and installation environment of the robot device 10. In the diagram, where arrows indicating direction are shown, the X-axis represents the front-to-back direction, the Y-axis represents the left-to-right direction, and the Z-axis represents the up-and-down direction. The X and Y axes are orthogonal and are contained in the horizontal plane. The Z-axis is contained in the vertical plane.

[0040] The robot device 10, for example, picks (grabs) various workpieces WK stored in the storage container 50A, moves them to the output container 50B, and places (stores) them. The arm 20 moves the hand 30 to any position in three-dimensional space. The hand 30 grips the workpiece WK with the working part 320 (see Figure 3). The robot device 10 may be any type of picking robot, such as an orthogonal suction robot, an orthogonal two-finger robot, an articulated robot, or any other type of picking robot. Although the robot device 10 shown in Figure 2 is given as an example, the configuration of the robot device 10 is not limited to this.

[0041] The robotic device 10 is a manipulator having a multi-axis (e.g., 6-axis) arm section 20, and a base section 21 is installed in a workpiece sorting area or production line. The base section 21 adjusts the height at which the arm section 20 can operate. The arm section 20 includes, for example, a swivel section 23 attached to the upper part of the base section 21 that rotates horizontally, an upper arm section 25 attached to the swivel section 23 that swings in the front-back direction, a forearm section 27 attached to the upper arm section 25 that swings in the up-down direction, and a wrist section 29 attached to the forearm section 27. The wrist section 29 has 3 degrees of freedom. The base section 21 may be included in the arm section 20.

[0042] A hand portion 30 is attached (connected) below the wrist portion 29, that is, below the arm portion 20. The type of hand portion 30 is arbitrary and may be, for example, a suction type, a two-fingered type, a multi-fingered type, or any other type of hand.

[0043] The arm section 20 holds the hand section 30 so that it can move freely in three mutually orthogonal directions. The robot device 10 also includes a motor and other components that supply the driving force to operate the arm section 20 and the hand section 30.

[0044] Figure 3 is an external perspective view showing an example of the configuration of the hand unit 30.

[0045] The hand unit 30 comprises a connecting part 310, an operating part 320, a camera part 330, and a gripping part 360. The connecting part 310 is a component for connecting the hand unit 30 to the arm unit 20, and may be fixed to the arm unit 20 using, for example, screws. The operating part 320 has a teaching start button 321, a hook part 322, a scraper part 323, and a tool mounting hole 324. Some components of the operating part 320 may be omitted.

[0046] The teaching start button 321 can be pressed by an operator or other person to detach the hand unit 30 from the arm unit 20. When the camera unit 330 detects that the hand unit 30 has been detached from the arm unit 20, it starts capturing motion teaching images.

[0047] The hook portion 322 can hook and grip a ring-shaped workpiece. The scraper portion 323 can scoop up and grip the workpiece WK. The tool mounting hole 324 allows various tools (e.g., a stamp) to be attached according to the work.

[0048] The gripping portion 360 is the part of the hand portion 30 that is gripped, for example, by an operator. For example, during the teaching process, the gripping portion 360 is gripped by an operator when the hand portion 30 is detached from the arm portion 20, and the hand portion 30 is moved by the operator. Note that other parts of the hand portion 30 besides the gripping portion 360 may also be gripped by the operator HM.

[0049] Note that the configuration of the hand unit 30 is not limited to the configuration shown in Figure 3, and other configurations are also possible.

[0050] Figure 4A is a diagram illustrating the operation of the hand unit 30 during the teaching process.

[0051] In Figure 4A, the connection part 310 of the hand unit 30 has been detached from the connection part 210 of the arm unit 20 of the robot device 10, and the operator HM is holding the gripping part 360 of the hand unit 30. The operator HM then operates the hand unit 30 to move it, for example, by hooking and gripping the workpiece WK with the hook part 322, or by scooping up and gripping the workpiece WK with the scraper part 323. In Figure 4A, the connection part 310 has been detached from the connection part 210 of the arm unit 20, and the camera unit 330 is capturing motion teaching images. The capture of motion teaching images continues, for example, until the connection part 310 is connected to the connection part 210 of the arm unit 20. Therefore, the hand unit 30 is capable of capturing motion teaching images that capture the movement of the hand unit 30 when it grips the workpiece WK. When the hand unit 30 is connected to the arm unit 20, the position and orientation of the hand unit 30 correspond to the position and orientation of the arm unit 20. Therefore, the motion teaching images become images for teaching the movements of the arm unit 20. The motion teaching images are stored, for example, in the memory unit 250 of the arm unit 20.

[0052] Figure 4B is a diagram illustrating the operation of the hand unit 30 during playback processing.

[0053] In Figure 4B, the connection part 210 of the arm part 20 of the robot device 10 is connected to the connection part 310 of the hand part 30. When the playback unit 240 receives an operation start instruction, it starts playback of the operation teaching image and causes the camera unit 330 to start capturing the actual operation image. The motion control unit 230 controls the position and orientation of the hand part 30, that is, the position and orientation of the arm part 20, so that the actual operation image matches the operation teaching image. In this case, the motion control unit 230 controls the operation of the arm part 20 based on, for example, a Visual Servo. The motion control unit 230 may control the operation of the arm part 20 by comparing the actual operation image and the operation teaching image frame by frame. The motion control unit 230 may control the operation of the arm part 20 based on an Image-Based Visual Servo (IBVS) so that each pixel in the images being compared matches each other.

[0054] The motion control of the arm portion 20 based on motion teaching images and a visual servo may be carried out, for example, using the technology described in Reference Non-Patent Document 1 below.

[0055] (Reference non-patent document 1) Aseem Saxena, Harit Pandya, Gourav Kumar, Ayush Gaud and K. Madhava Krishna, “Exploring Convolutional Networks for End-to-End Visual Servoing”, 2017 IEEE International Conference on Robotics and Automation (ICRA)

[0056] For example, the motion control unit 230 inputs two images as example data and the relative positions of the two images as correct data into a learning model for the visual servo, trains the learning model using deep learning, generates a trained model for the visual servo, and stores it in, for example, the memory unit 250. The motion control unit 230 may derive the relative position based on two consecutive images in the motion training image according to the trained model for the visual servo, and control the movement of the arm unit 20 so that this relative position is obtained.

[0057] Figure 5 is a diagram illustrating the motion control of the arm section 20.

[0058] The robot device 10 controls the movement of the arm 20 based on motion teaching images captured during the teaching process and motion performance images captured during the playback process. The motion teaching images include multiple images in a time series, including the first image, the second image, ... The motion performance images, like the motion teaching images, may include multiple images in a time series, and as time progresses, they will include the first image, the second image, ...

[0059] During playback processing, as shown in Figure 4B, the connection part 210 of the arm part 20 is connected to the connection part 310 of the hand part 30. When the robot device 10 performs playback, it compares the images included in the motion record images captured by the camera part 330 with the images included in the motion teaching images. The robot device 10 then controls the movement (e.g., position and orientation) of the arm part 20 so that the first image included in the motion record images is the same as the first image included in the motion teaching images.

[0060] Next, the robot device 10 controls the movement of the arm 20 so that, similarly for the second motion performance image, the second image included in the motion performance image is the same as the second image included in the motion teaching image. In other words, the robot device 10 controls the arm 20 so that the position and orientation of the camera unit 330 during motion performance image acquisition follows the position and orientation of the camera unit 330 during motion teaching image acquisition, and the movement of the hand unit 30 during motion performance image acquisition follows the movement of the hand unit 30 during motion teaching image acquisition. Since the motion performance image follows the motion teaching image, the timing at which the same image is obtained for the motion performance image is delayed compared to the motion teaching image. In Figure 5, the first image included in the motion teaching image and the second image including the motion performance image are images with the same content.

[0061] Figure 6 is a flowchart showing an example of the operation of the motion teaching system 5 during the teaching process. The teaching process is the process for obtaining motion teaching images.

[0062] The motion control unit 230 of the arm unit 20 puts the hand unit 30 into an imaging (recording) standby state while the hand unit 30 is connected to the arm unit 20 (step S11). When the camera unit 330 of the hand unit 30 detects that the connection part 310 has been disconnected from the connection part 210 of the arm unit 20, it starts capturing (recording) motion teaching images (step S12).

[0063] While the operator HM grasps the gripping part 360 of the hand part 30 and performs a movement operation, the camera part 330 captures motion instruction images and sends these motion instruction images to the storage part 250 of the arm part 20 via, for example, the communication part 260 and the communication part 350, that is, sequentially sends the images included in the motion instruction images (step S13). The camera part 330 causes the storage part 250 to store the motion instruction images, that is, sequentially stores the images included in the motion instruction images (step S13).

[0064] Similarly, the camera unit 330 captures motion teaching images and sends them to the storage unit 420 of the information processing device 40 via, for example, the communication unit 260 and the communication unit 430, that is, it sequentially sends the images included in the motion teaching images. The camera unit 330 causes the storage unit 420 to store the motion teaching images, that is, it sequentially stores the images included in the motion teaching images. Various processing is performed on the motion teaching images in the information processing device 40. Alternatively, the arm unit 20 may send the motion teaching images to the information processing device 40 all at once after the teaching process is completed, for example, via the communication unit 260 or the storage unit 250.

[0065] The processing unit 410 of the information processing device 40 calculates the certainty (reliability) of the teaching process based on two time-series consecutive images (also called consecutive images) included in the motion teaching image (step S14). The certainty of the teaching process can be said to be the certainty of the playback process, for example, whether or not the playback process can be accurately executed based on the motion teaching operation, and is expressed as, for example, the success rate of the playback process or the success rate of grasping the workpiece WK. In this case, the accuracy of being able to discriminate the change in the position (imaging position) of the camera unit 330 from the two consecutive images (also called the position change discrimination accuracy) contributes to the above certainty. If this position change discrimination accuracy is high, the success rate of the playback process and the success rate of grasping will also be high. It should be noted that the processing unit 410 can also derive the certainty using only the position change discrimination accuracy.

[0066] The processing unit 410 determines whether the calculated certainty is greater than or equal to the threshold th1 (step S15). If the certainty is less than the threshold th1 (No. in step S15), the notification unit 340 notifies warning information (step S16). This warning information includes, for example, information guiding the user to redo the teaching process.

[0067] If the certainty is greater than or equal to the threshold th1 (Yes in step S15), the camera unit 330 continues to capture the motion instruction image without issuing a warning in step S15.

[0068] The processes in steps S14 to S16 are included in the certainty determination process, which determines the certainty of the result. Details of the certainty determination process will be described later.

[0069] Then, the operator HM moves the hand unit 30 and attaches the connection part 310 of the hand unit 30 to the connection part 210 of the arm unit 20, thus connecting them. When the camera unit 330 detects that the connection part 310 has been attached (connected) to the connection part 210, it terminates the capture (recording) of the motion teaching image (step S17). Consequently, the camera unit 330 also terminates the transmission of the motion teaching image to the storage unit 250 of the arm unit 20 and the transmission of the motion teaching image to the storage unit 420 of the information processing device 40.

[0070] The processing unit 410 determines whether or not the workpiece WK is visible based on the image included in the operation teaching image. If the workpiece WK is not visible, the notification unit 340 notifies warning information (step S18). This warning information may include, for example, information indicating that the workpiece WK is not visible. The warning information is notified to the user, for example, and confirmed by the user. On the other hand, if the workpiece WK is visible, the process proceeds to step S19. Step S18 is included in the workpiece absence notification process, which notifies when the workpiece WK is absent.

[0071] The processing unit 410 performs a decimation process to delete a portion of the sequence of images included in the motion instruction image (step S19). Details of the decimation process will be described later.

[0072] The processing unit 410 displays the sequence of images included in the motion teaching image on the display unit 450 and performs a masking process to mask a portion of the area of ​​at least one image included in the motion teaching image (step S20). Masking here means, for example, ignoring the information within the masked area when controlling the arm unit 20 during playback processing. Details of the masking process will be described later.

[0073] With this teaching process, the motion teaching system 5 can easily start and stop recording motion teaching images, and acquire motion teaching images safely and easily. Furthermore, by performing the certainty determination process while the teaching process is running, the motion teaching system 5 can quickly redo the teaching process. As a result, the motion teaching system 5 can, for example, increase the number of feature points in the image that are effective for certainty determination, and is expected to improve certainty.

[0074] Figure 7 is a flowchart showing an example of the operation of the motion teaching system 5 during playback processing.

[0075] The playback unit 240 of the arm unit 20 starts playback processing when it receives, for example, an operation start instruction. Playback processing is a process that controls the arm unit 20 so that the hand unit 30 performs the same operations as the hand unit 30 performed during teaching processing.

[0076] When playback processing is started, the playback unit 240 acquires an image from the storage unit 250 that is included in the motion instruction image at a predetermined point in time (step S31).

[0077] Furthermore, when playback processing begins, the playback unit 240 sends an imaging instruction to the camera unit 330, for example via the communication unit 260 and the communication unit 350, causing the camera unit 330 to start capturing (recording) images of the operational performance. The playback unit 240 then acquires the captured operational performance images from the camera unit 330, for example via the communication unit 260 and the communication unit 350, that is, it sequentially acquires the images included in the operational performance images (step S32).

[0078] The playback unit 240 compares the image included in the motion instruction image with the image included in the actual motion image and estimates the difference between the camera position (position of the camera unit 330) at the time the image included in the motion instruction image was captured and the camera position at the time the image included in the actual motion image was captured (step S33).

[0079] The motion control unit 230 controls the arm unit 20 so that the difference in the estimated camera position is less than or equal to the threshold th2 (step S34). The control of the arm unit 20 here includes controlling the movement of the arm unit 20 (e.g., position and orientation).

[0080] The playback unit 240 refers to the storage unit 250 and determines whether or not there is a subsequent image in the time series to the image included in the motion instruction image acquired in step S31 (step S35).

[0081] If subsequent images exist (Yes in step S35), the arm unit 20 repeatedly performs processes such as acquiring the next image included in the motion teaching image, acquiring the image included in the motion performance image for the time after the time acquired in step S32, estimating the difference in camera position based on these images, and controlling the arm unit 20 until no further images exist.

[0082] If there are no subsequent images included in the motion instruction image (No. in step S35), the playback process shown in Figure 7 is terminated.

[0083] With this playback processing, the motion teaching system 5 can use the motion teaching image to control the arm unit 20 to obtain an image included in the motion performance image that follows the image included in the motion teaching image. For example, if the motion teaching image is captured in such a way that it shows the operator HM gripping the workpiece WK while moving the hand unit 30 during the teaching process, the motion teaching system 5 can perform playback processing so that the arm unit 20 can grip the workpiece WK while moving the hand unit 30.

[0084] Next, we will explain the details of the certainty determination process.

[0085] If there are no features in the image included in the motion training image, the accuracy (success rate) of the playback process may decrease. This is because, when recognizing the movement of the hand unit 30 by tracking changes in feature points in the image during playback processing, it becomes difficult to recognize the movement of the hand unit 30.

[0086] An example of an image lacking distinctive features is an image in which regular shapes, patterns, or colors, such as a checkerboard pattern, appear regularly. In such an image, even if the image changes between this image and subsequent images included in the motion instruction image, the content of the image does not change much, making it difficult to determine, for example, which direction the hand unit 30 moved. In this case, the accuracy of the playback processing will be low.

[0087] Similarly, an example of an image lacking features is one in which the image contained within the motion instruction image is monochromatic. This monochromatic image may occur, for example, when the image is too dark or when the camera unit 330 is too close to the workpiece WK.

[0088] In response, the processing unit 410 of the information processing device 40 performs a certainty determination process. The certainty determination process may be performed, for example, during the teaching process as shown in Figure 7, or after the teaching is completed and before the playback process. In the certainty determination process, the processing unit 410 calculates the certainty and displays information regarding the certainty on the display unit 450. If the certainty is less than the threshold th1, the processing unit 410 notifies the notification unit 340 of warning information, for example, via the communication unit 430 and the communication unit 350. Upon receiving the notification of the warning information, the operator HM may grasp the hand unit 30 and repeat the teaching process.

[0089] Furthermore, in the certainty determination process, if the certainty is less than the threshold th1, the processing unit 410 may derive the reason for the low certainty. Reasons for low certainty may include, for example, the presence of a regular pattern in the image or the image being a single color. The processing unit 410 may determine the reason for the low certainty based, for example, on each pixel in the image included in the motion teaching image that is the subject of the certainty calculation.

[0090] Furthermore, while the following two methods are possible for calculating certainty, other methods may also be used.

[0091] In the first method of calculating certainty, the processing unit 410 extracts feature points from the image contained in the motion instruction image and derives (e.g., calculates) certainty based on the number of feature points and the randomness of the position of each feature point.

[0092] In the second method of calculating certainty, the processing unit 410 derives (e.g., calculates) the certainty according to a trained model for certainty derivation. In this case, the processing unit 410 acquires two consecutive time-series images included in the operational performance image and calculates a certainty value based on these two images according to the trained model for certainty derivation.

[0093] Figure 8 is a diagram illustrating the method for deriving certainty using a pre-trained model. The training model for certainty derivation uses as example data two consecutive time-series images included in the motion training images: an image at a first time point and an image at a second time point preceding the first time point. The training model for certainty derivation also uses as ground truth data the estimated relative coordinates (i.e., difference) between the camera position at the first time point and the target camera position, and the certainty value as the predicted difference between the estimated relative coordinates and the true value. The processing unit 410 trains the training model using deep learning based on the example data and ground truth data, generates a pre-trained model for certainty derivation, and stores it in, for example, the storage unit 420. The processing unit 410 may also acquire a pre-trained model for certainty derivation generated by a device other than the information processing device 40 and store it in the storage unit 420.

[0094] Figure 9 shows examples of both feature-rich and featureless images.

[0095] In Figure 9, an image of an apple is shown as a characteristic image. A predetermined image in the motion teaching image is shown as the current image, and the image immediately following the current image is shown as the target image. With characteristic images, it is possible to determine how the camera unit 330 should operate from the state of the apple in the current image to the state of the target image. This sequence of apple images is determined to have a certainty of th1 or higher.

[0096] On the other hand, Figure 9 shows a checkerboard pattern image as an image without features. With an image without features, it is impossible to determine how the camera unit 330 should operate to change the current checkerboard pattern state to the target image state. This sequence of checkerboard images is determined to have a certainty less than the threshold th1.

[0097] Through this certainty determination process, the motion teaching system 5 can inform the user of the certainty of the teaching process. Furthermore, if the certainty is low, the user can take necessary measures, such as gripping the hand unit 30 and running the teaching process again.

[0098] Next, we will explain the details of the thinning process.

[0099] In the teaching process, the movement of the hand unit 30 by the operator HM may include movements that are unnecessary for the movement of the hand unit 30 by the arm unit 20 during the playback process. In this case, unnecessary time is required for the playback process. Unnecessary movements here may include, for example, movements of the arm unit 20 that take an unnecessary path in the Cartesian coordinate system, movements that are not the shortest when considering the movement of the joints of the arm unit 20, or movements caused by hand tremors.

[0100] In response, the processing unit 410 performs a decimation process to remove (delete) some of the images included in the motion teaching image in a way that does not adversely affect the playback process. In other words, the processing unit 410 may control, for example, the communication unit 430 and the communication unit 260 to delete the corresponding images to be decimated from the storage unit 250. To avoid adversely affecting the playback process means, for example, that the accuracy of the teaching process does not become too low, and that the number of images showing the periphery of the workpiece WK does not decrease.

[0101] FIG. 10 is a flowchart showing an operation example during the thinning process of the operation instruction system 5. In FIG. 10, as an example, the operation instruction image is 30 fps, and 30 images are obtained per second.

[0102] The processing unit 410 of the information processing apparatus 40 randomly acquires three consecutive images among the images included in the operation instruction image (step S41). Here, the three consecutive images are, in ascending order of time series, image K i-1 , image K i , and image K i+1 . Note that three is an example, and four or more consecutive images may be acquired.

[0103] The processing unit 410 determines whether any of the images of image K i-1 , image K i , and image K i+1 has been deleted (thinned) (step S42). If it is determined that it has been deleted (Yes in step S42), the processing unit 410 ends the thinning process of FIG. 10. This is to avoid thinning an amount of images above a certain ratio.

[0104] On the other hand, if it is determined that it has not been deleted (No in step S42), the processing unit 410 estimates the certainty of the teaching process from image K i-1 to image K i+1 (between image K i-1 and image K i+1 ) (step S43).

[0105] The processing unit 410 determines whether the certainty value from image K i-1 to image K i+1 is equal to or greater than the threshold th11 (step S44). If the certainty value is less than the threshold th11 (No in step S44), the processing unit 410 ends the thinning process of FIG. 10. This is to suppress the playback process from being performed based on consecutive images with low certainty and to prevent the failure of the hand unit 30 to grip the workpiece WK.

[0106] On the other hand, if the certainty value is greater than or equal to the threshold th11 (Yes in step S44), the processing unit 410 processes the image K i-1 The system determines whether the hand portion 30 (e.g., the working portion 320) is far from the workpiece WK during imaging. In this case, it may be determined whether the distance between the hand portion 30 (e.g., the working portion 320) and the workpiece WK is greater than or equal to a threshold th3.

[0107] If the hand unit 30 is close to the workpiece WK (No. in step S45), that is, if the distance between the hand unit 30 and the workpiece WK is less than the threshold th3, the processing unit 410 terminates the decimation process shown in Figure 10. This is to record as many images as possible of the hand unit 30's movements related to gripping the workpiece WK in the range of close distance from the workpiece WK. As a result, the hand unit 30 can perform fine movements on the workpiece WK during playback processing, enabling smooth gripping of the workpiece WK.

[0108] If the hand unit 30 is far from the workpiece WK (Yes in step S45), that is, if the distance between the hand unit 30 and the workpiece WK is greater than or equal to the threshold th3 (Yes in step S45), the processing unit 410 processes image K i Remove (thin out) them (step S46).

[0109] The processing unit 410 processes all of the motion teaching images into image K i-1 Image K i Image K i+1 It is determined whether the processing in steps S41 to S46 has already been performed on any of the images (step S47). If it has been performed (Yes in step S47), the processing unit 410 terminates the decimation process in Figure 10.

[0110] On the other hand, if the process has not been completed for all images (No. in step S47), the processing unit 410 proceeds to step S41, again randomly acquires three other consecutive images from the images included in the motion instruction image, and repeats the subsequent processing. In this way, the processing unit 410 performs decimation as necessary for all images included in the motion instruction image.

[0111] Figures 11A and 11B are diagrams that supplement the thinning process.

[0112] In Figure 11A, image K i-1 An apple is reflected in the image K i and image K i+1 An apple and a banana are visible in the image. In Figure 11A, the processing unit 410 can estimate that there is the same regularity in the movement of the apple and the movement of the banana in the images arranged in chronological order, and can estimate the movement of the hand unit 30. In such a case, the certainty value becomes greater than or equal to the threshold th11, and the processing unit 410 determines the image K i You may delete it.

[0113] On the other hand, in Figure 11B, Image K i-1 An apple is reflected in the image K i Apples and bananas are visible in the image K i+1 A banana is visible in the image. The processing unit 410 determines that the movement of the apple and the movement of the banana are discontinuous in the images arranged in chronological order, and it is impossible to estimate that they follow the same regularity, making it impossible to estimate the movement of the hand unit 30. In such cases, the certainty value becomes less than the threshold th11, and the processing unit 410 determines that image K i Do not delete it.

[0114] With this type of decimation process, the motion teaching system 5 can accurately grasp the environment in which the workpiece WK is placed by the continuous images when the certainty is high. Therefore, in this case, even if some of the continuous images are decimated, the number of images can be reduced while suppressing a decrease in the accuracy of the playback process. Furthermore, the motion teaching system 5 can reduce the number of images when the spatial position corresponding to the image range of an image is far from the workpiece WK, and increase the number of images when it is close. In this case, by playing back the motion teaching images, the motion teaching system 5 can control the movement of the arm 20 so that the robot hand moves roughly and quickly when it is far from the workpiece WK, and can control the movement of the arm 20 so that the hand 30 moves finely and accurately when it is close to the workpiece WK.

[0115] Note that in Figure 10, either step S43 and step S44 or step S45 may be omitted. In other words, the certainty of the continuous images and the distance from the workpiece WK may not be taken into consideration.

[0116] The above decimation process was used to determine the images to be deleted from the motion instruction images, but this is not the only method. For example, the user may specify the desired images to be deleted using the UI. In this case, the processing unit 410 may receive user input via the input unit 440 and arbitrarily specify the images to be deleted. As a result, these images will be skipped during playback processing.

[0117] Next, we will explain the details of the masking process.

[0118] If the environment being played back changes, the teaching process must be executed again. This is because the movement of the arm 20 and hand 30 may change during the playback process, potentially leading to failure to grip the workpiece WK. Changes in the environment include, for example, the movement of objects in the environment, or the change in the state of objects in the environment (for example, changes in the markings on an object, such as the date or serial number).

[0119] Figure 12A shows the result of playback processing when no mask is used.

[0120] In Figure 12A, within the environment where the teaching process is performed, an apple exists as the workpiece WK in the image at a predetermined point in time (referred to as "present" in Figure 12A) included in the motion teaching image. Then, in the image immediately following the present in the time series (i.e., the target image), a tool that is not the workpiece WK has moved in, and both the apple and the tool are present. Therefore, the image included in the motion performance image shows both the apple and the tool. When the processing unit 410 performs playback processing in this state, the position of the hand unit 30 relative to the apple in the image included in the motion performance image is different from the position of the hand unit 30 relative to the apple in the present and target images.

[0121] Furthermore, a change in the state of an object in the environment could occur, for example, when the markings on the object change. For instance, suppose the serial number displayed on an object in the environment changes from "111000" during the teaching process to "000111" during the playback process. In this case, the number "1" is extracted as a feature point, and the relative position of the hand unit 30 to the object may differ between the teaching process and the playback process. In this case, the success rate of playback may decrease.

[0122] In response, the processing unit 410 masks a portion of the image included in the motion teaching image and controls the arm unit 20 based on the area of ​​the image other than the mask.

[0123] Figure 12B shows the result of the playback process when masking is applied.

[0124] In Figure 12B, within the environment where the teaching process is performed, an apple exists as the workpiece WK in the image at a predetermined point in time (referred to as "present" in Figure 12B) included in the motion teaching image. Then, in the image immediately following the present in the time series (i.e., the target image), a tool, which is not the workpiece WK, has moved in, and both the apple and the tool exist. Here, a mask is applied to the area where the tool exists. Therefore, the image is not affected by the tool's reflection during playback. Thus, even if the processing unit 410 performs playback processing in this state, the position of the hand unit 30 relative to the apple in the image included in the motion performance image is the same as the position of the hand unit 30 relative to the apple in the current and target images. In other words, the image included in the motion performance image follows the image included in the motion teaching image.

[0125] Figure 13 is a flowchart showing an example of the operation of the motion teaching system 5 during mask processing.

[0126] The processing unit 410 of the information processing device 40 generates a region-divided image G2 by dividing an arbitrary image G1 included in the motion teaching image into multiple regions, and displays the region-divided image G2 on the display unit 450 (step S51). Details of the generation of the region-divided image G2 will be described later.

[0127] The user confirms the displayed region division image G2. The processing unit 410 receives user input, for example via the input unit 440, and specifies the region to be masked (also called the mask region MK) from among the multiple regions in the region division image G2 (step S52). As an example, the mask region MK may be an area such as an object whose position or state may change, the entire background, or a person moving within the environment.

[0128] The processing unit 410 masks the mask region MK in the region segmentation image G2 (step S54). Masking here means ignoring the mask region MK in image G1 as if it did not exist, in other words, disabling the mask region MK.

[0129] The processing unit 410 automatically designates the same area as the mask area specified in step S52 as the mask area for other images (other consecutive images) included in the motion teaching image and masks them (step S55). In this case, other region division images corresponding to the other images may be generated based on the other images, and the mask area may be automatically designated in the other region division images. Note that the position of the mask area in each image designated as the "same area" may differ from image to image. This is because the position of the mask area in the image changes when the positional relationship between the position of the object to be masked (for example, the tool mentioned above) and the camera unit 330 changes.

[0130] The processing unit 410 calculates the confidence value between consecutive images in a time series for each image included in the motion training image, for both the state in which a mask is not applied to the mask region and the state in which a mask is applied to the mask region. In this case, for the state in which no mask is applied, the processing unit 410 derives the confidence between multiple consecutive images included in the motion training image (confidence without mask application) according to a trained model, for example. For the state in which a mask is applied, the processing unit 410 derives the confidence between multiple images in the motion training image where the mask region is masked (confidence with mask application) according to a trained model, for example.

[0131] The processing unit 410 derives (for example calculates) the change from the confidence level without mask application to the confidence level with mask application, and displays information regarding the change in confidence on the display unit 450 (step S56). The information regarding the change in confidence may include information such as, for example, "the estimation accuracy after masking has deteriorated by 2%." In other words, the degree to which the estimation accuracy (corresponding to confidence) of the operation during playback processing deteriorates due to the deletion of the mask area may be estimated and displayed.

[0132] The processing unit 410 determines whether or not to cancel the mask (step S56). Cancellation of the mask may be specified, for example, by pressing the cancel button. In other words, the processing unit 410 may determine whether or not the cancel button has been pressed. Note that cancellation of the mask may also be specified by means other than pressing the cancel button. For example, if the user checks the display of information regarding the change in certainty and determines that the mask reduces certainty and the impact is significant, the user may specify the cancellation of the mask via the input unit 440.

[0133] If the processing unit 410 cancels the mask (Yes in step S56), it cancels (releases) the mask of the mask region in each image of the motion teaching image (step S57).

[0134] After canceling the mask of the masked area in each image (after the processing in step S57), or if the mask is not canceled (No. in step S56), the processing unit 410 determines whether or not to terminate the masking process (step S58). The termination of the masking process may be specified, for example, by pressing the termination button. In other words, the processing unit 410 may determine whether or not the termination button has been pressed. Note that the termination of the masking process may also be specified by methods other than pressing the termination button.

[0135] If the masking process is not terminated (No in step S58), the processing unit 410 proceeds to step S52 and specifies another mask region in the region segmentation image. On the other hand, if the masking process is terminated (Yes in step S58), the processing unit 410 terminates the masking process shown in Figure 14.

[0136] Figure 14A shows an example of the display of region segmentation image G2. Figure 14B shows an example of image G1 with the mask region masked.

[0137] The processing unit 410 generates a segmented image G2 from an arbitrary image G1, for example, based on the SAM (Segment Anything Model). The display unit 450 may display image G1 and segmented image G2 simultaneously or individually. The processing unit 410 may also receive user input via the input unit 440 (e.g., a mouse) and manually specify the region.

[0138] Figure 14A displays the region segmentation image G2. The processing unit 410 may then display message information along with the region segmentation image G2, prompting the user to specify the mask region MK. This message information may include, for example, a message such as "Please select the region to ignore during operation." Figure 14B shows the state in which the specified mask region MK is masked, along with image G1. Note that the region segmentation image G2 in Figure 14A is generated based on image G1 before the mask region MK is masked.

[0139] With this type of masking, the motion teaching system 5 can, for example, designate a region in the motion teaching image containing an object whose state is expected to change during playback processing as a mask region. This reduces the certainty due to the change in state, thereby lowering the accuracy of the playback processing. Therefore, the motion teaching system 5 can suppress failures in gripping the workpiece WK.

[0140] Next, an example of the imaging range of the camera unit 330 will be described.

[0141] During the teaching process, if the workpiece WK is within the field of view of the camera unit 330, that is, within the imaging range of the camera unit 330, the hand unit 30 can follow the movement of the workpiece WK even if the workpiece WK moves during the playback process compared to the teaching process. On the other hand, if the workpiece WK is outside the field of view of the camera unit 330, that is, outside the imaging range of the camera unit 330, it is difficult for the hand unit 30 to follow the movement of the workpiece WK if the workpiece WK moves during the playback process compared to the teaching process.

[0142] Figure 15 shows that the workpiece WK is outside the field of view of the camera unit 330.

[0143] In Figure 15, since the workpiece WK is outside the imaging range CR, the image captured by the camera unit 330 does not show the workpiece WK, but rather the background. Therefore, during playback processing, the motion control unit 230 controls the movement of the arm unit 20 based on the background. As a result, the hand unit 30 cannot follow the movement of the workpiece WK, and the movement of the arm unit 20 is controlled so that the hand unit 30 performs the same action as during teaching processing, making it highly likely that the gripping of the workpiece WK will fail.

[0144] In contrast, in the hand unit 30, the relative positions of the working part 320 and the camera unit 330 may be fixed so that the working part 320 is located within the imaging range CR of the camera unit 330, that is, so that it is visible in the image captured by the camera unit 330.

[0145] Figure 16 shows that the working part 320 of the hand unit 30 is located within the imaging range CR of the camera unit 330.

[0146] Since the working part 320 is positioned within the imaging range CR of the camera unit 330, when the working part 320 grips the workpiece WK, the workpiece WK inevitably enters the imaging range CR. In this case, the arm unit 20 can be controlled to move in accordance with the movement of the workpiece WK, even if the workpiece WK moves during playback compared to the teaching process.

[0147] Alternatively, instead of fixing the relative position of the working part 320 and the camera part 330 so that the hand part 30 is visible in the image captured by the camera part 330, the processing unit 410 may perform work absence notification processing. Work absence notification processing is a process that notifies that work WK is absent when work WK is not within the imaging range CR.

[0148] In the work absence notification process, the processing unit 410 may display warning information on the display unit 450 if the work WK is not visible in any of the images included in the recorded operation instruction images, that is, if the work WK is not included in the imaging range CR of the camera unit 330. The warning information may include, for example, information indicating that the work WK is not present within the imaging range CR of the camera unit 330, information prompting the movement of the work WK or the hand unit 30 so that the work WK is present within the imaging range CR of the camera unit 330, etc.

[0149] Warning information may be notified by means other than display by the display unit 450. For example, a notification unit (e.g., notification unit 340) may be provided in any part of the operation teaching system 5 to notify warning information (e.g., by sound output, vibration, transmission, or other notification).

[0150] The camera unit 330 may have multiple cameras (for example, a main camera and a sub-camera). Instead of displaying warning information, the processing unit 410 may control the switching between multiple images captured by the multiple cameras, or control the widening of the imaging range CR of the camera unit 330, so that the operation unit 320 is positioned within the imaging range of the camera unit 330. For example, each of the multiple cameras has a different relative position to the operation unit 320, and the relative position is fixed. Each of the multiple cameras has captured an operation teaching image during the teaching process and stored it in one of the storage units. The processing unit 410 may select an image from the multiple operation teaching images captured and stored by the multiple cameras and perform playback processing. Image selection is performed, for example, by user input via the input unit 440.

[0151] Furthermore, the work absence notification process may also be applied when the hand unit 30 is configured such that the relative positions of the working unit 320 and the camera unit 330 are fixed so that the hand unit 30 is visible in the image captured by the camera unit 330.

[0152] Figure 17 is a flowchart showing an example of the operation when processing a work absence notification.

[0153] The processing unit 410 performs image recognition on a predetermined image included in the motion teaching image (step S61). The processing unit 410 determines whether or not the workpiece WK is visible in this image (step S62). If the workpiece WK is visible (Yes in step S62), the processing unit 410 terminates the workpiece detection process shown in Figure 17.

[0154] If the workpiece WK is not visible in the image (Yes in step S62), the processing unit 410 determines whether or not the presence or absence of the workpiece WK has been confirmed in all images included in the operation teaching image (step S63).

[0155] If there are images in the motion teaching images for which the presence or absence of workpiece WK has not been checked (No. in step S63), the processing unit 410 proceeds to step S61. In other words, for the next image, the processing in steps S61 to S63 is repeated until there are no more images for which the presence or absence of workpiece WK has not been checked.

[0156] If there are no images in the operation teaching images for which the presence or absence of workpiece WK has not been checked (Yes in step S63), the processing unit 410 notifies (for example, displays) warning information via the display unit 450 (step S64). In other words, the processing unit 410 notifies warning information if workpiece WK is not present in any of the images.

[0157] This workpiece absence notification process prevents the motion teaching system 5 from failing to grasp the workpiece WK during playback processing because it is unable to track the workpiece WK due to its absence in the motion teaching image. Furthermore, the user who receives the warning notification can take countermeasures against the absence of the workpiece WK, such as re-executing the teaching process or changing the position of the workpiece WK or the hand unit 30.

[0158] The aforementioned thinning, masking, and work absence notification processes are performed, for example, after the teaching process and before the playback process.

[0159] In this embodiment of the motion teaching system 5, the positional relationship between the working part 320 of the hand unit 30 and the camera unit 330 is fixed, so the position of the hand unit 30 is fixed with respect to the image captured by the camera unit 330. Therefore, for example, the movement of the hand unit 30 can be estimated based on the workpiece WK and background captured in the image by the camera unit 330. The motion teaching system 5 records the movement of the hand unit 30 with the camera unit 330 during the teaching process, and during the playback process, controls the arm unit 20 so that the hand unit 30 moves in a way that produces an image similar to the motion teaching image, thereby enabling the hand unit 30 to grasp the workpiece WK. Furthermore, the motion teaching system 5 eliminates the need to give specialized instructions for the arm unit 20 to move, making it easier for the user to intuitively understand the movement of the arm unit 20. In addition, the motion teaching system 5 eliminates the need for a person to directly grasp and move the arm unit 20 while it is operating with the power on, and can assist in moving the hand unit 30 accurately without a person applying force. Furthermore, the motion teaching system 5 can teach movements by detaching the hand unit 30 from the arm unit 20, and can perform teaching operations with the power to the arm unit 20 turned off, thus enabling safe teaching operations. Therefore, the motion teaching system 5 can teach the movements of the robot arm safely and easily.

[0160] In this embodiment, the start and end of recording of the motion teaching image are exemplified as being performed based on the attachment and detachment of the hand unit 30 to the arm unit 20, but this is not limited to this. For example, the hand unit 30 may be equipped with a recording start button, and the camera unit 330 may start and stop recording in response to the pressing of the recording start button.

[0161] In this embodiment, the arm unit 20 has a storage unit 250 that stores motion instruction images, motion record images, etc., but is not limited to this. For example, the storage unit that stores motion instruction images, motion record images, etc. may be provided in the hand unit 30, or it may be included in a storage unit 420 provided in the information processing device 40.

[0162] In this embodiment, the camera unit 330 of the hand unit 30 is shown as storing images included in the captured motion teaching image, that is, at least some of the images constituting the motion teaching image, in a storage unit (e.g., storage unit 250, storage unit 420), but it is not limited to this. The camera unit 330 or other components of the motion teaching system 5 may store images based on the motion teaching image in a storage unit. The images based on the motion teaching image may be, for example, any of the images constituting the motion teaching image itself, an image obtained as an intermediate product from the operation teaching image (e.g., an image obtained in the intermediate layer of deep learning), or any other image based on the motion teaching image.

[0163] (Summary of the embodiment) Based on the above, this disclosure contains at least the following information. The components and other elements in parentheses are examples of those corresponding to the embodiments described above, but are not limited to these.

[0164] (Item 1) A motion teaching system (motion teaching system 5) for teaching the movement of a robot arm (arm section 20), The robot arm is equipped with a robot hand (hand section 30) that can be connected to and detached from the robot arm, The aforementioned robot hand is The part that acts on the workpiece (acting part 320), A camera unit (camera unit 330) is fixed in position relative to the aforementioned working part and captures images, Equipped with, The camera unit captures a first teaching image (motion teaching image) while the robot hand, which has been detached from the robot arm and is being held by a person (worker HM), is moving, and stores an image based on the first teaching image in the storage unit (storage unit 250). Action teaching system.

[0165] As a result, the motion teaching system has a fixed positional relationship between the working part and the camera part, so the position of the robot hand is fixed relative to the image captured by the camera part, and the movement of the robot hand can be estimated. By capturing the movement of the robot hand with the camera part during the teaching process, which is when the first teaching image is captured, the robot arm will move in such a way that the robot hand moves to obtain an image similar to the first teaching image during the playback process, and it can be expected that the robot hand will be able to grasp the workpiece. Furthermore, it is not necessary to give specialized instructions for the robot arm to move, and the user can easily understand the movement of the robot arm intuitively. In addition, it is not necessary for a person to directly grasp and move the robot arm during its movement, and the robot hand can be moved accurately without human force. Therefore, the motion teaching system can easily teach the movement of the robot arm.

[0166] (Item 2) It further includes a processing unit (processing unit 410), The aforementioned processing unit, Based on a plurality of sequentially consecutive first images contained in the first teaching image, the certainty of controlling (playback processing) the movement of the robot arm so that a plurality of second images (images included in the motion record image) identical to the plurality of first images are captured by the camera unit while the robot hand is connected to the robot arm is estimated. If the certainty value is less than the first threshold (threshold th1), the notification unit (notification unit 340) is instructed to notify the first warning information. The operation teaching system described in item 1.

[0167] If the accuracy of the playback process is low, the robot hand is likely to perform different actions during playback than during the teaching process, potentially leading to failure to grasp the workpiece. In such cases, the motion teaching system will notify the user of a first warning, allowing the user to take countermeasures, such as re-capturing the first teaching image.

[0168] (Item 3) The aforementioned camera unit is When the robot hand is removed from the robot arm, the first teaching image is started to be captured. The acquisition of the first teaching image is terminated when the robot hand is connected to the robot arm. An operation teaching system as described in item 1 or 2.

[0169] During the teaching process, recording can be started in response to the removal of the robot hand from the robot arm, which is an essential action when the operator moves the robot hand. Similarly, during the playback process, recording can be started in response to the attachment of the robot hand to the robot arm, which is an essential action when the robot arm moves the robot hand. Therefore, the start and end of recording of the first teaching image can be easily instructed within the natural flow between the teaching and playback processes.

[0170] (Item 4) The camera unit captures the image, including at least a portion of the working portion within the imaging range. An action teaching system as described in any one of items 1 to 3.

[0171] As a result, even if the workpiece's position changes, the workpiece will be reflected in the first teaching image when the working part grips the workpiece. Therefore, the motion teaching system can reliably recognize and grip the workpiece during playback processing.

[0172] (Item 5) Further comprising a processing unit, The aforementioned processing unit, Determine whether the workpiece is visible in the first teaching image. If the aforementioned workpiece is not displayed, the notification unit will notify the second warning information. An action teaching system as described in any one of items 1 through 4.

[0173] If the workpiece is not visible, the robot arm cannot move to target the workpiece, making it highly likely that the robot will fail to grasp the workpiece. In such cases, the motion teaching system will notify the user of a second warning, allowing the user to take countermeasures, such as re-capturing the first teaching image with the workpiece visible.

[0174] (Item 6) Further comprising a processing unit, The aforementioned processing unit, Multiple consecutive first images (image K) included in the first teaching image i-1 Image K i Image K i+1 Based on this, the certainty of controlling the movement of the robot arm so that the camera unit captures the same plurality of second images as the plurality of first images while the robot hand is connected to the robot arm is estimated. If the certainty value is greater than or equal to the second threshold (threshold th11), then some of the images included in the plurality of first images (image K) i ) delete An action teaching system as described in any one of items 1 through 5.

[0175] When the accuracy of the playback process is high, the environment in which the workpiece is placed can be accurately understood from multiple first images, and there is a high probability that the robot hand's movements during the teaching process will be accurately reproduced during the playback process, leading to successful gripping of the workpiece. In such cases, there is a high probability that the robot arm can operate accurately even if some of the consecutive first images are deleted. Therefore, the motion teaching system can reduce the number of images included in the first teaching image while suppressing a decrease in workpiece gripping accuracy.

[0176] (Item 7) Further comprising a processing unit, The aforementioned processing unit, Based on a plurality of first images that are consecutive in time series and included in the first teaching image, the position of the workpiece is recognized. If the distance between the robot hand and the workpiece is greater than or equal to a third threshold (e.g., threshold th3), some of the images included in the plurality of first images are deleted. An action teaching system as described in any one of items 1 through 6.

[0177] This allows the motion teaching system to reduce the number of first images when the robot hand is far from the workpiece and increase the number of first images when the robot hand is close to the workpiece. In this case, by replaying the first teaching images, the motion teaching system can control the movement of the robot arm so that the robot hand moves roughly and quickly when the robot hand is far from the workpiece, and can control the movement of the robot arm so that the robot hand moves finely and precisely when the robot hand is close to the workpiece.

[0178] (Item 8) Further comprising a processing unit, The aforementioned processing unit, A predetermined image included in the first teaching image is divided into multiple regions, In the predetermined image, specify the area to be masked (mask area MK) from among the multiple areas, In each image included in the first teaching image, the same region as the region to be masked is designated as the region to be masked. It is decided to control the robot arm based on the first teaching image, ignoring the area to be masked in each of the aforementioned images. An action teaching system as described in any one of items 1 through 7.

[0179] This allows the motion teaching system to control the robot arm so that it operates independently of the features within the masked area by specifying and masking an area in the image included in the first teaching image that the user deems unnecessary.

[0180] (Item 9) The system further includes a control unit (operation control unit 230, regeneration unit 240), The control unit, With the robot hand connected to the robot arm, an operation start instruction is received to start the operation of the robot arm. In response to the operation start instruction, the camera unit is instructed to start capturing the second teaching image. The robot arm is controlled so that the second teaching image matches the first teaching image. An action teaching system as described in any one of items 1 through 8.

[0181] As a result, the motion teaching system can continuously control the position of the robot hand so that the image captured by the camera unit follows the image included in the first teaching image. Therefore, the action part can exert the same effect on the workpiece as during the teaching process, for example, the action part can grasp the workpiece. In addition, by the image captured by the camera unit following the image included in the first teaching image, the relative positional relationship between the workpiece and the hand unit is reproduced. Therefore, the motion teaching system can correct the position of the robot hand to help grasp the workpiece even if the position of the workpiece deviates from that during the teaching process in the actual operating environment of the robot arm and robot hand.

[0182] (Item 10) A motion teaching method for teaching the movements of a robot arm to which a robot hand can be connected, The robot hand comprises an action part that acts on a workpiece, and a camera part whose position is fixed relative to the action part and which captures images. The first teaching image is captured while the robot hand, which is being grasped by a person, is moving. The image based on the first teaching image described above is stored in the storage unit, A method for teaching an action that includes the following:

[0183] As a result, the action teaching method achieves the same effect as item 1.

[0184] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components of the above embodiments may be combined in any way without departing from the spirit of the invention.

[0185] Furthermore, the above embodiment may also apply to a program that realizes the functions of the operation teaching method, which is supplied to a computer (for example, the operation control unit 230 or the processing unit 410) via a network or various storage media, and which is read and executed by the computer's processor, as well as the recording medium on which this program is stored. [Industrial applicability]

[0186] This disclosure is useful for motion teaching systems and motion teaching methods that can easily teach the movements of a robot arm. [Explanation of Symbols]

[0187] 5. Action Teaching System 10 Robot equipment 20 Arm section 30 Hand section 210 Connection part 220 Moving parts 230 Operation Control Unit 240 Playback Department 250 Storage section 260 Communications Department 310 Connection part 320 Working part 321 Start Instruction button 322 Hook section 323 Scraper section 324 tool mounting holes 330 Camera Section 340 Notification Department 350 Communications Department 360 Gripping part 410 Processing Unit 420 Storage section 430 Communications Department 440 Input section 450 Display section G1 Image G2 region segmentation image MK Mask Area WK Work

Claims

1. A motion teaching system for teaching the movements of a robot arm, The robot arm is equipped with a robot hand that can be connected to and detached from the robot arm, The aforementioned robot hand is The part that acts on the workpiece, A camera unit is fixed in position relative to the aforementioned working part and captures images, Equipped with, The camera unit captures a first teaching image to teach the movement of the robot hand while it is being moved after being detached from the robot arm and grasped by a person, and stores an image based on the first teaching image in the storage unit. Action teaching system.

2. Further comprising a processing unit, The aforementioned processing unit, Based on a plurality of first images that are consecutive in time series and included in the first teaching image, the certainty of controlling the movement of the robot arm so that the camera unit captures a plurality of second images that are the same as the plurality of first images while the robot hand is connected to the robot arm is estimated. If the certainty value is less than the first threshold, the notification unit is instructed to notify the first warning information. The operation teaching system according to claim 1.

3. The aforementioned camera unit is When the robot hand is removed from the robot arm, the first teaching image is started to be captured. When the robot hand is connected to the robot arm, the acquisition of the first teaching image is terminated. The operation teaching system according to claim 1 or 2.

4. The camera unit captures the image, including at least a portion of the working portion within the imaging range. The operation teaching system according to claim 1 or 2.

5. Further comprising a processing unit, The aforementioned processing unit, Determine whether the workpiece is visible in the first teaching image. If the aforementioned workpiece is not displayed, the notification unit will notify the second warning information. The operation teaching system according to claim 1 or 2.

6. Further comprising a processing unit, The aforementioned processing unit, Based on a plurality of first images that are consecutive in time series and included in the first teaching image, the certainty of controlling the movement of the robot arm so that the camera unit captures a plurality of second images that are the same as the plurality of first images while the robot hand is connected to the robot arm is estimated. If the certainty value is greater than or equal to the second threshold, some of the images included in the plurality of first images are deleted. The operation teaching system according to claim 1 or 2.

7. Further comprising a processing unit, The aforementioned processing unit, Based on a plurality of first images that are consecutive in time series and included in the first teaching image, the position of the workpiece is recognized. If the distance between the robot hand and the workpiece is greater than or equal to a third threshold, some of the images included in the plurality of first images are deleted. The operation teaching system according to claim 1 or 2.

8. Further comprising a processing unit, The aforementioned processing unit, A predetermined image included in the first teaching image is divided into multiple regions, In the predetermined image, specify the area to be masked from among the plurality of areas, In each image included in the first teaching image, the same region as the region to be masked is designated as the region to be masked. It is decided to control the robot arm based on the first teaching image, ignoring the area to be masked in each of the aforementioned images. The operation teaching system according to claim 1 or 2.

9. It further comprises a control unit, The control unit, With the robot hand connected to the robot arm, an operation start instruction is received to start the operation of the robot arm. In response to the operation start instruction, the camera unit is instructed to start capturing the second teaching image. The robot arm is controlled so that the second teaching image matches the first teaching image. The operation teaching system according to claim 1 or 2.

10. A motion teaching method for teaching the movements of a robot arm to which a robot hand can be connected, The robot hand comprises an action part that acts on a workpiece, and a camera part whose position is fixed relative to the action part and which captures images. The first teaching image is captured while the robot hand, which is being held by a person, is moving. The image based on the first teaching image is stored in the storage unit, A method for teaching an action that includes the following:

Images