Machine learning device, robot system and machine learning method for learning the state of a cable

Abstract

Machine learning device (10) configured to learn a state of a cable (22) provided on a robot (20), the device (10) comprising: a training data acquisition section (12) which is configured to acquire as a training data set image data of the cable (22) as imaged by a camera (50) while the robot (20) performs a predefined operation, and data indicating a state of the cable (22) during the performance of the predefined operation by the robot (20), and a learning section (14) which is set up to generate a learning model using the learning data set, representing a correlation between the image data and the state of the cable (22).

Description

[0001] The present invention relates to a machine learning apparatus, a robot system, and a machine learning method for learning the state of a cable attached to a robot.

[0002] A device for monitoring a cable attached to a robot is known (see e.g. JP 2017 - 226 000 A).

[0003] DE 10 2018 204 184 A1 describes a method for monitoring a supply system of a robot having a robot arm and a robot hand movable thereto. The supply system comprises a supply line, a guide for the supply line, and a number of sensors for monitoring at least one state variable of the supply system. Based on the values ​​determined by the sensors, conclusions are drawn about the functionality of the supply system for the state variable.

[0004] DE 10 2017 008 475 A1 discloses a machine learning device with a state observation unit that observes, as a state variable, a wobble of an arm of a robot and / or a length of an operating trajectory of the arm of the robot. A determination data acquisition unit acquires, as determination data, a cycle time in which the robot performs processing. A learning unit learns an operating program of the robot based on an output of the state observation unit and an output of the determination data acquisition unit.

[0005] DE 10 2015 013 988 A1 discloses a robot simulation device comprising a model generation unit that generates a three-dimensional wire-shaped element model from a plurality of mass points and a plurality of spring elements. A focus point setting unit sets a plurality of focus points on the wire-shaped element model. A simulation unit simulates the behavior of the wire-shaped element model associated with the operation of a robot, and a display unit displays focus point images together with images of the wire-shaped element model using position data of the plurality of mass points and position data of the plurality of focus points obtained through the simulation.

[0006] US 2019 / 0 126 487 A1 describes a method for generating a data set that maps visual features of a plurality of objects. In the method, for each of a plurality of different objects, a robot system is instructed to move an arm holding the respective object to a plurality of positions. When the arm is in each of the plurality of positions, an image representing the respective object in the position is captured. Position information of the arm in the respective position is received, and the image is analyzed to identify at least one visual feature of the object in the respective position. Finally, an association between the at least one visual feature and the position information is stored in an image data set, and the image data set is output.

[0007] From JP 2004 - 074 368 A it is known to detect a twisted condition of a cable installed on a robot and to display it as a collection of points on a screen.

[0008] JP H10 - 275 007 A discloses a robot operation simulation method that checks whether a non-rigid body held by a robot in its hand is wound or excessively stretched or contracted.

[0009] There is a need for a technique to quantitatively and accurately assess the condition of a cable on a robot.

[0010] To achieve this object, the invention provides a machine learning device having the features of claim 1, a robot system having the features of claim 9 and a machine learning method having the features of claim 10.

[0011] According to one aspect of the disclosure, a machine learning device configured to learn a state of a cable provided to a robot includes a learning data acquisition section configured to acquire, as a learning data set, image data of the cable imaged by a camera while the robot performs a predetermined operation and data indicating a state of the cable while the robot is performing the predetermined operation, and a learning section configured to generate a learning model representing a correlation between the image data and the state of the cable using the learning data set.

[0012] According to another aspect of the disclosure, a machine learning method for learning a state of a cable provided to a robot includes performing, by a processor, the following steps: acquiring, as a learning data set, image data of the cable imaged by a camera while the robot performs a predetermined operation and data indicating a state of the cable during the robot's performance of the predetermined operation; and generating a learning model representing a correlation between the image data and the state of the cable using the learning data set.

[0013] According to the disclosure, a model representing a correlation between image data of a cable and the condition of the cable can be easily and accurately obtained. Fig. 1 is a block diagram showing a machine learning apparatus according to an embodiment; Fig. 2 is a perspective view showing a robot according to an embodiment; Fig. 3 is a sectional view showing a cable according to an embodiment; Fig. 4 is a sectional view showing a cable according to another embodiment; Fig. 5 is a sectional view showing a cable according to another embodiment; Fig. 6 is an enlarged view showing the Fig. 2 shows an enlarged view of the connection between a wrist and an upper arm; Fig. 7 is a view for explaining a state in which the cable is bent while the robot performs a predetermined operation; Fig. 8 is a view for explaining another state in which the cable is bent while the robot performs the predetermined operation; Fig. 9 is a view for explaining another state in which the cable is bent while the robot performs the predetermined operation; Fig. 10 is a flowchart showing a process through Fig. 1 shows an exemplary learning cycle executed by the machine learning device shown; Fig. 11 shows a schematic model of a neuron; Fig. 12 shows a schematic model of a multilayer neural network; Fig. 13 is a block diagram showing a robot system according to an embodiment; Fig. 14 is a flowchart showing an exemplary operation sequence of the Fig. 13 shows the robot system shown; Fig. 15 is a block diagram showing a robot system according to another embodiment; Fig. 16 is a flowchart showing an exemplary operation sequence of the Fig. 15 shows the robot system; and Fig. 17 is a block diagram showing a network system according to an embodiment.

[0014] Embodiments of the disclosure are described in detail below with reference to the drawings. It is noted that in various embodiments presented below, similar elements are assigned the same reference numerals and a repeated description thereof is omitted. First, with reference to Fig. 1 describes a machine learning device 10 according to one embodiment. The machine learning device 10 may be formed by a computer, e.g., comprising a processor (e.g., CPU, GPU), a memory (e.g., ROM, RAM), and an input device (e.g., a keyboard, a mouse, or a touchscreen), or by software, such as a learning algorithm, executed by a computer.

[0015] The machine learning device 10 is configured to learn a state (e.g., a normal state or an abnormal state) of a cable 22 provided on a robot 20 described later. Fig. 2, the robot 20 is described. In this embodiment, the robot 20 is a vertical articulated robot and includes a base 24, a rotary body 26, a lower arm 28, an upper arm 30, a wrist 32, and a cable 22.

[0016] The base 24 is fixed to the floor of a work cell. The rotating body 26 is mounted on the base 24 so that it can rotate about a vertical axis. The forearm 28 is mounted on the rotating body 26 so that it can rotate about a horizontal axis. The upper arm 30 is rotatably mounted on a distal end of the forearm 28. The wrist 32 is rotatably mounted on a distal end of the upper arm 30.

[0017] An end effector (e.g., a welding torch, a robot hand, a tool, or a paint applicator—not shown) for performing a predetermined operation (e.g., welding, workpiece handling, machining, or painting) is removably attached to a distal end of the wrist 32. The robot 20 includes servomotors (not shown) for driving movable elements (e.g., the rotary body 26, the forearm 28, the upper arm 30, and the wrist 32) of the robot 20.

[0018] The cable 22 is routed using holding means 34 on each movable element of the robot 20. Each holding means 34 is attached to a predetermined position on the movable element of the robot 20 and forms a hole for receiving the cable 22 therein. The cable 22 is inserted into the hole of each holding means 34 and is held thereby. The attachment positions of the holding means 34 on the robot 20 are determined for each type of robot 20.

[0019] Depending on the operation performed by the robot 20, different types of cables 22 are used. Fig. Figure 3 shows a sectional view of a first type of cable 22A. The cable 22A includes a feed tube 36, transmission lines 38, and a sheath 40. The feed tube 36 is a cylindrical member extending along the axis of the sheath 40. The feed tube 36 supplies a fluid or strand used for the operation of the robot 20.

[0020] For example, when the welding torch is attached to the wrist 32 as an end effector and the robot 20 is performing a welding operation on a workpiece, the feed tube 36 supplies an assist gas as a fluid to the welding torch. Alternatively, the feed tube 36 houses a welding wire as a cable element and supplies it to the welding torch. As another example, when the tool is attached to the wrist 32 and the robot 20 is performing a machining operation on a workpiece, the feed tube 36 supplies a pressurized gas as a fluid to the tool.

[0021] The transmission lines 38 each transmit an electrical signal. The transmission lines 38 transmit signals, for example, to the servo motors installed in the robot 20 or the end effector attached to the wrist 32. In the cable 22A, the feed tube 36 and the transmission lines 38 are arranged randomly within the sheath 40.

[0022] Fig. 4 shows a sectional view of a second type of cable 22B. Cable 22B includes feeder tube 36, a total of four transmission lines 38, and sheath 40. Transmission lines 38 are arranged on an outer peripheral side of feeder tube 36 so that they are aligned at substantially equal spacing in the circumferential direction of feeder tube 36. Thus, in cable 22B, feeder tube 36, transmission lines 38, and sheath 40 are arranged concentrically.

[0023] Fig. 5 shows a sectional view of a third type of cable 22C. Cable 22C includes a total of three transmission lines 38, a fiber optic cable 42, and the jacket 40. For example, when the laser processing head is attached to the wrist 32 as an end effector and the robot 20 performs laser processing on a workpiece, a laser beam propagates through the fiber optic cable 42 to guide the laser beam to the laser processing head. In cable 22C, the transmission lines 38 and the fiber optic cable 42 are arranged randomly within the jacket 40.

[0024] As described above, several types of cables 22 (e.g., cables 22A, 22B, and 22C) can be used in the robot 20. The types of cables 22 are classified depending on, for example, the type of cable elements to be used (the feeder tube 36, the transmission lines 38, and the optical fiber 42), the number of cable elements, the arrangement of the cable elements in the sheath 40, the material of the sheath 40, and the positions of binding wires for binding the cable elements in the sheath 40 at predetermined intervals.

[0025] Referring to Fig. 2, a camera 50 is provided for imaging the cable 22. The camera 50 images the cable 22 while the robot 20 performs a predetermined operation. The "prescribed operation" may, for example, include a series of actions of the robot 20 when the robot 20 performs a work process (e.g., welding, workpiece handling, machining, or painting). When performing the prescribed operation, the robot 20 operates each movable element (the rotating body 26, the forearm 28, the upper arm 30, the wrist 32) according to a pre-generated robot program to perform the series of actions for the work process.

[0026] Alternatively, the “predetermined operation” may include an operation to rotate a first member (i.e., the rotating body 26, the forearm 28, the upper arm 30, or the wrist 32) of the robot 20 with respect to a second member (the base 24, the rotating body 26, the forearm 28, or the upper arm 30) of the robot 20 between a predetermined first position and a predetermined second position.

[0027] The camera 50 is arranged to image a predetermined position of the cable 22 along a predetermined line of sight direction while the robot 20 performs the predetermined operation. The camera 50 may be mounted at a position other than the robot 20 or attached to a member (the base 24, the rotating body 26, the lower arm 28, or the upper arm 30) of the robot 20. In this embodiment, a case where the camera 50 is arranged to image the cable 22 at a position near the connection between the wrist 32 and the upper arm 30 will be described as an example.

[0028] When the wrist 32 rotates with respect to the upper arm 30 while the robot 20 performs the predetermined operation, the cable 22 is bent at a position near the connection between the wrist 32 and the upper arm 30. Fig. 6 to 9 schematically show bending states of the cable 22 during the specified operation. It is assumed that the wrist 32 moves with respect to the upper arm 30 from a Fig. 6 shown starting position to a Fig. 7 to 9 shown first reference position.

[0029] In the Fig. 6, an axis A1 of the wrist 32 and an axis A2 of the upper arm 30 are substantially parallel to each other. In the position shown in Fig. In the first reference position shown in Figures 7 to 9, however, the axis A1 of the wrist 32 is inclined by an angle θ1 with respect to the axis A2 of the upper arm 30. Fig. 7 to 9 show exemplary bending states of the cable 22 when the wrist 32 is arranged in the first reference position relative to the upper arm 30 during the specified operation. For comparison, the Fig. 7 shown cables in the form of a dashed line over Fig. 8 and the Fig. 7 and Fig. The cables shown in Figure 8 are shown in the form of a dashed line or a dash-dotted line over Fig. 9 laid.

[0030] For example, due to an error in the holding positions of the cable 22 or a shape error of an element of the robot 20, the degree of bending of the cable 22 of the same type may be different when the wrist 32, as shown in Fig. 7 to 9, is arranged in the first reference position relative to the upper arm 30.

[0031] As in Fig. 7 and Fig. 8, for example, it can be determined that the degree of bending of the cable 22 is within an allowable range and has no influence on the performance of the cable 22 (e.g., on the signal transmission characteristics of the transmission line 28, the fluid flowability through the feed pipe 36, the light propagation characteristic of the optical fiber 42, or the bending resistance of the sheath 40). On the other hand, it can be determined that the degree of bending of the Fig. 9 is not permitted and has a certain influence on the performance of the cable 22.

[0032] The machine learning device 10 learns the state (e.g., the normal state or abnormal state) of the cable 22 while the robot 20 performs the predetermined operation. Referring to Fig. 1, the machine learning device 10 comprises a learning data acquisition section 12 and a learning section 14. The learning data acquisition section 12 acquires image data D iof the cable 22 imaged by the camera 50 while the robot 20 performs the predetermined operation, and data indicating the state S of the cable 22 while the robot 20 is performing the predetermined operation (hereinafter referred to as state data D s ''), as a learning data set DS.

[0033] Specifically, the camera 50 images the cable 22 at a position near the connection between the wrist 32 and the upper arm 30 along the line-of-sight direction VD while the robot 20 performs the specified operation. The camera 50 feeds the captured image data D i to the learning data acquisition section 12. The state data D s of the cable 22 include, for example, label information such as "normal state" or "abnormal state". The normal state means, for example, the state in which the bending of the cable 22, as shown in Fig. 7 and Fig. 8, is within the permissible range. The abnormal condition, however, refers, for example, to the condition in which the bending of the cable 22, as shown in Fig. 9 is outside the permissible range.

[0034] It is pointed out that the status data D s may include only the "normal condition" or the "abnormal condition" label information, or both the "normal condition" and the "abnormal condition" label information. Furthermore, multiple individual pieces of label information, such as "brand-new condition" and "used condition," may be specified as the "normal condition" label information. Furthermore, depending on the severity of the abnormal condition, multiple individual pieces of label information, such as "abnormal condition level 1," "abnormal condition level 2," and "abnormal condition level 3," may be specified as the "abnormal condition" label information.

[0035] The status data D s(ie, the label information) of the cable 22 are determined by an operator. For example, the operator confirms the state S (the normal state or the abnormal state) of the cable 22 by visually observing the image data D captured by the camera 50. i and outputs the status data D s into the learning data acquisition section 12. The learning data acquisition section 12 acquires the image data D acquired by the camera 50 i and the status data D determined by the operator s as learning dataset DS.

[0036] The learning section 14 generates using the learning data set DS of the image data D i and the status data D s a learning model (function) LM that calculates a correlation between the image data D iand the state S of the cable 22. The learning section 14 generates the learning model LM, for example, by performing supervised learning. In this case, the robot 20 repeatedly attempts to perform the specified operation, and the learning data acquisition section 12 repeatedly acquires the learning data set DS as teacher data each time the robot 20 performs the specified operation.

[0037] Learning section 14 learns the learning model LM by identifying features that determine the correlation between the image data D acquired as teacher data i (e.g.

[0038] Image data of the Fig. 7 to 9 shown cables 22) and status data D s (Label information, such as normal state or abnormal state). An algorithm such as a support vector machine (SVM) or a Gaussian mixture model (GMM) can be used for such supervised learning.

[0039] The flow of the learning cycle executed by the machine learning device 10 is described below with reference to Fig. 10. In step S1, the robot 20 performs the specified operation, and the camera 50 images the cable 22 while the robot 20 performs the specified operation. As an example, the camera 50 images the cable 22 at a time τ1 at which the wrist 32 is in the position shown in Fig. 7 to 9, only once while the robot 20 performs the specified operation.

[0040] As another example, the camera 50 may continuously image the cable 22 (ie, capture a moving image) while the robot 20 performs the specified operation. In this case, the camera 50 images the cable 22 at each time τ nfrom which the wrist 32 is arranged relative to the upper arm 30 in an n-th reference position (n = 1, 2, 3, ...), in which the axis A1 of the wrist 32 is displaced relative to the axis A2 of the upper arm 30 by a predetermined angle θ n is inclined.

[0041] In step S2, the learning data acquisition section 12 acquires the learning data set DS. Specifically, the learning data acquisition section 12 acquires image data D from the camera 50. i of the cable 22 imaged by the camera 50 while the robot 20 performs the predetermined operation in step S1.

[0042] Furthermore, the learning data acquisition section 12 acquires the label information of the normal state or the label information of the abnormal state as data representing the state S (state data D s) of the cable 22 during the execution of step S1. Specifically, the operator confirms whether the cable 22 is in the normal state or in the abnormal state by examining the image data D i of the cable 22 imaged by the camera 50 is visually viewed.

[0043] The operator then operates the input device of the machine learning device 10 to input the label information of the normal state or the label information of the abnormal state as state data D s The learning data acquisition section 12 acquires the input label information of the normal state and / or label information of the abnormal state. Thus, the learning data acquisition section 12 acquires the learning data set DS of the image data D i and the status data D s of the cable 22 and stores them in memory in association with each other.

[0044] In step S3, the learning section 14 generates the learning model LM, which calculates the correlation between the image data D i and the state S, using the learning data set DS acquired in step S2. Specifically, the learning section 14 learns the learning model LM by executing the supervised learning algorithm (e.g., SVM or GMM). Then, the flow returns to step S1.

[0045] By executing such a learning cycle, the learning of the learning model LM is promoted and led to an optimal solution. The operator executes the Fig. 10 for several robots 20. This makes it possible to acquire a larger number of learning data sets DS and, in turn, to promote the learning of the learning model LM to a higher level. According to this embodiment, it is possible to create a model that estimates the correlation between the image data D i and the state S of the cable 22 quantitatively represented, easily and accurately.

[0046] As another function of the machine learning device 10, the learning data acquisition section 12 may further acquire, in step S2, data on a type (a first type, a second type, ..., an nth type) of the cable 22 or data on a type of the robot 20 as a learning data set DS. The data on the type of the cable 22 includes, for example, information for identifying the type of the cable 22, information for identifying the type of the cable elements used (the feeder pipe 36, the transmission lines 38, and the optical fiber 42), the number of cable elements, information indicating the arrangement of the cable elements in the sheath 40, the material of the sheath 40, and information on the positions of the binding wires for binding the cable elements at predetermined intervals in the sheath 40.

[0047] The information for identifying the type of cable 22 may include, for example, symbol information such as "nth type" or a product number assigned to each cable by a cable manufacturer. The information for identifying the type of cable elements may include, for example, symbol information such as "feeder tube," "transmission line," or "optical fiber."

[0048] The information indicating the arrangement of the cable elements may include, for example, character information such as "arbitrary arrangement" or "concentric arrangement." The material of the sheath 40 may be indicated, for example, by character information such as "synthetic resin" or "rubber," or a product number assigned by a manufacturer of the material of the sheath 40. The information about the positions of the tie wires may include, for example, the average distance between two tie wires or the number of tie wires.

[0049] The data of the type of the robot 20 includes, for example, information for identifying the type of the robot 20, the number of laid cables 22 and information about the attachment positions of the holding means 34 on the robot 20. The information for identifying the type of the robot 20 may include, for example, character information such as "n-th type" or a product number assigned to each robot by a robot manufacturer.

[0050] As an example, in step S2, the operator inputs data D by operating the input device of the machine learning device 10 t the type of cable 22 or robot 20. Thus, the learning data acquisition section 12 acquires, in addition to the image data D i and the status data D s the data D t of the type as a learning data set DS. In step S3, the learning section 14 creates the learning model LM by adding the relationship between the image data D i and the data D tof the type for correlation between the image data D i and the status data D s .

[0051] The degree of bending of the cable 22 during the robot 20's performance of the specified operation may vary depending on the type of cable 22. For example, the greater the number of cable elements laid in the sheath 40 or the lower the flexibility of the material of the sheath 40, the more difficult the cable 22 is to bend.

[0052] In addition, the degree of bending of the cable 22 may vary depending on the type of robot 20 while the robot 20 performs the specified operation. For example, the mounting positions of the holding means 34 vary for each type of robot 20, which may cause the degree of bending of the cable 22 to vary during the specified operation.

[0053] Thus, there may be a correlation between the degree of bending of the cable 22 and the type of the cable 22 or the robot 20. Therefore, it allows the creation of the learning model LM taking into account the relationship between the image data D i and the type of cable 22 or robot 20, to obtain the learning model LM adapted to the type of cable 22 or robot 20.

[0054] As a further function of the machine learning device 10, the learning data acquisition section 12 in step S2 may, in addition to the image data D i and the status data D s further data of the position and orientation of the robot 20 (hereinafter referred to as “position data D p “) while the robot 20 performs the predetermined operation in step S1, as a learning data set DS.

[0055] As an example, when the camera 50 images the cable 22 only once at time τ1 in step S1, the learning data acquisition section 12 acquires position data D p of the robot 20 at time τ1. As another example, when the camera 50 detects the cable 22 at times τ n (n = 1, 2, 3, ...) continuously (records a moving image of the same), successively position data D p of the robot 20 at any time τ n .

[0056] The position data D p of the robot 20 can be obtained from the angle of rotation θ of each servo motor installed in the robot 20. In this case, each servo motor of the robot 20 is equipped with a sensor (e.g., an encoder or a Hall element) that detects the angle of rotation θ during image acquisition by the camera 50. As position data D pof the robot 20, the angle of rotation θ can be used. In step S3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the position and orientation of the robot 20 for the correlation between the image data D i and the status data D s .

[0057] The degree of bending of the cable 22 during the execution of the specified operation by the robot 20 may vary depending on the position and orientation of the robot 20 (e.g., the angle of the wrist 32 with respect to the upper arm 30). In other words, there may be a correlation between the degree of bending of the cable 22 and the position and orientation of the robot 20. Therefore, it allows the generation of the learning model LM taking into account the relationship between the image data D i and the position and orientation of robot 20, to obtain the learning model LM adapted to the position and orientation of robot 20.

[0058] As another function of the machine learning device 10, the learning data acquisition section 12 may further acquire, in step S2, an ambient temperature T during the performance of the predetermined operation by the robot 20 in step S1 as a learning data set DS. In this case, a sensor (temperature sensor) that detects the ambient temperature T is provided.

[0059] In step S3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the ambient temperature T for the correlation between the image data D i and the status data D s . There may be a correlation between the degree of bending of the cable 22 and the ambient temperature T during the execution of the specified operation by the robot 20.

[0060] In particular, depending on the material of the sheath 40, the higher the ambient temperature T and the higher the flexibility of the sheath 40, the easier it is to bend the cable 22. On the other hand, the lower the ambient temperature T and the lower the flexibility of the sheath 40, the harder it is to bend the cable 22. Therefore, it allows the generation of the learning model LM taking into account the relationship between the image data D i and the ambient temperature T, to obtain the learning model LM adapted to the ambient temperature T.

[0061] As another function of the machine learning device 10, the learning data acquisition section 12 may further acquire, in step S2, an electric current A flowing through the transmission line 38 or a resistance R of the transmission line 38 during the robot 20's performance of the predetermined operation in step S1 as a learning data set DS. In this case, a sensor (e.g., an ammeter or an ohmmeter) is provided that detects the electric current A or the resistance R of the transmission line 38 during the predetermined operation.

[0062] In step S3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the current A or the resistance R to correlate between the image data D i and the status data D sThe electric current A or the resistance R may vary depending on the degree of bending of the cable 22 while the robot 20 is performing the specified operation. For example, if the degree of bending of the cable 22 is outside the allowable range, the transmission line 38 may partially break, and the resistance R of the transmission line 38 may increase, while the electric current A of the transmission line 38 may decrease.

[0063] Thus, there may be a correlation between the degree of bending of the cable 22 and the electric current A or the resistance R. Therefore, it allows the generation of the learning model LM taking into account the relationship between the image data D i and the current A or the resistance R, to obtain the learning model LM adapted to the current A or the resistance R.

[0064] As a further function of the machine learning device 10, if the Fig. 3 shown cable 22A or the one in Fig. 4 is used, the learning data acquisition section 12 may further acquire, in step S2, a pressure P of the fluid in the supply pipe 36 during the performance of the predetermined operation by the robot 20 in step S1 as a learning data set DS. In this case, a sensor (e.g., a pressure gauge) is provided that detects the pressure P during the predetermined operation.

[0065] In step S3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the pressure P for the correlation between the image data D i and the status data D s The degree of bending of the cable 22 during the execution of the predetermined operation by the robot 20 may vary depending on the pressure P of the fluid in the supply pipe 36.

[0066] In particular, the higher the pressure P of the fluid in the supply pipe 36, the harder it is to bend the cable 22. Thus, there may be a correlation between the degree of bending of the cable 22 and the pressure P. Therefore, it is possible to generate the learning model LM taking into account the relationship between the image data D i and the pressure P, to obtain the learning model LM adapted to the pressure P.

[0067] As a further function of the machine learning device 10, if the Fig. 5 is used, the learning data acquisition section 12 may further acquire, in step S2, an optical characteristic OP of light (e.g., a laser beam) propagating through the optical fiber 42 while the robot 20 performs the predetermined operation in step S1, as a learning data set DS. The optical characteristic OP may include, for example, the intensity, power, or intensity distribution of the light propagating through the optical fiber 42 (or return light of the light). In this case, a sensor (e.g., an optical power meter or a beam profiler) is provided that detects the optical characteristic OP during the predetermined operation.

[0068] In step S3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the optical characteristic OP for the correlation between the image data D i and the status data D sThe optical characteristic OP may vary depending on the degree of bending of the cable 22 while the robot 20 is performing the specified operation. For example, if the degree of bending of the cable 22 is outside the allowable range, stray light may increase in the bent portion of the cable 22, which may decrease the intensity or power of the light (or return light).

[0069] Thus, there may be a correlation between the degree of bending of the cable 22 and the optical characteristic OP. Therefore, it allows the generation of the learning model LM taking into account the relationship between the image data D i and the optical characteristic OP, to obtain the learning model LM adapted to the optical characteristic OP.

[0070] The learning algorithm implemented in Section 14 is not limited to supervised learning; it can also use a well-known learning algorithm, such as unsupervised learning and reinforcement learning, or a neural network for machine learning. As an example, Fig. 11 shows a schematic model of a neuron. Fig. Figure 12 shows schematically a model of a three-layer neural network constructed by combining Fig. 11. The neural network can be implemented, for example, as a computing device or storage device that imitates a model of a neuron.

[0071] The Fig. The neuron shown in Figure 11 gives in response to inputs x (inputs x1 to x3 are in Fig. 11 as an example) produces a result y. The individual inputs x (x1, x2, and x3) are each multiplied by the weights w (w1, w2, and w3). The relationship between the inputs x and the result y can be expressed by the following equation. Equation 1 y=fk(∑i=1nxiwi−θ)

[0072] The inputs x, the output y, and the weights w are all vectors. In Equation 1, θ is the bias and f k the activation function.

[0073] In the Fig. 12, the inputs x (inputs x1 to x3 are in Fig. 12 as an example) is entered on the left side and the results y (the results y1 to y3 are in Fig. 12 as an example) is displayed on the right side. Fig. In the example shown in Figure 12, each of the inputs x1, x2 and x3 is multiplied by a corresponding weight (collectively denoted by W1) and all of the individual inputs x1, x2 and x3 are input to three neurons N11, N12 and N13.

[0074] In Fig. 12, the outputs of each of the neurons N11 to N13 are collectively denoted by Z1. Z1 can be regarded as a feature vector obtained by extracting a feature magnitude from an input vector. Fig. In the example shown in Figure 12, each feature vector Z1 is multiplied by a corresponding weight (collectively denoted by W2), and all of the individual feature vectors Z1 are input to two neurons N21 and N22. The feature vectors Z1 represent features between the weight W1 and the weight W2.

[0075] In Fig. 12, the outputs of each of the neurons N21 and N22 are collectively denoted by Z2. Z2 can be regarded as a feature vector obtained by extracting a feature magnitude of the feature vector Z1. Fig. In the example shown in Figure 12, each feature vector Z2 is multiplied by a corresponding weight (collectively denoted by W3) and all of the individual feature vectors Z2 are input to three neurons N31, N32 and N33.

[0076] The feature vectors Z2 represent features between the weight W2 and the weight W3. Finally, the neurons N31 to N33 output the results y1 to y3, respectively. The machine learning device 10 can learn the learning model LM using the learning data set DS as input by performing a multi-layer structure calculation according to the aforementioned neural network.

[0077] The configuration of the machine learning device 10 can be described as a machine learning process (or software) executed by a processor of a computer. In the machine learning process, the processor acquires the image data D i of the cable 22 imaged by the camera 50 while the robot 20 performs the specified operation, and the data D s , which indicate the state S of the cable 22 during the execution of the specified operation by the robot 20, as a learning data set DS and generates the learning model LM, which determines the correlation between the image data D i and the state S of the cable 22, using the learning data set DS.

[0078] Next, with reference to Fig. 13, a robot system 60 according to one embodiment is described. The robot system 60 includes the robot 20, the camera 50, and a controller 62. The controller 62 includes a processor 64 and a memory 66. The processor 64 and the memory 66 are communicatively connected to each other via a bus 68. The processor 64 includes, for example, a CPU (central processing unit) or a GPU (graphics processor), performs various computations, and simultaneously communicates with the memory 66. The controller 62 may be implemented as a computer that includes an input device (e.g., a keyboard, a mouse, or a touchscreen) and a display device (e.g., an LCD or organic EL display unit).

[0079] In this embodiment, the machine learning device 10 is installed in the control device 12 in the form of hardware or software, and the processor 64 performs various computations to execute the function of the machine learning device 10. Thus, in this embodiment, the processor 64 functions as the learning data acquisition section 12 and the learning section 14.

[0080] The memory 66 includes, for example, a ROM and a RAM, and prestores the learning model LM learned by the machine learning device 10. The robot system 60 according to this embodiment is installed, for example, in a factory for manufacturing the robot system 60 or a factory in which the robot 20 performs an actual work process (e.g., welding, workpiece handling, machining, or painting).

[0081] The operation of the robot system 60 is described below with reference to Fig. 14. The Fig. The process shown in Figure 14 is started when the processor 64 receives an operation start command from an operator, a host controller, or a robot program. In step S11, the processor 64 causes the robot 20 to perform the specified operation.

[0082] As an example of the predetermined operation, the processor 64 moves each movable element (the rotating body 26, the forearm 28, the upper arm 30, and the wrist 32) according to a predetermined robot program to cause the robot 20 to perform a series of actions to cause the robot 20 to perform a work operation (e.g., welding, workpiece handling, machining, or painting). Alternatively, the processor 64 causes the robot 20 to rotate the first element (wrist 32) relative to the second element (upper arm 30) of the robot 20 between the predetermined first position and the predetermined second position as a predetermined operation.

[0083] The predetermined operation performed by the robot 20 in step S11 is substantially the same as that performed for learning by the robot 20 in the aforementioned step S1. For example, in this step S11, the processor 64 may operate the robot 20 according to the same robot program that was used to operate the robot 20 in the aforementioned step S1.

[0084] However, the predetermined operation performed in step S11 is not limited to an operation under completely the same conditions (e.g., type of end effector, position and orientation of the robot 20, and trajectory of the robot 20) as in step S1, and may be an operation in which at least one condition is different from those in step S1, as long as the degree of bending of the cable 22 is the same.

[0085] In step S12, the processor 64 causes the camera 50 to image the cable 22 while the robot 20 performs the specified operation. In this step S12, the processor 64 images the same position of the cable 22 along the same line of sight direction VD at the same time as in the aforementioned step S1, in which the camera 50 imaged the cable 22.

[0086] As an example, if the camera 50 has imaged the cable 22 only once at time τ1 in the aforementioned step S1, the processor 64 causes the camera 50 to image the cable 22 along the line of sight direction VD at the position near the junction between the wrist 32 and the upper arm 30 at time τ1 in step S12, at which the wrist 32 is arranged in the first reference position relative to the upper arm 30 during the execution of step S11.

[0087] As another example, in step S12, when the camera 50 detects the cable 22 in the above-mentioned step S1 at the times τ n (n = 1, 2, 3, ...) continuously imaged, the camera 50, the cable 22 along the line of sight direction VD at the position near the connection between the wrist 32 and the upper arm 30 at each time τ n to which the wrist 32 is arranged in the n-th reference position relative to the upper arm 30 during the execution of step S11. The camera 50 feeds the recorded image data D i ' of the cable 22 to the processor 64.

[0088] In step S13, the processor 64 determines the state S of the cable 22 based on the learning model LM. In particular, the processor 64 outputs the image data D acquired in step S12 i ' of the cable 22 into the learning model LM stored in the memory 66. Then, the learning model LM estimates with the input image data Di ' correlated state data D s (Label information, such as "normal state" or "abnormal state") and outputs it. Thus, the processor 64 can determine the state S of the cable 22 based on the acquired image data D i ' determine.

[0089] If the state S of the cable 22 is not abnormal (or normal), the processor 64 determines NO and completes the operation started in step S11 to perform the Fig. 14. On the other hand, if the state S of the cable 22 is abnormal (or not normal), the processor 64 determines YES and proceeds to step S14. Thus, in this embodiment, the processor 64 functions as the state determining section 70 ( Fig. 13) which is configured to determine the state S of the cable 22.

[0090] In step S14, the processor 64 generates a warning signal. The processor 64 generates, for example, an audio or visual signal indicating that "an abnormality may occur in the condition of the cable" and outputs it via a speaker or a display device (not shown) provided in the control device 62. The processor 64 then terminates the Fig. 14 shown process.

[0091] It is pointed out that if in the Fig. 10, the data Dt of the type of the cable 22 or the robot 20 are further acquired as a learning data set DS for learning the learning model LM, the operator before step S13 by operating the input device provided in the control device 62 data D t ' of the type of robot 20 or cable 22 of robot system 60.

[0092] In this case, the processor 64 of the robot system 60 outputs the input data D in step S13 t ' of the type and the image data D acquired in step S12 i ' of the cable 22 into the learning model LM. Then the learning model LM estimates the state data D s based on the image data D i ' and the data D t ' of the type and determines the state S of the cable 22. According to this configuration, the state S of the cable 22 can be determined more accurately so that it corresponds to the data D t ' of the type.

[0093] Next, with reference to Fig. 15, a robot system 80 according to another embodiment is described. The robot system 80 differs from the aforementioned robot system 60 in that it further includes a sensor 82. As an example, the sensor 82, when in the Fig. 10 also the position data D pas a learning data set DS for learning the learning model LM, e.g., an encoder or a Hall element installed in each servo motor of the robot 20 of the robot system 80, and detects the rotation angle θ' of the servo motor as a detection value.

[0094] As another example, the sensor 82, when in the Fig. 10, the ambient temperature T is further detected as a learning data set DS for learning the learning model LM, e.g., a temperature sensor and detects an ambient temperature T' in the vicinity of the robot 20 of the robot system 80 as a detection value. As another example, the sensor 82, when in the Fig. 10, the electric current A or the resistance R of the transmission line 38 is further acquired as a learning data set DS for learning the learning model LM, e.g., an ammeter or an ohmmeter, and detects an electric current A' or a resistance R' of the transmission line 38 of the cable 22 of the robot system 80 as a detection value.

[0095] As another example, the sensor 82, when in the Fig. 10, the pressure P of the fluid in the feed pipe 36 is further detected as a learning data set DS for learning the learning model LM, e.g. a manometer and detects a pressure P' of the fluid in the feed pipe 36 of the cable 22 of the robot system 80 as a detection value.

[0096] As another example, the sensor 82, when in the Fig. 10, the optical characteristic OP is further acquired as a learning data set DS for learning the learning model LM, e.g., an optical power meter or beam profiler, and detects an optical characteristic OP' of light propagating through the optical fiber 42 of the cable 22 of the robot system 80 as a detection value.

[0097] The operation of the robot system 80 is described below with reference to Fig. 16. It should be noted that in the Fig. 16 shown sequence of processes that correspond to the Fig. 14, the same step numbers are assigned, and a repeated description thereof is omitted. In step S21, the processor 64 of the robot system 80 causes the camera 50 to image the cable 22, while the robot 20 performs the predetermined operation in step S11. In this step S21, the processor 64 causes the camera 50 to image the same position of the cable 22 along the same line of sight direction VD as in the aforementioned step S1 in which the camera 50 imaged the cable 22.

[0098] As an example, the processor 64 causes, if in the Fig. 10 also the position data D p to learn the learning model LM, the camera 50 to image the cable 22 at any time while the robot 20 performs the specified operation in step S11. However, if in the Fig. 10 shown learning cycle no position data D p are detected, the processor 64 causes the camera 50 to image the cable 22 at the same time as in step S1 of the learning phase, in which the camera 50 imaged the cable 22 similarly to the above-mentioned step S12.

[0099] In step S22, the processor 64 of the robot system 80 operates the sensor 82 to acquire the detection value, while the robot 20 performs the predetermined operation in step S11. For example, the sensor 82 detects when Fig. 10 also the position data D p for learning the learning model LM, at the time the camera 50 images the cable 22 in step S21, the rotation angle θ' as a recognition value. The processor 64 acquires position data D p ' of the robot 20 of the robot system 80 based on the angle of rotation θ'.

[0100] As another example, the sensor 82 detects when in the Fig. 10, the ambient temperature T is further detected for learning the learning model LM, during (or before or after) the execution of step S11, the ambient temperature T' is used as a detection value. As another example, the sensor 82 detects when in the Fig. 10, the electric current A or the resistance R is further detected for learning the learning model LM, during the execution of step S11 (e.g., at the time when the camera 50 images the cable 22 in step S21), the current A' or the resistance R' is detected as the detection value.

[0101] As another example, the sensor 82 detects when in the Fig. 10, the pressure P for learning the learning model LM is further detected, during the execution of step S11 (e.g., at the time when the camera 50 images the cable 22 in step S12), the pressure P' is detected as a detection value. As a further example, the sensor 82 detects when in the Fig. 10, the optical characteristic OP is also recorded for learning the learning model LM, the optical characteristic OP' as the recognition value.

[0102] In step S23, the processor 64 of the robot system 80 determines the state S of the cable 22 based on the learning model LM. As an example, the processor 64 specifies that if in the above-mentioned step S22 the position data D p ' are acquired, the image data D acquired in step S21 i ' of the cable 22 and the position data D acquired in step S22 p ' into the learning model LM stored in memory 66.

[0103] Then the learning model LM estimates the state data D s based on the image data D i ' and the position data D p ' and determines the state S of the cable 22. Thus, if the position data D p as a learning data set DS for learning the learning model LM, by inputting image data D displayed at the arbitrary time in step S21 i ' into the learning model LM, the state S of the cable 22 can be determined. In addition, the state S of the cable 22 can be determined more precisely so that it corresponds to the position and orientation of the robot 20.

[0104] As another example, when the detection value of the ambient temperature T', the current A', the resistance R', the pressure P' or the optical characteristic OP' is detected in the above-mentioned step S22, the processor 64 outputs the detection value T', A', R', P' or OP' detected in step S22 together with the image data D detected in step S21 i' of cable 22 into the learning model LM.

[0105] Then the learning model LM estimates the state data D s based on the image data D i ' and the detection value T', A', R', P' or OP' and determines the state S of the cable 22. According to this configuration, the state S of the cable 22 can be more accurately determined to correspond to the ambient temperature T', the current A', the resistance R', the pressure P' or the optical characteristic OP'.

[0106] As described above, in the robot systems 60 and 80, the processor 64 determines the state S of the cable 22 based on the learning model LM learned by the machine learning device 10 while the robot 20 performs the specified operation. According to this configuration, it is possible to automatically and accurately determine whether an abnormality has occurred in the state S of the cable 22 at a workplace such as a factory.

[0107] It should be noted that in the above-mentioned step S23, in addition to determining the state S of the cable 22 by means of the learning model LM, the processor 64 may further determine whether the current A' or the resistance R', the pressure P' or the optical characteristic OP' detected by the sensor 82 is within a predetermined permissible range.

[0108] If the current A' or the resistance R', the pressure P', or the optical characteristic OP' is outside the allowable range, the processor 64 may determine YES in step S23 and proceed to step S14 even if the learning model LM determines that the state S of the cable 22 is not abnormal (or normal). According to this configuration, in addition to determining the state S by the learning model LM, the processor 64 can more reliably detect an abnormality of the cable 22 based on the detection value, such as the current A' or the resistance R', the pressure P', or the optical characteristic OP'.

[0109] Next, with reference to Fig.17, a network system 90 is described according to one embodiment. The network system 90 includes a plurality of machine learning devices 10A, 10B, 10C, and 10D, a storage device 94, and a network 92 communicatively connecting the machine learning devices 10A, 10B, 10C, and 10D to the storage device 92.

[0110] Each of the machine learning devices 10A, 10B, 10C, and 10D has a similar configuration to the aforementioned machine learning device 10. The machine learning devices 10A, 10B, 10C, and 10D may be installed in different factories or on different production lines within the same factory. The storage device 94 includes, for example, a server and can store data. The network 92 is, for example, a LAN or the Internet and bidirectionally communicates data between the storage device 94 and each of the machine learning devices 10A, 10B, 10C, and 10D.

[0111] Each of the machine learning devices 10A, 10B, 10C, and 10D sends the learning data set DS acquired by the learning data acquisition section 12 to the storage device 94 via the network 92. The storage device 94 stores the learning data set DS received from each of the machine learning devices 10A, 10B, 10C, and 10D.

[0112] In addition, each of the machine learning devices 10A, 10B, 10C, and 10D acquires the learning data set DS stored in the storage device 94 via the network 92. The learning section 14 of each of the machine learning devices 10A, 10B, 10C, and 10D learns the learning model LM using the learning data set DS acquired by the storage device 94.

[0113] According to this embodiment, the learning data sets DS acquired by the plurality of machine learning devices 10A, 10B, 10C, and 10D are collected in the storage device 94 to enable sharing of the learning data sets DS among the machine learning devices 10A, 10B, 10C, and 10D. According to this configuration, the machine learning devices 10A, 10B, 10C, and 10D can promote the learning of the learning model LM using a larger number of learning data sets DS, thereby efficiently promoting the learning of the learning model LM.

[0114] It is noted that the learning data acquisition section 12 further acquires at least two (e.g., all) of the sets of position data D p , the data D tof the type, the ambient temperature T, the electric current A or resistance R, the pressure P and the optical characteristic OP as a learning data set DS. The learning section 14 can be achieved by adding the relationships between the image data D i and at least two sets of position data D p , the data concerning the type D t , the ambient temperature T, the current A or the resistance R, the pressure P and the optical characteristic OP for the correlation between the image data D i and the status data D s create the learning model LM.

[0115] Furthermore, in the above-mentioned step S2, the operator can detect the electric current A or the resistance R, the pressure P or the optical characteristic OP using the sensor, confirm the state S of the cable 22 based on the electric current A or the resistance R, the pressure P or the optical characteristic OP and output the state data D s into the learning data acquisition section 12. In this case, the machine learning device 10 may further include a recognition value determination section configured to determine whether the electric current A, the resistance R, the pressure P, or the optical characteristic OP is within a predetermined allowable range.

[0116] Furthermore, if the camera 50 in the above-mentioned step S1 continuously images the cable 22 over a predetermined period of time Δ (ie, records a moving video) and a group of image data D i_1 , D i_2, ... D i_n (“n” is a positive integer), the operator can in step S2 of the recorded group image data D i_1 to D i_n (ie the video during the time period Δ) a single element of the state data D s (ie, a single label information) and enter it into the learning data acquisition section 12.

[0117] In this case, the learning data acquisition section 12 acquires the learning data set DS of the group image data D i_1 to D i_n and one of the group image data D i_1 to D i_n assigned element of the state data D s . Then, the learning section 14 generates the learning model LM, which calculates the correlation between the group of image data D i_1 to D i_n and the state S of the cable 22, using the learning data set D s the group Image Data D i_1 to D i_n and the status data D s .

[0118] Then, in the above-mentioned step S12 or S21, the camera 50 continuously images (records a video) the cable 22 at the same time as in step S1 for the period Δ and acquires a group of image data D i_1 ', D i_2 ', ... D i_n ' (“n” is a positive integer). Then, in the above-mentioned step S13 or S23, the processor 64 outputs the group of image data D i_1 ' to D i_n ' into the learning model LM and the learning model LM estimates the group of image data D i_1 ' to D i_n ' correlated state data D s and outputs them. According to this configuration, the state S of the cable 22 can be determined based on the video (the group of image data D i_1 ' to D i_n ') of the cable 22.

[0119] In the embodiments described above, the camera 50 is arranged to image the cable 22 at a position near the junction between the wrist 32 and the upper arm 30. However, the camera 50 may also be arranged to image any position of the cable 22 where the cable 22 is bent while the robot 20 performs the specified operation. Furthermore, multiple cameras 50 may be provided to image different positions of the cable 22.

[0120] The robot 20 is not limited to a vertical articulated robot, but can be any type of robot, such as a horizontal articulated robot, a parallel-link robot, or a loading robot, as long as the robot includes a movable member and a cable. The disclosure has been described above with reference to embodiments; however, the embodiments described above are not intended to limit the invention according to the scope of the claims.

Claims

[1] A machine learning device (10) configured to learn a state of a cable (22) provided on a robot (20), the device (10) comprising: a learning data acquisition section (12) configured to acquire image data of the cable (22) imaged by a camera (50) while the robot (20) performs a predetermined operation and data indicating a state of the cable (22) during the performance of the predetermined operation by the robot (20) as a learning data set, and a learning section (14) configured to generate a learning model using the learning data set, which model represents a correlation between the image data and the condition of the cable (22). [2] The machine learning device (10) according to claim 1, wherein the learning data acquiring section (12) further acquires data of a position and orientation of the robot (20) during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the position and orientation for correlation. [3] The machine learning device (10) according to claim 1 or 2, wherein the learning data acquiring section (12) further acquires data of a type of the cable (22) or the robot (20) as a learning data set, the learning section (14) generating the learning model by adding a relationship between the image data and the type for correlation. [4] The machine learning device (10) according to any one of claims 1 to 3, wherein the learning data acquiring section (12) further acquires an ambient temperature during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the ambient temperature for correlation. [5] Machine learning device (10) according to one of claims 1 to 4, wherein the cable (22A, 22B, 22C) comprises a transmission line (38) configured to transmit an electrical signal, wherein the learning data acquisition section (12) further acquires an electric current flowing through the transmission line (38) or a resistance of the transmission line (38) as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the current or the resistance to the correlation. [6] Machine learning device (10) according to one of claims 1 to 5, wherein the cable (22A, 22B) comprises a supply tube (36) adapted to supply a fluid, wherein the learning data acquisition section (12) further acquires a pressure of the fluid during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the pressure to the correlation. [7] Machine learning device (10) according to one of claims 1 to 6, wherein the cable (22C) comprises a glass fiber (42) adapted to propagate light, wherein the learning data acquisition section (12) further acquires an optical characteristic of the light during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the optical characteristic to the correlation. [8] Machine learning device (10) according to one of claims 1 to 7, which is arranged to send the learning data set to a storage device (94) which is communicatively connected to the machine learning device (10) via a network (92) in order to store the learning data set in the storage device (94). [9] Robot system (60) comprising: a robot (20), the machine learning device (10) according to one of claims 1 to 8, the camera (50) configured to image the cable (22) while the robot (20) performs the predetermined operation and to supply a captured image to the learning data acquisition section (12) as image data, and a state determination section (70) configured to determine the state of the cable (22) while the robot (20) performs the predetermined operation based on the learning model generated by the learning section (14). [10] A machine learning method for learning a state of a cable (22) provided on a robot (20), the method comprising performing the following steps by a processor (64): Acquiring image data of the cable (22) imaged by a camera (50) while the robot (20) performs a predetermined operation and data indicating a state of the cable (22) while the robot (20) is performing the predetermined operation as a learning data set, and Generating a learning model representing a correlation between the image data and the condition of the cable (22) using the learning data set.

Images