Machine learning device, robot system and machine learning method for learning the state of a cable

Abstract

Machine learning device (10) configured to learn the state of a cable (22) provided to a robot (20), the device (10) comprising: a learning data acquisition section (12) configured to acquire as a learning data set image data of the cable (22) as imaged by a camera (50) while the robot (20) performs a predetermined operation, and data indicating a state of the cable (22) during the performance of the predetermined operation by the robot (20), and a learning section (14) configured to generate, using the learning data set, a learning model representing a correlation between the image data and the state of the cable (22).

Description

The present invention relates to a machine learning device, a robot system, and a machine learning method for learning the state of a cable attached to a robot.An apparatus for monitoring a cable attached to a robot is known (see, e.g., JP 2017-226 000 A).DE 10 2018 204 184 A1 describes a method for monitoring a supply system of a robot, which has a robot arm and a robot hand movable relative to it. The supply system has a supply line, a guide for the supply line and a number of sensors for monitoring at least one state variable of the supply system. On the basis of the values determined by the sensors, the operability of the supply system is deduced for the state variable.DE 10 2017 008 475 A1 discloses a machine learning device having a state observation unit that monitors, as a state variable, wobbling of an arm of a robot and / or a length of an operation trajectory of the arm of the robot. A determination data acquisition unit acquires, as determination data, a cycle time in which the robot performs processing. A learning unit learns an operation program of the robot on the basis of an output of the state observation unit and an output of the determination data acquisition unit.DE 10 2015 013 988 A1 discloses a robot simulation apparatus having a model generation unit which generates a three-dimensional wire-shaped element model from a plurality of mass points and a plurality of spring elements. A focal point setting unit sets a plurality of focal points on the wire-shaped element model. A simulation unit performs a simulation of the behavior of the wire-shaped element model associated with the operation of a robot, and a display unit displays focus point images together with images of the wire-shaped element model using position data of the plurality of mass points and position data of the plurality of focus points acquired via the simulation.US 2019 / 0 126 487 A1 describes a method for generating a dataset that maps visual features of a plurality of objects. The method instructs a robotic system to move an arm holding the respective object to a plurality of positions for each of a plurality of different objects. When the arm is in each of the plurality of positions, an image representing the respective object in the position is captured. Position information of the arm in the respective position is received and the image is analyzed to identify at least one visual feature of the object in the respective position. Finally, an association between the at least one visual feature and the position information is stored in an imaging data record and the imaging data record is output.From JP 2004-074 368 A it is known to detect a twisted state of a cable installed on a robot and to arrange it as a collection of points on a screen.JP H10-275 007 A discloses a robot operation simulation method that checks whether a non-rigid body holding a robot in the hand is wound or excessively stretched or contracted.There is a need for a technique for quantitatively and accurately evaluating the state of a cable located on a robot.To achieve the object, the invention provides a machine learning device having the features of claim 1, a robot system having the features of claim 9, and a machine learning method having the features of claim 10.According to an aspect of the disclosure, a machine learning device configured to learn a state of a cable provided to a robot includes a learning data acquisition section configured to acquire image data of the cable imaged by a camera while the robot performs a predetermined operation and data indicating a state of the cable during the performance of the predetermined operation by the robot as a learning data set, and a learning section configured to generate, using the learning data set, a learning model representing a correlation between the image data and the state of the cable.According to another aspect of the disclosure, a machine learning method for learning a state of a cable provided to a robot includes performing, by a processor, the following steps: acquiring, as a learning data set, image data of the cable imaged by a camera while the robot performs a predetermined operation and data indicating a state of the cable while the predetermined operation is performed by the robot, and generating a learning model representing a correlation between the image data and the state of the cable using the learning data set.According to the disclosure, a model representing a correlation between image data of a cable and the state of the cable can be easily and accurately obtained. FIG. 1 is a block diagram illustrating a machine learning device according to an embodiment; FIG. 2 is a perspective view showing a robot according to an embodiment; FIG. 3 is a sectional view showing a cable according to an embodiment; FIG. 4 is a sectional view showing a cable according to another embodiment; FIG. 5 is a sectional view showing a cable according to another embodiment; FIG. 6 is an enlarged view illustrating the connection between a wrist and an upper arm shown in FIG. 2 in an enlarged manner; FIG. 7 is a view for explaining a state in which the cable is bent while the robot is performing a predetermined operation; FIG. 8 is a view for explaining another state in which the cable is bent while the robot is performing the predetermined operation; FIG. 9 is a view for explaining another state in which the cable is bent while the robot is performing the predetermined operation; FIG. 10 is a flowchart showing an example learning cycle performed by the machine learning device shown in FIG. 1 ; FIG. 11 schematically shows a model of a neuron; FIG. 12 schematically shows a model of a multilayer neural network; FIG. 13 is a block diagram showing a robot system according to an embodiment; FIG. 14 is a flowchart showing an exemplary operation sequence of the robot system shown in FIG. 13 ; FIG. 15 is a block diagram showing a robot system according to another embodiment; FIG. 16 is a flowchart showing an exemplary operation sequence of the robot system shown in FIG. 15 ; and FIG. 17 is a block diagram illustrating a network system according to an embodiment.Embodiments of the disclosure will be described below in detail with reference to the drawings. Note that, in various embodiments to be described below, similar elements are assigned similar reference numerals, and repeated description thereof will be omitted. First, with reference to FIG. 1, a machine learning device 10 according to an embodiment will be described. The machine learning device 10 may be constituted by a computer including, for example, a processor (e.g., CPU, GPU), a memory (e.g., ROM, RAM), and an input device (e.g., a keyboard, a mouse, or a touch panel), or by software such as a learning algorithm executed by a computer.The machine learning device 10 is configured to learn a state (e.g., a normal state or an abnormal state) of a cable 22 provided to a robot 20 described later. Next, referring to FIG. 2, the robot 20 will be described. In this embodiment, the robot 20 is a vertical articulated robot, and includes a base 24, a rotating body 26, a forearm 28, an upper arm 30, a wrist 32, and a cable 22.The base 24 is fixed to the bottom of a work cell. The rotating body 26 is mounted on the base 24 so as to be rotatable about a vertical axis. The forearm 28 is mounted on the rotary body 26 so as to be rotatable about a horizontal axis. The upper arm 30 is rotatably mounted to a distal end of the forearm 28. The wrist 32 is rotatably mounted on a distal end of the upper arm 30.An end effector (e.g., a welding torch, a robot hand, a tool, or a paint applicator - not shown) for performing a predetermined operation (e.g., welding, workpiece handling, machining, or painting) is detachably attached to a distal end of the wrist 32. The robot 20 includes servomotors (not shown) for driving movable members (i.e., the rotating body 26, the forearm 28, the upper arm 30, and the wrist 32) of the robot 20.The cable 22 is laid on each movable member of the robot 20 using holding means 34. Each holding means 34 is fixed at a predetermined position on the movable member of the robot 20 and forms a hole for receiving the cable 22 therein. The cable 22 is inserted into the hole of each holding means 34 and is thereby held. The attachment positions of the holding means 34 to the robot 20 are set for each type of robot 20.Depending on the operation performed by the robot 20, various types of cables 22 are used. FIG. 3 is a sectional view of a cable 22A of a first type. The cable 22A includes a supply pipe 36, transmission lines 38, and a jacket 40. The supply pipe 36 supplies a fluid or a string used for the operation of the robot 20.As an example, when the welding torch is attached to the wrist 32 as an end effector and the robot 20 performs welding operation on a workpiece, the supply pipe 36 supplies an assist gas as a fluid to the welding torch. Alternatively, the supply pipe 36 accommodates therein a welding wire as a cable member and supplies it to the welding torch. As another example, when the tool is attached to the wrist 32 and the robot 20 cut a workpiece, the supply pipe 36 supplies a pressurized gas as a fluid to the tool.The transmission lines 38 each transmit an electrical signal. The transmission lines 38 transmit signals, for example, to the servomotors installed in the robot 20 or to the end effector attached to the wrist 32. In the cable 22A, the supply pipe 36 and the transmission lines 38 are arranged in the jacket 40 as desired.FIG. 4 shows a sectional view of a cable 22B of a second type. The cable 22B includes the supply pipe 36, a total of four transmission lines 38, and the jacket 40, wherein the transmission lines 38 are disposed on an outer circumferential side of the supply pipe 36 so as to be aligned at substantially equal intervals in the circumferential direction of the supply pipe 36. Thus, in the cable 22B, the feed tube 36, the transmission lines 38 and the jacket 40 are concentrically arranged.FIG. 5 is a sectional view of a cable 22C of a third type. The cable 22C includes a total of three transmission lines 38, an optical fiber 42, and the jacket 40. As an example, when the laser machining head is attached to the wrist 32 as an end effector and the robot 20 performs laser machining on a workpiece, a laser beam propagates through the optical fiber 42 to guide the laser beam to the laser machining head. In the cable 22C, the transmission lines 38 and the optical fiber 42 are arranged in the sheath 40 as desired.As described above, in the robot 20, a plurality of types of cables 22 (e.g., the cables 22A, 22B, and 22C) may be used. The types of the cables 22 are classified depending on, for example, the type of cable members to be used (the feed pipe 36, the transmission lines 38 and the glass fiber 42), the number of cable members, the arrangement of the cable members in the jacket 40, the material of the jacket 40, and the positions of binding wires for binding the cable members in the jacket 40 at predetermined intervals.Referring to FIG. 2, a camera 50 is provided for imaging the cable 22. The camera 50 images the cable 22 while the robot 20 performs a predetermined operation. The "predetermined operation" may include, for example, a series of operations of the robot 20 when the robot 20 performs an operation (e.g., welding, workpiece handling, cutting, or painting). In performing the predetermined operation, the robot 20 operates each movable element (the rotating body 26, the forearm 28, the upper arm 30, the wrist 32) according to a robot program generated in advance to perform the series of operations for the operation.Alternatively, the "predetermined operation" may include an operation of rotating a first member (i.e., the rotating body 26, the forearm 28, the upper arm 30, or the wrist 32) of the robot 20 with respect to a second member (the base 24, the rotating body 26, the forearm 28, or the upper arm 30) of the robot 20 between a predetermined first position and a predetermined second position.The camera 50 is arranged to image a predetermined position of the cable 22 along a predetermined line-of-sight direction while the robot 20 performs the predetermined operation. The camera 50 may be fixed to a position other than the robot 20 or to an element (the base 24, the rotating body 26, the forearm 28, or the upper arm 30) of the robot 20. In this embodiment, a case where the camera 50 is disposed to image the cable 22 at a position near the connection between the wrist 32 and the upper arm 30 will be described as an example.When the wrist 32 rotates with respect to the upper arm 30 while the robot 20 performs the predetermined operation, the cable 22 is bent at a position near the connection between the wrist 32 and the upper arm 30. FIGS. 6 to 9 schematically show bending states of the cable 22 during the predetermined operation. It is assumed that the wrist 32 rotates with respect to the upper arm 30 from a home position shown in FIG. 6 to a first reference position shown in FIGS. 7 to 9 during the predetermined operation.In the initial position shown in FIG. 6, an axis A 1 of the wrist 32 and an axis A 2 of the upper arm 30 are substantially parallel to each other. In the first reference position shown in FIGS. 7 to 9, on the other hand, the axis A 1 of the wrist 32 is inclined with respect to the axis A 2 of the upper arm 30 by an angle θ 1. FIGS. 7 through 9 show exemplary bending states of the cable 22 when the wrist 32 is disposed relative to the upper arm 30 in the first reference position during the predetermined operation. For comparison, the cable shown in FIG. 7 is laid in the form of a broken line over FIG. 8, and the cables shown in FIGS. 7 and 8 are laid in the form of a broken line and a dot-and-dash line over FIG. 9, respectively.For example, due to a failure of the holding positions of the cable 22 or a shape failure of a member of the robot 20, the bending degree of the cable 22 of the same type may be different when the wrist 32 is disposed relative to the upper arm 30 in the first reference position as shown in FIGS. 7 to 9.For example, as shown in FIGS. 7 and 8, it may be determined that the degree of bending of the cable 22 is in an allowable range and has no influence on the performance of the cables 22 (e.g., the signal transmission characteristics of the transmission line 28, the fluid flowability through the feed pipe 36, the light propagation characteristic of the optical fiber 42, or the bending resistance of the jacket 40). On the other hand, it may be determined that the degree of bending of the cable 22 shown in FIG. 9 is not allowed and has some influence on the performance of the cable 22.The machine learning device 10 learns the state (e.g., the normal state or abnormal state) of the cable 22 while the robot 20 performs the predetermined operation. Referring to FIG. 1, the machine learning device 10 includes a learning data acquisition section 12 and a learning section 14. the learning data acquisition section 12 acquires image data D i of the cable 22 imaged by the camera 50 while the robot 20 performs the predetermined operation and data indicating the state S of the cable 22 during the performance of the predetermined operation by the robot 20 (hereinafter referred to as state data D s'' ) as the learning data set DS.Specifically, the camera 50 images the cable 22 at a position near the connection between the wrist 32 and the upper arm 30 along the line-of-sight direction VD while the robot 20 performs the predetermined operation. The camera 50 supplies the captured image data D i to the learning data acquisition section 12. The state data D s of the cable 22 includes, for example, label information such as "normal state" or "abnormal state". The normal state means, for example, the state in which the bending of the cable 22 is within the allowable range as shown in FIGS. 7 and 8. By the abnormal state, on the other hand, it is meant, for example, the state in which the bending of the cable 22 is out of the allowable range as shown in FIG. 9.Note that the state data D s may include only the label information of the "normal state" or the "abnormal state", or both label information of the "normal state" and the "abnormal state". Further, a plurality of individual label information such as "new state of fire" and "used state" may be set as label information of "normal state". In addition, depending on the severity of the abnormal state, a plurality of individual label information such as "abnormal state level 1", "abnormal state level 2", and "abnormal state level 3" may be set as label information of the "abnormal state".The state data D s( i.e., the label information) of the cable 22 is determined by an operator. For example, the operator confirms the state S (the normal state or the abnormal state) of the cable 22 by visually observing the image data D i captured by the camera 50, and inputs the state data D s to the learning data acquisition section 12. The learning data acquisition section 12 acquires the image data D i acquired by the camera 50 and the state data D s determined by the operator as the learning data set DS.The learning section 14 generates, using the learning data set DS of the image data D i and the state data D s a learning model (function) LM representing a correlation between the image data D i and the state S of the cable 22. The learning section 14 generates the learning model LM by performing supervised learning, for example. In this case, the robot 20 repeatedly tries to perform the predetermined operation, and the learning data acquisition section 12 repeatedly acquires, every time the robot 20 performs the predetermined operation, the learning data set DS as teacher data.The learning section 14 learns the learning model LM by identifying features that imply the correlation between the image data D acquired as teacher data i( e.g., image data of the cables 22 shown in FIGS. 7 to 9 ) and state data D s( label information such as a normal state or an abnormal state). As such supervised learning, an algorithm such as a support vector machine (SVM) or a Gaussian mixture model (GMM) may be employed.The flow of the learning cycle executed by the machine learning device 10 will be described below with reference to FIG. 10. In step S 1, the robot 20 performs the predetermined operation, and the camera 50 images the cable 22 while the robot 20 performs the predetermined operation. As an example, the camera 50 images the cable 22 only once while the robot 20 is performing the predetermined operation at a time τ 1, at which the wrist 32 is disposed relative to the upper arm 30 at the first reference position shown in FIGS. 7 to 9.As another example, the camera 50 may continuously image the cable 22 (i.e., capture a moving image) while the robot 20 is performing the predetermined operation. In this case, the camera 50 images the cable 22 at each time τ n when the wrist 32 is disposed relative to the upper arm 30 at an n-th reference position (n=1, 2, 3,... ) in which the axis A 1 of the wrist 32 is inclined with respect to the axis A 2 of the upper arm 30 by a predetermined angle θ n.In step S 2, the learning data acquisition section 12 acquires the learning data set DS. Specifically, the learning data acquisition section 12 acquires, from the camera 50, image data D i of the cable 22 imaged by the camera 50 while the robot 20 performs the predetermined operation in step S 1.Further, the learning data acquisition section 12 acquires the label information of the normal state or the label information of the abnormal state as data indicating the state S (state data D s) of the cable 22 during the execution of step S 1. Specifically, the operator confirms whether the cable 22 is in the normal state or in the abnormal state by visually observing the image data D i of the cable 22 imaged by the camera 50.The operator then operates the input device of the machine learning device 10 to input the normal state label information or the abnormal state label information as the state data D s. The learning data acquisition section 12 acquires the input label information of the normal state and label information of the abnormal state. Thus, the learning data acquisition section 12 acquires the learning data set DS of the image data D i and the state data D s of the cable 22, and stores these in the memory in association with each other.In step S 3, the learning section 14 generates the learning model LM representing the correlation between the image data D i and the state S using the learning data set DS acquired in step S 2. Specifically, the learning section 14 learns the learning model LM by executing the supervised learning algorithm (e.g., SVM or GMM). Then, the flow returns to step S1.By executing such a learning cycle, the learning of the learning model LM is promoted and led to an optimum solution. The operator executes the learning cycle shown in FIG. 10 for a plurality of robots 20. This makes it possible to acquire a larger number of learning data sets DS and to promote learning of the learning model LM at a higher level again. According to this embodiment, it is possible to easily and accurately obtain a model quantitatively representing the correlation between the image data D i and the state S of the cable 22.As another function of the machine learning device 10, in step S 2, the learning data acquisition section 12 may further acquire data of a type (a first type, a second type,... of an n-th type) of the cable 22 or data of a type of the robot 20 as the learning data set DS. The cable 22 type data includes, for example, information for identifying the type of the cable 22, information for identifying the type of the cable elements used (the feed pipe 36, the transmission lines 38, and the optical fiber 42), the number of cable elements, information indicating the arrangement of the cable elements in the jacket 40, the material of the jacket 40, and information on the positions of the binding wires for binding the cable elements at predetermined intervals in the jacket 40.The information for identifying the type of the cable 22 may include, for example, sign information such as "nthtype" or a product number assigned to each cable by a cable manufacturer. The information for identifying the type of the cable elements may include, for example, sign information such as "feed pipe", "transmission line", or "optical fiber".The information indicating the arrangement of the cable elements may include, for example, character information such as "arbitrary arrangement" or "concentric arrangement". The material of the jacket 40 may be indicated by, for example, sign information such as "resin" or "rubber", or a product number assigned by a manufacturer of the material of the jacket 40. The information on the positions of the binding wires can comprise, for example, the average distance between two binding wires or the number of binding wires.The data of the type of the robot 20 includes, for example, information for identifying the type of the robot 20, the number of the wired lines 22, and information on the attachment positions of the holding means 34 to the robot 20.As an example, in step S 2, by operating the input device of the machine learning device 10, the operator inputs data D t of the type of the cable 22 or the robot 20. Thus, the learning data acquisition section 12 acquires, in addition to the image data D i and the state data D s the data D t of the type as the learning data set DS. In step S 3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the data D t of the type for correlation between the image data D i and the state data D s.The degree of bending of the cable 22 during performance of the predetermined operation by the robot 20 may vary depending on the type of the cable 22. For example, the greater the number of cable elements laid in jacket 40, or the less the flexibility of the material of jacket 40, the more difficult cable 22 can bend.In addition, the degree of bending of the cable 22 during execution of the predetermined operation by the robot 20 may vary depending on the type of the robot 20. For example, the attachment positions of the support means 34 vary for each type of robot 20, whereby the degree of bending of the cable 22 may vary during the given operation.Thus, there may be a correlation between the degree of bending of the cable 22 and the type of the cable 22 or the robot 20. Therefore, generating the learning model LM in consideration of the relationship between the image data D i and the type of the cable 22 or the robot 20 makes it possible to obtain the learning model LM matched to the type of the cable 22 or the robot 20.As another function of the machine learning device 10, the learning data acquisition section 12 may further acquire, as the learning data DS, data of the position and orientation of the robot 20 (hereinafter referred to as "position data D p") while the robot 20 performs the predetermined operation in step S 2, in addition to the image data D i and the state data D s in step S 1.As an example, when the camera 50 images the cable 22 only once at the time τ 1 in step S 1, the learning data acquisition section 12 acquires position data D p of the robot 20 at the time τ 1. As another example, when the camera 50 continuously images (captures a moving image thereof) the cable 22 at the timings τ n( n=1, 2, 3,... ), the learning data acquisition section 12 acquires position data D p of the robot 20 at each timing τ n. in sequence in step S 1.The position data D p of the robot 20 can be obtained from the rotation angle θ of each servomotor installed in the robot 20. In this case, each servomotor of the robot 20 is equipped with a sensor (e.g., an encoder (rotary encoder) or a Hall element) that detects the rotation angle θ during imaging by the camera 50. As position data D p of the robot 20, the rotation angle θ may be used. In step S 3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the position and orientation of the robot 20 to the correlation between the image data D i and the state data D s.The degree of bending of the cable 22 during performance of the predetermined operation by the robot 20 may vary depending on the position and orientation of the robot 20 (e.g., the angle of the wrist 32 with respect to the upper arm 30). In other words, a correlation may exist between the degree of bending of the cable 22 and the position and orientation of the robot 20. Therefore, generating the learning model LM in consideration of the relationship between the image data D i and the position and orientation of the robot 20 makes it possible to obtain the learning model LM matched to the position and orientation of the robot 20.As another function of the machine learning device 10, the learning data acquisition section 12 may further acquire, as the learning data set DS, an ambient temperature T during execution of the predetermined operation by the robot 20 in step S 1. In this case, a sensor (temperature sensor) that detects the ambient temperature T is provided.In step S 3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the ambient temperature T to the correlation between the image data D i and the state data D s. There may be a correlation between the degree of bending of the cable 22 and the ambient temperature T during execution of the predetermined operation by the robot 20.In particular, depending on the material of the jacket 40, the higher the ambient temperature T and the higher the flexibility of the jacket 40, the easier the cable 22 can be bent. On the other hand, the lower the ambient temperature T and the lower the flexibility of the jacket 40, the more difficult the cable 22 can bend. Therefore, generating the learning model LM in consideration of the relationship between the image data D i and the ambient temperature T makes it possible to obtain the learning model LM matched to the ambient temperature T.As another function of the machine learning device 10, the learning data acquisition section 12 may further acquire, in step S 2, an electric current A flowing through the transmission line 38 or a resistance R of the transmission line 38 during execution of the predetermined operation by the robot 20 as the learning data set DS in step S 1. In this case, a sensor (e.g., an ammeter or an ohmmeter) is provided that detects the electric current A or the resistance R of the transmission line 38 during the predetermined operation.In step S 3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the current A or the resistance R to correlate the image data D i and the state data D s. The electric current A or the resistance R may vary depending on the degree of bending of the cable 22 during the execution of the predetermined operation by the robot 20. For example, when the degree of bending of the cable 22 is outside the allowable range, the transmission line 38 may partially break and the resistance R of the transmission line 38 may increase and, at the same time, the electric current A of the transmission line 38 may decrease.Thus, there may be a correlation between the degree of bending of the cable 22 and the electric current A or the resistance R. Therefore, generating the learning model LM in consideration of the relationship between the image data D i and the current A or the resistance R makes it possible to obtain the learning model LM matched to the current A or the resistance R.As another function of the machine learning device 10, when the cable 22A shown in FIG. 3 or the cable 22B shown in FIG. 4 is used, the learning data acquisition section 12 may further acquire a pressure P of the fluid in the supply pipe 36 during the execution of the predetermined operation by the robot 20 as the learning data set DS in step S 2. In this case, a sensor (e.g., a manometer) is provided that detects the pressure P during the predetermined operation.In step S 3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the pressure P to the correlation between the image data D i and the state data D s. The degree of bending of the cable 22 during execution of the predetermined operation by the robot 20 may vary depending on the pressure P of the fluid in the supply pipe 36.In particular, the higher the pressure P of the fluid in the supply pipe 36, the more difficult the cable 22 can bend. Thus, there may be a correlation between the degree of bending of the cable 22 and the pressure P. Therefore, generating the learning model LM in consideration of the relationship between the image data D i and the pressure P makes it possible to obtain the learning model LM matched with the pressure P.As another function of the machine learning device 10, when the cable 22C shown in FIG. 5 is used, the learning data acquisition section 12 may further acquire, as the learning data set DS, an optical characteristic OP of light (e.g., a laser beam) propagating through the optical fiber 42 while the robot 20 performs the predetermined operation in step S 2. The optical characteristic OP may include, for example, the intensity, power, or intensity distribution of the light propagating through the optical fiber 42 (or return light of the light). In this case, a sensor (e.g. an optical power meter or a beam profile meter) is provided which detects the optical characteristic OP during the predefined operation.In step S 3, the learning section 14 generates the learning model LM by adding the relationship between the image data D i and the optical characteristic OP to the correlation between the image data D i and the state data D s. The optical characteristic OP may vary depending on the degree of bending of the cable 22 during performance of the predetermined operation by the robot 20. For example, if the degree of bending of the cable 22 is outside the allowable range, stray light may increase in the bent portion of the cable 22, whereby the intensity or power of the light (or return light) may decrease.Thus, there may be a correlation between the degree of bending of the cable 22 and the optical characteristic OP. Therefore, generating the learning model LM in consideration of the relationship between the image data D i and the optical characteristic OP makes it possible to obtain the learning model LM matched with the optical characteristic OP.The learning algorithm executed by the learning section 14 is not limited to supervised learning, but a known learning algorithm such as unsupervised learning and reinforcement learning, or a neural network for machine learning may also be employed. As an example, FIG. 11 schematically shows a model of a neuron. FIG. 12 schematically shows a model of a three-layer neural network formed by combining neurons shown in FIG. 11. The neural network can be implemented, for example, as a computing device or a storage device that imitates a model of a neuron.The neuron shown in FIG. 11 outputs a result y in response to inputs x (the inputs x1 to x3 are defined as an example in FIG. 11). The individual inputs x (x1, x2 and x3) are multiplied by the weights w (w1, w2 and w3), respectively. The relationship between the inputs x and the result y can be expressed by the following equation. Equation 1The inputs x, the result y and the weights w are all vectors. In Equation 1, θ is the systematic measurement deviation and f k is the activation function.In the three-layer neural network shown in FIG. 12, the inputs x (the inputs x 1 to x 3 are defined as an example in FIG. 12 ) are input on the left side, and the results y (the results y 1 to y 3 are defined as an example in FIG. 12 ) are output on the right side. In the example shown in FIG. 12, each of the inputs x 1, x 2, and x 3 is multiplied by a corresponding weight (collectively referred to as W 1) and all of the individual inputs x 1, x 2, and x 3 are input to three neurons N 11, N 12, and N 13.In FIG. 12, the outputs of each of the neurons N 11 to N 13 are collectively denoted by Z 1. Z 1 may be regarded as a feature vector obtained by extracting a feature amount of an input vector. In the example shown in FIG. 12, each feature vector Z 1 is multiplied by a corresponding weight (collectively referred to as W 2) and all of the individual feature vectors Z 1 are input to two neurons N 21 and N 22. The feature vectors Z 1 represent features between the weight W 1 and the weight W 2.In FIG. 12, the outputs of each of the neurons N 21 and N 22 are collectively denoted by Z 2. Z 2 may be regarded as a feature vector obtained by extracting a feature amount of the feature vector Z 1. In the example shown in FIG. 12, each feature vector Z 2 is multiplied by a corresponding weight (collectively referred to as W 3) and all of the individual feature vectors Z 2 are input to three neurons N 31, N 32, and N 33.The feature vectors Z 2 represent features between the weight W 2 and the weight W 3. Finally, the neurons N 31 to N 33 output the results y 1 to y 3, respectively. The machine learning device 10 can learn the learning model LM using the learning dataset DS as an input by performing a multi-layer structure calculation according to the above-mentioned neural network.The configuration of the machine learning device 10 can be described as a machine learning method (or software) executed by a processor of a computer. In the machine learning method, the processor acquires the image data D i of the cable 22 imaged by the camera 50 while the robot 20 performs the predetermined operation and the data D s, which indicates the state S of the cable 22 during the performance of the predetermined operation by the robot 20, as the learning data set DS, and generates the learning model LM representing the correlation between the image data D i and the state S of the cable 22 using the learning data set DS.Next, referring to FIG. 13, a robot system 60 according to an embodiment will be described. The robot system 60 includes the robot 20, the camera 50, and a controller 62, The controller 62 includes a processor 64 and a memory 66, and the processor 64 and the memory 66 are communicatively connected to each other via a bus 68. The processor 64 includes, for example, a CPU (central processing unit) or a GPU (graphics processor), performs various computations, and simultaneously communicates with the memory 66. the controller 62 may be implemented as a computer including an input device (e.g., a keyboard, a mouse, or a touch panel) and a display device (e.g., an LCD or organic EL display unit).In this embodiment, the machine learning device 10 is installed in the controller 12 in the form of hardware or software, and the processor 64 performs various computations for executing the function of the machine learning device 10. Thus, the processor 64 functions as the learning data acquisition section 12 and the learning section 14 in this embodiment.The memory 66 includes, for example, a ROM and a RAM, and prestores the learning model LM learned by the machine learning device 10. The robot system 60 according to this embodiment is installed, for example, in a factory for manufacturing the robot system 60 or a factory in which the robot 20 performs an actual operation (e.g., welding, workpiece handling, cutting, or painting).The operation of the robot system 60 will be described below with reference to FIG. 14. The flow shown in FIG. 14 is started when the processor 64 receives an operation start command from an operator, a host controller, or a robot program. In step S 11, the processor 64 causes the robot 20 to perform the predetermined operation.As an example of the predetermined operation, the processor 64 moves each movable member (the rotating body 26, the forearm 28, the upper arm 30, and the wrist 32) according to a predetermined robot program to cause the robot 20 to perform a series of operations to cause the robot 20 to perform an operation (e.g., welding, workpiece handling, cutting, or painting). Alternatively, the processor 64 causes the robot 20 to rotate, as a predetermined operation, the first member (wrist 32) with respect to the second member (upper arm 30) of the robot 20 between the predetermined first position and the predetermined second position.The predetermined operation performed by the robot 20 in step S 11 is substantially the same as that performed by the robot 20 for learning in the above-mentioned step S 1. For example, in this step S 11, the processor 64 may operate the robot 20 according to the same robot program used to operate the robot 20 in the above-mentioned step S 1.However, the predetermined operation performed in step S 11 is not limited to an operation under completely equal conditions (e.g., type of end effector, position and orientation of the robot 20, and trajectory of the robot 20) as in step S 1, and may be an operation in which at least one condition is different from those in step S 1 as long as the bending degree of the cable 22 is the same.In step S 12, the processor 64 causes the camera 50 to image the cable 22 while the robot 20 performs the predetermined operation. In this step S 12, the processor 64 images the same position of the cable 22 along the same line-of-sight direction VD at the same time as in the aforementioned step S 1 in which the camera 50 has imaged the cable 22.As an example, in step S 12, when the camera 50 has imaged the cable 22 only once at the time τ 1 in the above-mentioned step S 1, the processor 64 causes the camera 50 to image the cable 22 along the line-of-sight direction VD at the position near the connection between the wrist 32 and the upper arm 30 at the time τ 1 at which the wrist 32 is disposed at the first reference position relative to the upper arm 30 during the execution of step S 11.As another example, in step S 12, when the camera 50 has continuously imaged the cable 22 at the timings τ n( n=1, 2, 3,... ) in the above step S 1, the processor 64 causes the camera 50 to image the cable 22 along the line-of-sight direction VD at the position near the connection between the wrist 32 and the upper arm 30 at each timing τ n at which the wrist 32 is disposed at the n-th reference position relative to the upper arm 30 during the execution of step S 11. The camera 50 supplies the captured image data D i' of the cable 22 to the processor 64.In step S 13, the processor 64 determines the state S of the cable 22 based on the learning model LM. Specifically, the processor 64 inputs the image data D i' of the cable 22 acquired in step S 12 to the learning model LM stored in the memory 66. Then, the learning model LM estimates and outputs state data D i' correlated with the input image data D s( label information such as "normal state" or "abnormal state"). Thus, the processor 64 may determine the state S of the cable 22 from the captured image data D i'.When the state S of the cable 22 is not abnormal (or normal), the processor 64 determines NO and concludes the operation started in step S 11 to end the flow illustrated in FIG. 14. On the other hand, when the state S of the cable 22 is abnormal (or abnormal), the processor 64 determines YES and proceeds to step S 14. Thus, in this embodiment, the processor 64 functions as the state determination section 70 (FIG. 13 ) configured to determine the state S of the cable 22.In step S 14, the processor 64 generates a warning signal. The processor 64 generates, for example, an audio or image signal indicating that "abnormality may occur in the state of the cable" and outputs it via a speaker or a display device (not shown) provided in the controller 62. The processor 64 then terminates the flow shown in Fig. 14.Note that, in the learning cycle shown in FIG. 10, when the data D t of the type of the cable 22 or the robot 20 is acquired as the learning data set DS for learning the learning model LM, the operator may input data D t' of the type of the robot 20 or the cable 22 of the robot system 60 by operating the input device provided in the controller 62, before step S 13.In this case, the processor 64 of the robot system 60 inputs the input data D t' of the type and the image data D i' of the cable 22 acquired in step S 12 into the learning model LM in step S 13. Then, the learning model LM estimates the state data D s from the image data D i' and the type data D t' and determines the state S of the cable 22.Next, referring to FIG. 15, a robot system 80 according to another embodiment will be described. The robot system 80 is different from the aforementioned robot system 60 in that it further includes a sensor 82. As an example, when the position data D p is further acquired as the learning data set DS for learning the learning model LM, for example, an encoder or a Hall element installed in each servomotor of the robot 20 of the robot system 80, in the learning cycle shown in FIG. 10, the sensor 82 is a detection value and detects the rotation angle θ' of the servomotor as a detection value.As another example, when the ambient temperature T is further acquired as the learning data set DS for learning the learning model LM, for example, in the learning cycle shown in FIG. 10, the sensor 82 is a temperature sensor and detects an ambient temperature T' in the vicinity of the robot 20 of the robot system 80 as a detection value. As another example, when the electric current A or the resistance R of the transmission line 38 is further detected as the learning data set DS for learning the learning model LM, for example, an ammeter or an ohmmeter, in the learning cycle shown in FIG. 10, the sensor 82 is a detection value of an electric current A' or a resistance R' of the transmission line 38 of the cable 22 of the robot system 80.As another example, when the pressure P of the fluid in the supply pipe 36 is further acquired as the learning data set DS for learning the learning model LM, the sensor 82 is, for example, a manometer, and detects a pressure P' of the fluid in the supply pipe 36 of the cable 22 of the robot system 80 as a detection value in the learning cycle shown in FIG. 10.As another example, when the optical characteristic OP is further acquired as the learning data set DS for learning the learning model LM, e.g., an optical power meter or beam profile meter, in the learning cycle shown in FIG. 10, the sensor 82 detects an optical characteristic OP' of light propagating through the optical fiber 42 of the cable 22 of the robot system 80 as a detection value.The operation of the robot system 80 will be described below with reference to FIG. 16. Note that, in the flow shown in FIG. 16, processes similar to the flow shown in FIG. 14 are assigned the same step numbers, and repeated description thereof will be omitted. In step S 21, the processor 64 of the robot system 80 causes the camera 50 to image the cable 22 while the robot 20 performs the predetermined operation in step S 11. In this step S 21, the processor 64 causes the camera 50 to image the same position of the cable 22 along the same line-of-sight direction VD as in the aforementioned step S 1 in which the camera 50 has imaged the cable 22.As an example, when the position data D p for learning the learning model LM is further acquired in the learning cycle shown in FIG. 10, the processor 64 causes the camera 50 to image the cable 22 at an arbitrary timing while the robot 20 performs the predetermined operation in step S 11. On the other hand, in the learning cycle shown in FIG. 10, when no position data D p is acquired, the processor 64 causes the camera 50 to image the cable 22 at the same timing as in step S 1 of the learning phase in which the camera 50 has imaged the cable 22 similarly to the above-mentioned step S 12.In step S 22, the processor 64 of the robot system 80 operates the sensor 82 to acquire the detection value while the robot 20 performs the predetermined operation in step S 11. As an example, when the position data D p for learning the learning model LM is further acquired in the learning cycle shown in FIG. 10, at the time when the camera 50 images the cable 22 in step S 21, the sensor 82 recognizes the rotation angle θ' as a recognition value. The processor 64 acquires position data D p' of the robot 20 of the robot system 80 from the rotation angle θ'.As another example, when the ambient temperature T for learning the learning model LM is further detected in the learning cycle shown in FIG. 10, the sensor 82 detects the ambient temperature T' as a detection value during (or before or after) the execution of step S 11. As another example, when the electric current A or the resistance R for learning the learning model LM is further detected in the learning cycle shown in FIG. 10, during the execution of step S 11 (e.g., at the time point at which the camera 50 images the cable 22 in step S 21), the sensor 82 detects the current A' or the resistance R' as a detection value.As another example, when the pressure P for learning the learning model LM is further detected in the learning cycle shown in FIG. 10, during the execution of step S 11 (e.g., at the time when the camera 50 images the cable 22 in step S 12), the sensor 82 detects the pressure P' as a detection value. As another example, when the optical characteristic OP for learning the learning model LM is further detected in the learning cycle shown in FIG. 10, the sensor 82 detects the optical characteristic OP' as a detection value.In step S 23, the processor 64 of the robot system 80 determines the state S of the cable 22 based on the learning model LM. As an example, when the position data D p' is acquired in the above-mentioned step S 22, the processor 64 inputs the image data D i' of the cable 22 acquired in step S 21, and the position data D p' acquired in step S 22 to the learning model LM stored in the memory 66.Then, the learning model LM estimates the state data D s from the image data D i' and the position data D p' and determines the state S of the cable 22, and thus, when the position data D p is acquired as the learning data set DS for learning the learning model LM, by inputting image data D i' mapped at the arbitrary timing in step S 21 into the learning model LM, the state S of the cable 22 can be determined. In addition, the state S of the cable 22 can be more accurately determined to correspond to the position and orientation of the robot 20.As another example, when the detection value of the ambient temperature T', the current A', the resistance R', the pressure P', or the optical characteristic OP' is detected in the above-mentioned step S22, the processor 64 inputs the detection value T', A', R', P', or OP' detected in the step S22 into the learning model LM together with the image data D i' of the cable 22 detected in the step S21.Then, the learning model LM estimates the state data D s from the image data D i' and the detection value T', A', R', P' or OP' and determines the state S of the cable 22.As described above, in the robot systems 60 and 80, the processor 64 determines the state S of the cable 22 while the robot 20 is performing the predetermined operation, based on the learning model LM learned by the machine learning device 10. According to this configuration, it is possible to automatically and accurately determine whether an abnormality has occurred in the state S of the cable 22 at a worksite such as a worksite.Note that in the above-mentioned step S23, in addition to determining the state S of the cable 22 by the learning model LM, the processor 64 may further determine whether the current A' or the resistance R', the pressure P' or the optical characteristic OP' detected by the sensor 82 is within a predetermined allowable range.If the current A' or the resistance R', the pressure P' or the optical characteristic OP' is out of the allowable range, the processor 64 may determine YES in step S23 and proceed to step S14 even if the learning model LM determines that the state S of the cable 22 is not abnormal (or normal). According to this configuration, in addition to determining the state S by the learning model LM from the detection value such as the current A' or the resistance R', the pressure P' or the optical characteristic OP', the processor 64 can more reliably detect an abnormality of the cable 22.Next, referring to FIG. 17, a network system 90 according to an embodiment will be described. The network system 90 includes a plurality of machine learning devices 10A, 10B, 10C, and 10D, a storage device 94, and a network 92 that communicably connects the machine learning devices 10A, 10B, 10C, and 10D to the storage device 92.Each of the machine learning devices 10A, 10B, 10C, and 10D has a similar configuration to the aforementioned machine learning device 10. The storage device 94 includes, for example, a server and can store data. The network 92 is, for example, a LAN or the Internet, and bidirectionally communicates data between the storage device 94 and each of the machine learning devices 10A, 10B, 10C, and 10D.Each of the machine learning devices 10A, 10B, 10C, and 10D transmits the learning data set DS acquired by the learning data acquisition section 12 to the storage device 94 via the network 92. the storage device 94 stores the learning data set DS received from each of the machine learning devices 10A, 10B, 10C, and 10D.In addition, each of the machine learning devices 10A, 10B, 10C, and 10D acquires the learning data set DS stored in the storage device 94 via the network 92.According to this embodiment, the learning data sets DS acquired by the plurality of machine learning devices 10A, 10B, 10C, and 10D are collected in the storage device 94 to enable sharing of the learning data sets DS by the machine learning devices 10A, 10B, 10C, and 10D. According to this configuration, the machine learning devices 10A, 10B, 10C, and 10D can promote learning of the learning model LM using a larger number of learning data sets DS, whereby learning of the learning model LM can be promoted efficiently.Note that the learning data acquisition section 12 may further acquire at least two (e.g., all) of the sets of the position data D p, the data D t of the type, the ambient temperature T, the electric current A or the resistance R, the pressure P, and the optical characteristic OP as the learning data set DS. The learning section 14 can generate the learning model LM by adding the relationships between the image data D i and at least two sets of the position data D p, the type-related data D t, the ambient temperature T, the current A or the resistance R, the pressure P, and the optical characteristic OP for correlation between the image data D i and the state data D s.Further, in the above-mentioned step S 2, the operator can recognize the electric current A or the resistance R, the pressure P, or the optical characteristic OP using the sensor, confirm the state S of the cable 22 based on the electric current A or the resistance R, the pressure P, or the optical characteristic OP, and input the state data D s to the learning data acquisition section 12. In this case, the machine learning device 10 may further include a detection value determination section configured to determine whether the electric current A or the resistance R, the pressure P, or the optical characteristic OP is within a predetermined allowable range.Further, when the camera 50 images (i.e., captures a moving video) the cable 22 continuously over a predetermined time period Δ and a group of image data D i_1, D i_2,... in the above step S 1. D i_n( "n" is a positive integer), the operator may assign a single item of the state data D s( i.e., label information) to the captured group of image data D i_1 to D i_n( i.e., the video during the period Δ) in step S 2, and input it to the learning data acquisition section 12.In this case, the learning data acquisition section 12 acquires the learning data set DS of the group of image data D i_1 to D i_n and an element of the state data D s. assigned to the group of image data D i_1 to D i_n. Then, the learning section 14 generates the learning model LM representing the correlation between the group of image data D i_1 to D i_n and the state S of the cable 22, using the learning data set D s the group of image data D i_1 to D i_n and the state data D s.Then, in the above-mentioned step S 12 or S 21, the camera 50 continuously images (captures a video) the cable 22 over the period Δ at the same time as in step S 1, and acquires a group of image data D i_1', D i_2',... D i_n' ( "n" is a positive integer). Then, in the above-mentioned step S 13 or S 23, the processor 64 inputs the group of image data D i_1' to D i_n' to the learning model LM, and the learning model LM estimates and outputs the state data D s correlated with the group of image data D i_1' to D i_n'. According to this configuration, the state S of the cable 22 can be determined based on the video (the group of image data D i_1' to D i_n') of the cable 22.In the above-described embodiments, the camera 50 is arranged so that the cable 22 is imaged at a position near the connection between the wrist 32 and the upper arm 30. However, the camera 50 may be arranged to image an arbitrary position of the cable 22 where the cable 22 is bent while the robot 20 performs the predetermined operation. In addition, multiple cameras 50 may be provided to image different positions of the cable 22.The robot 20 is not limited to a vertical articulated robot, but may be any type of robot such as a horizontal articulated robot, a parallel link robot, or a loading robot as long as this robot includes a movable member and a cable. The disclosure has been described above with reference to embodiments, but the above-described embodiments are not intended to limit the invention according to the scope of the claims.

Claims

A machine learning device (10) configured to learn a state of a cable (22) provided to a robot (20), the device (10) comprising: a learning data acquisition section (12) configured to acquire, as a learning data set, image data of the cable (22) imaged by a camera (50) while the robot (20) performs a predetermined operation and data indicating a state of the cable (22) during the performance of the predetermined operation by the robot (20); and a learning section (14) configured to generate, using the learning data set, a learning model representing a correlation between the image data and the state of the cable (22).The machine learning device (10) according to claim 1, wherein the learning data acquisition section (12) further acquires data of a position and orientation of the robot (20) during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the position and orientation to the correlation.The machine learning device (10) according to claim 1 or 2, wherein the learning data acquisition section (12) further acquires data of a type of the cable (22) or the robot (20) as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the type for correlation.The machine learning device (10) according to any one of claims 1 to 3, wherein the learning data acquisition section (12) further acquires an ambient temperature during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the ambient temperature to the correlation.The machine learning device (10) according to any one of claims 1 to 4, wherein the cable (22A, 22B, 22C) includes a transmission line (38) configured to transmit an electric signal, wherein the learning data acquisition section (12) further acquires an electric current flowing through the transmission line (38) or a resistance of the transmission line (38) as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the current or the resistance for correlation.The machine learning device (10) according to any one of claims 1 to 5, wherein the cable (22A, 22B) includes a supply pipe (36) configured to supply a fluid, wherein the learning data acquisition section (12) further acquires a pressure of the fluid during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the pressure for correlation.The machine learning device (10) according to any one of claims 1 to 6, wherein the cable (22C) includes an optical fiber (42) configured to propagate light, wherein the learning data acquisition section (12) further acquires an optical characteristic of the light during the predetermined operation as a learning data set, wherein the learning section (14) generates the learning model by adding a relationship between the image data and the optical characteristic to the correlation.The machine learning device (10) according to any one of claims 1 to 7, configured to send the learning data set to a storage device (94) communicably connected to the machine learning device (10) via a network (92) to store the learning data set in the storage device (94).A robot system (60) comprising: a robot (20); the machine learning device (10) according to any one of claims 1 to 8; the camera (50) configured to image the cable (22) while the robot (20) performs the predetermined operation, and supply a captured image as image data to the learning data acquisition section (12); and a state determination section (70) configured to determine the state of the cable (22) while the robot (20) performs the predetermined operation based on the learning model generated by the learning section (14).A machine learning method for learning a state of a cable (22) provided on a robot (20), the method comprising performing, by a processor (64), the following steps: acquiring, as a learning data set, image data of the cable (22) imaged by a camera (50) while the robot (20) performs a predetermined operation and data indicating a state of the cable (22) while the robot (20) performs the predetermined operation, and generating a learning model representing a correlation between the image data and the state of the cable (22) using the learning data set.

Images