Machine learning device, robot system and machine learning method for learning cable status

Abstract

The present invention provides a machine learning device, a robot system, and a machine learning method for learning the state of a cable. The machine learning device can quantitatively and accurately evaluate the state of a robot's cable. The machine learning device learns the state of a cable set in a robot, and comprises: a learning data acquisition unit, which acquires image data of the cable captured by a camera when the robot performs a prescribed action, and data indicating the state of the cable when the robot performs the prescribed action as a learning data set; and a learning unit, which uses the learning data set to generate a learning model indicating the correlation between the image data and the state of the cable.

Description

Technical Field

[0001] The present invention relates to a machine learning device, a robot system, and a machine learning method for learning the state of a cable provided in a robot. Background Art

[0002] A device for monitoring cables installed in a robot is known (for example, Japanese Patent Application Laid-Open No. 2017-226000). Conventionally, a technology for quantitatively and accurately evaluating the state of a robot cable has been required. Summary of the invention

[0003] In one embodiment of the present invention, a machine learning device is provided for learning the state of a cable installed in a robot, and comprises: a learning data acquisition unit for acquiring image data of the cable captured by a camera when the robot performs a prescribed action, and data indicating the state of the cable when the robot performs the prescribed action as a learning data set; and a learning unit for using the learning data set to generate a learning model indicating the correlation between the image data and the state of the cable.

[0004] In other embodiments of the present disclosure, a machine learning method is provided for learning the state of a cable provided in a robot, wherein a processor obtains image data of the cable captured by a camera when the robot performs a prescribed action, and data representing the state of the cable when the robot performs the prescribed action as a learning data set, and the processor uses the learning data set to generate a learning model representing the correlation between the image data and the state of the cable.

[0005] According to the present disclosure, it is possible to easily and accurately obtain a model that quantitatively represents the correlation between the image data of the cable and the state of the cable. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 This is a block diagram of a machine learning device according to one embodiment.

[0007] Figure 2 This is a perspective view of a robot according to one embodiment.

[0008] Figure 3 This is a cross-sectional view of a cable according to one embodiment.

[0009] Figure 4 It is a cross-sectional view of a cable according to another embodiment.

[0010] Figure 5 It is a cross-sectional view of a cable according to another embodiment.

[0011] Figure 6 It is magnified Figure 2 An enlarged view of the connection between the wrist and the upper arm is shown.

[0012] Figure 7 This is a diagram for explaining the bending state of the cable when the robot performs a predetermined operation.

[0013] Figure 8 This is a diagram for explaining the bending state of the cable when the robot performs a predetermined operation.

[0014] Fig. 9 This is a diagram for explaining the bending state of the cable when the robot performs a predetermined operation.

[0015] Fig.10 Yes means Figure 1 A flowchart of an example of a learning cycle performed by a machine learning device is shown.

[0016] Fig.11 A model schematically representing a neuron.

[0017] Fig.12 Schematic representation of a model of a multi-layer neural network.

[0018] Fig.13 This is a block diagram of a robot system according to one embodiment.

[0019] Fig.14 Yes means Fig.13 A flowchart showing an example of the operation flow of the robot system shown.

[0020] Fig.15 This is a block diagram of a robot system according to another embodiment.

[0021] Fig.16 Yes means Fig.15 A flowchart showing an example of the operation flow of the robot system shown.

[0022] Fig.17 This is a block diagram of a network system according to one embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in the various embodiments to be described below, the same elements are marked with the same symbols, and repeated descriptions are omitted. Figure 1 A machine learning device 10 according to one embodiment is described. The machine learning device 10 can be composed of a computer having a processor (CPU, GPU, etc.), a storage unit (ROM, RAM, etc.), and an input device (keyboard, mouse, touch panel, etc.), or software such as a learning algorithm executed by the computer.

[0024] The machine learning device 10 learns the state (normal state, abnormal state, etc.) of the cable 22 provided in the robot 20 described later. Figure 2 The robot 20 will be described. In the present embodiment, the robot 20 is a vertical multi-joint robot, and includes a base 24 , a rotating cylinder 26 , a lower arm 28 , an upper arm 30 , a wrist 32 , and a cable 22 .

[0025] The base 24 is fixed to the bottom plate of the working unit. The rotating cylinder 26 is provided on the base 24 so as to be rotatable around a vertical axis. The lower arm 28 is provided on the rotating cylinder 26 so as to be rotatable around a horizontal axis. The upper arm 30 is provided on the distal end of the lower arm 28 so as to be rotatable. The wrist 32 is provided on the front end of the upper arm 30 so as to be rotatable.

[0026] The distal end of the wrist 32 can be detachably mounted on a terminal effector (welding gun, robot hand, tool, paint applicator, etc., not shown) that performs a predetermined operation (welding, workpiece handling, processing, painting, etc.). The robot 20 has a plurality of built-in servo motors (not shown), which drive the movable elements of the robot 20 (i.e., the rotating cylinder 26, the lower arm 28, the upper arm 30, and the wrist 32).

[0027] The cable 22 is laid to each movable element of the robot 20 using a plurality of restraining tools 34. Each restraining tool 34 is fixed to the movable element of the robot 20 at a predetermined position, and defines a hole therein for accommodating the cable 22. The cable 22 is inserted into the hole of the restraining tool 34 and restrained by the restraining tool 34. The fixing position of the restraining tool 34 of the robot 20 is determined according to the type of the robot 20.

[0028] Various cables 22 are used according to the work performed by the robot 20 . Figure 3 The cable 22A includes a supply tube 36, a plurality of transmission wires 38, and a wire sheath 40. The supply tube 36 is a cylindrical member extending along the axis of the wire sheath 40. The supply tube 36 supplies fluid or wires used in the operation of the robot 20.

[0029] As an example, a welding gun is mounted on the wrist 32 as a terminal effector, and when the robot 20 performs welding operations on a workpiece, the supply pipe 36 supplies auxiliary gas as a fluid to the welding gun. Alternatively, the supply pipe 36 accommodates a welding wire as a wire material in its internal space and supplies the welding wire to the welding gun. In addition, as another example, a tool is mounted on the wrist 32, and when the robot 20 performs processing operations on a workpiece, the supply pipe 36 supplies compressed gas as a fluid to the tool.

[0030] The transmission line 38 transmits an electrical signal. For example, the transmission line 38 transmits a signal to a servo motor built into the robot 20 or an end effector mounted on the wrist 32. The supply tube 36 and the transmission line 38 are randomly arranged inside the wire sheath 40 in the cable 22A.

[0031] Figure 4 The cross-sectional view of the second type cable 22B is shown. The cable 22B has a supply tube 36, a total of four transmission wires 38, and a wire sheath 40, and the transmission wires 38 are arranged at substantially equal intervals in the circumferential direction of the supply tube 36 on the outer peripheral side of the supply tube 36. In this way, the supply tube 36, the transmission wire 38, and the wire sheath 40 are arranged concentrically in the cable 22B.

[0032] Figure 5 The figure shows a cross-sectional view of the third type of cable 22C. The cable 22B has: a total of three transmission lines 38, optical fibers 42, and a wire sheath 40. As an example, a laser processing head is installed on the wrist 32 as a terminal effector. When the robot 20 performs laser processing on a workpiece, the optical fiber 42 propagates laser light inside and provides laser light to the laser processing head. In the cable 22C, the transmission lines 38 and the optical fibers 42 are randomly arranged inside the wire sheath 40.

[0033] As described above, a variety of cables 22 (22A, 22B, 22C, etc.) can be used in the robot 20. For example, the types of cables 22 are classified according to the type of wires (supply tube 36, transmission line 38, optical fiber 42) used, the number of wires, the arrangement of the wires in the wire sheath 40, the material of the wire sheath 40, and the position of the binding wires inside the wire sheath 40 that bind the plurality of wires at predetermined intervals.

[0034] Reference Figure 2 , a camera 50 is provided to photograph the cable 22. The camera 50 photographs the cable 22 when the robot 20 performs a prescribed action. For example, the "prescribed action" is a series of actions of the robot 20 when the robot 20 performs an operation (welding, workpiece handling, processing, painting, etc.). When performing the prescribed action, the robot 20 moves each movable element (rotating cylinder 26, lower arm 28, upper arm 30, wrist 32) according to a preset robot program, and performs a series of actions for the operation.

[0035] In other words, the "prescribed action" may be an action that causes the first element of the robot 20 (i.e., the rotating cylinder 26, the lower arm 28, the upper arm 30, or the wrist 32) to rotate relative to the second element (the base 24, the rotating cylinder 26, the lower arm 28, or the upper arm 30) between a predetermined first position and a second position.

[0036] The camera 50 is provided to photograph a preset position of the cable 22 in a preset sight direction when the robot 20 performs a predetermined action. The camera 50 may be fixed at a position different from the robot 20, or may be fixed to an element (base 24, rotating cylinder 26, lower arm 28, or upper arm 30) of the robot 20. In addition, in the present embodiment, as an example, a case where the camera 50 is provided to photograph the cable 22 at a position near the connection portion between the wrist 32 and the upper arm 30 is described.

[0037] When the robot 20 performs a predetermined operation, if the wrist 32 rotates relative to the upper arm 30 , the cable 22 bends at a position near the connection portion between the wrist 32 and the upper arm 30 . Figure 6 to Figure 9 The bending state of the cable 22 during the predetermined action is schematically shown. For example, during the predetermined action, the wrist 32 moves relative to the upper arm 30 from Figure 6 The initial position shown is rotated to Figure 7 to Figure 9 The first reference position is shown.

[0038] Figure 6 In the initial position shown, the axis A1 of the wrist 32 is substantially parallel to the axis A2 of the upper arm 30, and Figure 7 to Figure 9 At the first reference position shown, the axis A1 of the wrist 32 is inclined at a predetermined angle θ1 with respect to the axis A2 of the upper arm 30 . Figure 7 to Figure 9 2 shows an example of the bending state of the cable 22 when the wrist 32 is arranged at the first reference position relative to the upper arm 30 during the predetermined action. Figure 8 Overlapping is shown by dashed lines Figure 7 Cable positions shown, in addition, Fig. 9 The dashed lines and dashed lines are overlapped to show Figure 7 as well as Figure 8 Cable locations shown.

[0039] like Figure 7 to Figure 9 As shown, due to errors in the position where the cables 22 are fastened or dimensional errors in the elements of the robot 20, the bending degree of the cables 22 when the wrist 32 is arranged at the first reference position relative to the upper arm 30 may differ between the same type of cables 22.

[0040] For example, Figure 7 and Figure 8 The bending degree of the cable 22 shown is within the allowable range and is determined not to affect the performance of the cable 22 (signal transmission performance of the transmission line 38, fluid flowability of the supply tube 36, light propagation performance of the optical fiber 42, bending resistance of the cable sheath 40, etc.). Fig. 9 The degree of bending of the cable 22 shown is outside the allowable range, and it is determined that the performance of the cable 22 is affected.

[0041] The machine learning device 10 learns the state (normal state, abnormal state, etc.) of the cable 22 when the robot 20 performs a predetermined operation. Figure 1 The machine learning device 10 includes a learning data acquisition unit 12 and a learning unit 14. The learning data acquisition unit 12 acquires image data D of the cable 22 captured by the camera 50 when the robot 20 performs a predetermined action. i , and data indicating the state S of the cable 22 when the robot 20 performs a predetermined action (hereinafter referred to as the state data D s ) is used as the learning data set DS.

[0042] Specifically, when the robot 20 performs a predetermined operation, the camera 50 captures an image of the cable 22 along the line of sight direction VD at a position near the connection between the wrist 32 and the upper arm 30. The camera 50 generates captured image data D i The state data D of the cable 22 is provided to the learning data acquisition unit 12. s Contains label information such as "normal state" and "abnormal state". Here, the so-called normal state is Figure 7 and Figure 8 As shown, the bending of the cable 22 is within the allowable range. On the other hand, the so-called abnormal state is such as Fig. 9 As shown, the bend of the cable 22 is out of the allowable range.

[0043] In addition, the status data D s It may be just one of the label information of "normal state" and "abnormal state", or it may include both the label information of "normal state" and the label information of "abnormal state". In addition, as the label information of "normal state", multiple label information such as "new product state" and "old product state" may be set. In addition, as the label information of "abnormal state", multiple label information corresponding to the degree of abnormal state may be set, such as "abnormal state level 1", "abnormal state level 2", and "abnormal state level 3".

[0044] Status data D of cable 22 s (ie, label information) is specified by the operator. For example, the operator can i Visually check to confirm the state S (normal state, abnormal state) of the cable 22, and convert the state data D s The learning data acquisition unit 12 acquires the image data D acquired from the camera 50. i and the status data D specified by the operator s As the learning data set DS.

[0045] The learning unit 14 uses the image data D i And the status data D s The learning dataset DS is used to generate image data D i The learning model (function) LM of the correlation with the state S of the cable 22. For example, the learning unit 14 generates the learning model LM by performing supervised learning. In this case, the robot 20 repeatedly tries the prescribed action, and the learning data acquisition unit 12 repeatedly acquires the learning data set DS as training data each time the robot 20 performs the prescribed action.

[0046] The learning unit 14 recognizes and implies the image data D acquired as training data. i (For example, Figure 7 to Figure 9 The image data of the cable 22 shown) and the status data D s The learning model LM is learned by using the features of the correlation between the labels (normal state, abnormal state, etc.). For such supervised learning, for example, algorithms such as support vector machine (SVM) or Gaussian mixture model (GMM) can be used.

[0047] Below, refer to Fig.10 , the flow of the learning cycle performed by the machine learning device 10 is described. In step S1, the robot 20 performs a predetermined action, and the camera 50 photographs the cable 22 when the robot 20 performs the predetermined action. As an example, the camera 50 is arranged at a position where the wrist 32 is relative to the upper arm 30 during the period when the robot 20 performs the predetermined action. Figure 7 to Figure 9 At the time point τ1 of the first reference position shown, the cable 22 is imaged once.

[0048] As another example, the camera 50 may continuously photograph the cable 22 (in other words, photograph an animation) while the robot 20 performs a predetermined action. In this case, the camera 50 photographs the cable 22 at a time point τn when the wrist 32 is disposed at an nth reference position relative to the upper arm 30, wherein the nth reference position is a time point when the axis A1 of the wrist 32 is at a predetermined angle θ relative to the axis A2 of the upper arm 30. n (n=1, 2, 3, ...) inclined.

[0049] In step S2, the learning data acquisition unit 12 acquires the learning data set DS. Specifically, the learning data acquisition unit 12 acquires the image data D of the cable 22 captured by the camera 50 during the predetermined operation of the robot 20 in step S1. i .

[0050] In addition, the learning data acquisition unit 12 acquires normal state label information or abnormal state label information as data indicating the state S of the cable 22 when step S1 is executed (state data D sSpecifically, the operator uses the image data D of the cable 22 captured by the camera 50 to obtain the image data D of the cable 22. i Visual confirmation is performed to confirm whether the cable 22 is in a normal state or an abnormal state.

[0051] Furthermore, the operator operates the input device of the machine learning device 10 to input the label information of the normal state or the label information of the abnormal state as the state data D s The learning data acquisition unit 12 acquires the input normal state label information or abnormal state label information. In this way, the learning data acquisition unit 12 acquires the image data D of the cable 22. i With status data D s The learning data sets DS are associated with each other and stored in the storage unit.

[0052] In step S3, the learning unit 14 generates image data D using the learning data set DS obtained in step S2. i The learning model LM of the correlation with the state S. Specifically, the learning unit 14 learns the learning model LM by executing a supervised learning algorithm (SVM, GMM, etc.). Then, the flow returns to step S1.

[0053] By executing such a learning cycle, the learning model LM is learned and the learning model LM is guided toward the optimal solution. Fig.10 As a result, more learning data sets DS can be obtained, and the learning model LM can be learned at a higher level. According to this embodiment, the image data D quantitatively representing the cable 22 can be easily and accurately obtained. i Model of the dependency on the state S of the cable 22 .

[0054] As another function of the machine learning device 10, in step S2, the learning data acquisition unit 12 can acquire data on the type of the cable 22 (first type, second type, ... nth type) or data on the type of the robot 20 as a learning data set DS. The data on the type of the cable 22 includes information for identifying the type of the cable 22, information for identifying the type of the wire (supply pipe 36, transmission line 38, optical fiber 42) used, the number of wires, information indicating the configuration of the wires in the wire sheath 40, the material of the wire sheath 40, and information on the position of the binding wire that binds a plurality of wires at predetermined intervals inside the wire sheath 40.

[0055] The information identifying the type of cable 22 may be, for example, text information such as "nth type", or may be a product number marked by the cable manufacturing company. In addition, the information identifying the type of line may be, for example, text information such as "supply pipe", "transmission line", or "optical fiber".

[0056] In addition, the information indicating the arrangement of the wire body may be text information such as "random arrangement" or "concentric arrangement". In addition, the material of the wire sheath 40 may be text information such as "resin" or "rubber", or may be a product number marked by a manufacturing company of the material of the wire sheath 40. In addition, the information on the position of the binding wire may be, for example, the average distance between two binding wires or the number of binding wires.

[0057] In addition, the type data of the robot 20 includes information for identifying the type of the robot 20, the number of laid cables 22, information on the fixed position of the restraining tool 34 in the robot 20, etc. The information for identifying the type of the robot 20 may be text information such as "nth type", or may be a product number marked by the robot manufacturing company for each robot.

[0058] As an example, in step S2, the operator operates the input device of the machine learning device 10 to input data D of the type of the cable 22 or the robot 20. t In this way, the learning data acquisition unit 12 obtains the image data D i And the status data D s In addition, we also obtain data of type D t As the training data set DS. Then, in step S3, the learning unit 14 performs the training on the image data D i With status data D s Correlation, adding image data D i Data with type D t The correlation is used to generate the learning model LM.

[0059] The degree of bending of the cable 22 when the robot 20 performs a predetermined action may vary depending on the type of the cable 22. For example, the cable 22 may be more difficult to bend as the number of wires routed inside the wire sheath 40 increases or the material of the wire sheath 40 becomes less flexible.

[0060] Furthermore, the degree of bending of the cable 22 when the robot 20 performs a predetermined action may be different depending on the type of robot 20. For example, the fixing position of the restraint tool 34 varies depending on the type of robot 20, and as a result, the degree of bending of the cable 22 when performing a predetermined action varies.

[0061] Thus, there is a correlation between the degree of bending of the cable 22 and the type of the cable 22 or the robot 20. Therefore, by adding the image data D i By generating the learning model LM based on the correlation with the type of the cable 22 or the robot 20 , the learning model LM corresponding to the type of the cable 22 or the robot 20 can be acquired.

[0062] As another function of the machine learning device 10, in step S2, the learning data acquisition unit 12 acquires the image data D i And the status data D s In addition, the position and posture data of the robot 20 when the robot 20 performs the prescribed action in step S1 (hereinafter referred to as position data D p ) is used as the learning data set DS.

[0063] For example, when the camera 50 captures an image of the cable 22 at time τ1 in step S1, the learning data acquisition unit 12 acquires the position data D of the robot 20 at time τ1. p As another example, in step S1, the camera 50 is at multiple time points τ n When the cable 22 is continuously photographed (video recording) (n=1, 2, 3, ...), the learning data acquisition unit 12 sequentially acquires a plurality of time points τ n The position data D of each robot 20 p .

[0064] Position data D of the robot 20 p The rotation angle θ can be obtained from the rotation angle θ of each servo motor built into the robot 20. Therefore, in this case, each servo motor of the robot 20 is provided with a sensor (encoder, Hall element, etc.) that detects the rotation angle θ when the camera 50 takes an image. In addition, the rotation angle θ can also be used as the position data D of the robot 20. p Then, in step S3, the learning unit 14 processes the image data D i With status data D s The correlation of adding image data D i The learning model LM is generated based on the correlation with the position and posture of the robot 20.

[0065] Here, the degree of bending of the cable 22 when the robot 20 performs a predetermined action may be different according to the position and posture of the robot 20 (for example, the angle of the wrist 32 relative to the upper arm 30). That is, there is a correlation between the degree of bending of the cable 22 and the position and posture of the robot 20. Therefore, by adding the image data D i By generating the learning model LM based on the correlation with the position and posture of the robot 20 , the learning model LM corresponding to the position and posture of the robot 20 can be obtained.

[0066] As another function of the machine learning device 10, in step S2, the learning data acquisition unit 12 may acquire the ambient temperature T when the robot 20 performs a predetermined operation in step S1 as a learning data set DS. Therefore, in this case, a sensor (temperature sensor) for detecting the ambient temperature T is provided.

[0067] Then, in step S3, the learning unit 14 processes the image data D i With status data D s The correlation of adding image data D i The learning model LM is generated by analyzing the correlation between the cable 22 and the ambient temperature T. Here, there is a correlation between the degree of bending of the cable 22 and the ambient temperature T when the robot 20 performs a predetermined operation.

[0068] Specifically, depending on the material of the cable sheath 40, if the ambient temperature T increases, the flexibility of the cable sheath 40 increases, thereby making it easier for the cable 22 to bend, while if the ambient temperature T is low, the flexibility of the cable sheath 40 decreases, thereby making it difficult for the cable 22 to bend. i By generating the learning model LM based on the correlation with the ambient temperature T, the learning model LM corresponding to the ambient temperature T can be obtained.

[0069] As another function of the machine learning device 10, in step S2, the learning data acquisition unit 12 may also acquire the current A flowing through the transmission line 38 or the resistance R of the transmission line 38 when the robot 20 performs a predetermined action in step S1 as a learning data set DS. Therefore, in this case, a sensor (ammeter, resistance meter, etc.) is provided to detect the current A or resistance R of the transmission line 38 when performing a predetermined action.

[0070] Then, in step S3, the learning unit 14 performs the image data D i With status data D s The correlation of adding image data D i The learning model LM is generated by analyzing the correlation between the current A and the resistance R. Here, the current A or the resistance R may vary in accordance with the degree of bending of the cable 22 when the robot 20 performs a predetermined action. For example, if the degree of bending of the cable 22 exceeds the allowable range and causes a partial disconnection of the transmission line 38, the resistance R of the transmission line 38 will increase, and the current A will decrease.

[0071] Thus, there is a correlation between the bending degree of the cable 22 and the current A or the resistance R. Therefore, by adding the image data D i By generating the learning model LM based on the correlation with the current A or the resistance R, the learning model LM corresponding to the current A or the resistance R can be obtained.

[0072] As another function of the machine learning device 10, for example, when using Figure 3 The cable 22A shown or Figure 4When the cable 22B is shown, in step S2, the learning data acquisition unit 12 can also acquire the pressure P of the fluid in the supply pipe 36 when the robot 20 performs the prescribed action in step S1 as the learning data set DS. Therefore, in this case, a sensor (pressure gauge, etc.) is provided to detect the pressure P when performing the prescribed action.

[0073] Then, in step S3, the learning unit 14 processes the image data D i With status data D s The correlation of adding image data D i The learning model LM is generated by analyzing the correlation between the cable 22 and the pressure P. Here, the degree of bending of the cable 22 when the robot 20 performs a predetermined operation may be changed in accordance with the pressure P of the fluid in the supply pipe 36 .

[0074] Specifically, the greater the pressure P of the fluid in the supply pipe 36, the harder it is for the cable 22 to bend. Thus, there is a correlation between the degree of bending of the cable 22 and the pressure P. Therefore, by adding the image data D i The learning model LM is generated based on the correlation with the pressure P, and the learning model LM corresponding to the pressure P can be obtained.

[0075] As another function of the machine learning device 10, for example, using Figure 5 When the cable 22C shown in FIG. 1 is used, in step S2, the learning data acquisition unit 12 may also acquire the optical characteristics OP of the light (e.g., laser light) propagating in the optical fiber 42 when the robot 20 performs the prescribed action in step S1 as the learning data set DS. The optical characteristics OP may include, for example, the intensity, power, or intensity distribution of the light (or the return light of the light) propagating in the optical fiber 42. Therefore, in this case, a sensor (light intensity meter, beam analyzer, etc.) is provided to detect the optical characteristics OP when the prescribed action is performed.

[0076] Then, in step S3, the learning unit 14 processes the image data D i With status data D s The correlation of adding image data D i The learning model LM is generated by analyzing the correlation between the optical characteristic OP and the cable 22. Here, the optical characteristic OP may vary in accordance with the degree of bending of the cable 22 when the robot 20 performs a predetermined action. For example, when the degree of bending of the cable 22 exceeds the allowable range, the light leakage at the bending portion of the cable 22 increases, thereby reducing the intensity or power of the light (or the return light).

[0077] Thus, there is a correlation between the bending degree of the cable 22 and the optical characteristic OP. i By generating the learning model LM based on the correlation with the optical characteristic OP, the learning model LM corresponding to the optical characteristic OP can be obtained.

[0078] In addition, the learning algorithm executed by the learning unit 14 is not limited to supervised learning, and for example, unsupervised learning, reinforcement learning, neural network, and other learning algorithms known as machine learning can be used. As an example, Fig.11 A model schematically representing a neuron. Fig.12 Schematically indicating that Fig.11 The model of a three-layer neural network composed of a combination of neurons shown. The neural network can be composed of, for example, a computing device or a storage device that simulates a neuron model.

[0079] Fig.11 The neuron shown outputs result y for multiple inputs x (inputs x1 to x3 as an example in the figure). Each input x (x1, x2, x3) is multiplied by a weight w (w1, w2, w3). The relationship between input x and result y can be expressed by the following mathematical formula 1. In addition, input x, result y and weight w are all vectors. In addition, in mathematical formula 1, θ is the bias and fk is the activation function.

[0080]

[0081] Fig.11 The three-layer neural network shown in the figure inputs multiple inputs x from the left side (input x1 to input x3 as an example in the figure), and outputs results y from the right side (results y1 to y3 as an example in the figure). In the example shown in the figure, each of the inputs x1, x2, and x3 is multiplied by a corresponding weight (uniformly represented by W1), and each of the inputs x1, x2, and x3 is input to each of the three neurons N11, N12, and N13.

[0082] Fig.12 In the example, Z1 is used to represent the output of each neuron N11 to N13. Z1 can be regarded as a feature vector that extracts the feature value of the input vector. In the example shown in the figure, each feature vector Z1 is multiplied by a corresponding weight (uniformly represented by W2), and each feature vector Z1 is input to each of the two neurons N21 and N22. The feature vector Z1 represents the feature between the weight W1 and the weight W2.

[0083] Fig.12 In the example shown in the figure, Z2 is used to represent the output of each neuron N21 to N22. Z2 can be regarded as a feature vector that extracts the feature value of feature vector Z1. In the example shown in the figure, each feature vector Z2 is multiplied by a corresponding weight (uniformly represented by W3), and each feature vector Z2 is input to each of the three neurons N31, N32, and N33.

[0084] The feature vector Z2 represents the feature between the weight W2 and the weight W3. Finally, neurons N31 to N33 output results y1 to y3 respectively. The machine learning device 10 takes the learning data set DS as input and performs operations according to the multi-layer structure of the above-mentioned neural network, thereby learning the learning model LM.

[0085] The structure of the machine learning device 10 can be described as a machine learning method (or software) executed by a processor of a computer. Regarding the machine learning method, the processor obtains the image data D of the cable 22 captured by the camera 50 when the robot 20 performs a predetermined action. i , and data D indicating the state S of the cable 22 when the prescribed action is performed s As the learning dataset DS, the learning dataset DS is used to generate the representation image data D i A learning model LM of the dependency on the state S of the cable 22 .

[0086] Next, refer to Fig.13 , a robot system 60 involved in one embodiment is described. The robot system 60 includes: a robot 20, a camera 50, and a control device 62. The control device 62 includes a processor 64 and a storage unit 66. The processor 64 and the storage unit 66 are connected to each other via a bus 68 so as to be able to communicate with each other. The processor 64 includes a CPU or a GPU, etc., communicates with the storage unit 66 and performs various operations. The control device 62 can also be composed of a computer having an input device (keyboard, mouse, touch panel, etc.) and a display device (LCD, organic EL, etc.).

[0087] In this embodiment, the machine learning device 10 is installed in the control device 62 as hardware or software, and the processor 64 executes various operations for obtaining the functions of the machine learning device 10. That is, in this embodiment, the processor 64 functions as the learning data acquisition unit 12 and the learning unit 14.

[0088] The storage unit 66 includes a ROM and a RAM, etc., and pre-stores the learning model LM learned by the machine learning device 10. The robot system 60 involved in this embodiment is installed in, for example, a factory that manufactures the robot system 60, or a factory where the robot 20 performs actual operations (welding, workpiece handling, processing, painting, etc.).

[0089] Next, refer to Fig.14 The operation of the robot system 60 will be described. When the processor 64 receives an operation start instruction from an operator, a host controller, or a robot program, the robot system 60 starts to operate. Fig.14 In step S11, the processor 64 causes the robot 20 to perform a predetermined action.

[0090] For example, the processor 64 immediately causes the robot 20 to perform an operation (welding, workpiece handling, processing, painting, etc.) as a prescribed action, and causes each movable element (rotating cylinder 26, lower arm 28, upper arm 30, wrist 32) to move according to a preset robot program, so that the robot 20 performs a series of actions. In other words, the processor 64 performs an action of rotating the first element (wrist 32) of the robot 20 relative to the second element (upper arm 30) between a preset first position and a preset second position as a prescribed action.

[0091] Here, the predetermined action executed by the robot 20 in step S11 is substantially the same as the predetermined action executed by the robot 20 for learning in step S1. For example, in step S11, the processor 64 may cause the robot 20 to move according to the same robot program as that used to cause the robot 20 to move in step S1.

[0092] However, the prescribed action performed in step S11 is not limited to actions with exactly the same conditions as step S1 (type of terminal actuator, position and posture of robot 20, movement path of robot 20, etc.). As long as the degree of bending of cable 22 is the same, at least one of the conditions can be an action different from step S1.

[0093] In step S12, the processor 64 uses the camera 50 to photograph the cable 22 when the robot 20 performs a predetermined action. Here, in step S12, the processor 64 photographs the same position of the cable 22 as when the camera 50 photographed the cable 22 in the above step S1 along the same line of sight direction VD and at the same time.

[0094] As an example, in the above-mentioned step S1, when the camera 50 takes a picture of the cable 22 at the time point τ1, in the step S12, the processor 64 takes a picture of the cable 22 along the line of sight direction VD at the time point τ1 when the wrist 32 is configured relative to the upper arm 30 at the first reference position during the execution of step S11, through the camera 50 at a position near the connection part between the wrist 32 and the upper arm 30.

[0095] As another example, in the above step S1, the camera 50 is at multiple time points τ n (n=1, 2, 3, ...) When the cable 22 is continuously photographed, in step S12, the processor 64 respectively performs step S11 at the time point τ when the wrist 32 is arranged relative to the upper arm 30 at the nth reference position. n The camera 50 captures the cable 22 along the sight line direction VD at a position near the connection between the wrist 32 and the upper arm 30. The camera 50 captures the image data D of the cable 22. i 'Provided to processor 64.

[0096] In step S13, the processor 64 determines the state S of the cable 22 based on the learning model LM. Specifically, the processor 64 converts the image data D of the cable 22 obtained in step S12 into i ' is input to the learning model LM stored in the storage unit 66. In this way, the learning model LM estimates the input image data D i 'Status data D with correlation s (label information such as "normal state" and "abnormal state") and output it. In this way, the processor 64 can obtain the image data D i 'To determine the state S of the cable 22.

[0097] When the state S of the cable 22 is not an abnormal state (or is a normal state), the processor 64 determines that it is not, and ends the operation started in step S11, and ends the operation. Fig.14 On the other hand, when the state S of the cable 22 is abnormal (or not normal), the processor 64 determines that it is yes and transfers to step S14. Thus, in this embodiment, the processor 64 serves as the state determination unit 70 ( Fig.13 ) to perform its function.

[0098] In step S14, the processor 64 generates a warning signal. For example, the processor 64 generates a sound or image signal such as "the cable status may be abnormal" and outputs it through a speaker or monitor (not shown) provided in the control device 62. Then, the processor 64 ends Fig.14 The process shown.

[0099] in addition, Fig.10 In the learning cycle shown, data D of the type of the cable 22 or the robot 20 is also obtained. t When learning the learning model LM as the learning data set DS, before step S13, the operator may operate the input device provided in the control device 62 to input data D of the type of the robot 20 or the cable 22 of the robot system 60. t '.

[0100] In this case, in step S13, the processor 64 of the robot system 80 converts the image data D of the cable 22 acquired in step S12 into i ', and the type of data D t 'Input to the learning model LM. In this way, the learning model LM is based on the image data D i ' and type data D t 'To estimate the state data D s , determine the state S of the cable 22. According to this structure, it is possible tot 'Correspondingly, the state S of the cable 22 is determined with higher accuracy.

[0101] Next, refer to Fig.15 , a robot system 80 according to another embodiment will be described. The robot system 80 is different from the robot system 60 described above in that it further includes a sensor 82. As an example, in Fig.10 In the learning cycle shown, position data D is also obtained. p When learning the learning model LM as the learning data set DS, the sensor 82 is an encoder or a Hall element or the like built into each servo motor of the robot 20 of the robot system 80 , and detects the rotation angle θ′ of the servo motor as a detection value.

[0102] As another example, in Fig.10 In the learning cycle shown, when the ambient temperature T is also obtained as the learning data set DS to learn the learning model LM, the sensor 82 is a temperature sensor, etc., and detects the ambient temperature T' of the robot 20 of the robot system 80 as a detection value. As another example, in Fig.10 In the learning cycle shown, when the current A or resistance R of the transmission line 38 is obtained as the learning data set DS to learn the learning model LM, the sensor 82 is an ammeter or a resistance meter, etc., which detects the current A' or resistance R' of the transmission line 38 of the cable 22 of the robot system 80 as the detection value.

[0103] As another example, in Fig.10 In the learning cycle shown, when the pressure P of the fluid in the supply pipe 36 is obtained as the learning data set DS to learn the learning model LM, the sensor 82 is a pressure gauge, etc., which detects the pressure P' of the fluid in the supply pipe 36 of the cable 22 of the robot system 80 as a detection value.

[0104] As another example, in Fig.10 In the learning cycle shown, when the optical characteristic OP is also obtained as the learning data set DS to learn the learning model LM, the sensor 82 is a light intensity meter or a beam analyzer, etc., which detects the optical characteristic OP' of the light propagating in the optical fiber 42 of the cable 22 of the robot system 80 as a detection value.

[0105] Next, refer to Fig.16 The operation of the robot system 80 will be described. Fig.16 In the process shown, Fig.14The same process in the flow chart is marked with the same step number, and repeated description is omitted. In step S21, the processor 64 of the robot system 80 uses the camera 50 to photograph the cable 22 when the robot 20 performs the prescribed action in step S11. Here, in step S21, the processor 64 causes the camera 50 to photograph the same position of the cable 22 as when the camera 50 photographed the cable 22 in the above step S1 along the same line of sight direction VD.

[0106] As an example, in Fig.10 In the learning cycle shown, position data D is also obtained. p When learning the learning model LM, the processor 64 causes the camera 50 to photograph the cable 22 at any time during the robot 20 performs a predetermined action in step S11. Fig.10 No position data D is obtained in the learning cycle shown p When the camera 50 images the cable 22 , similarly to the above-mentioned step S12 , the camera 50 is caused to image the cable 22 at the same timing as when the camera 50 images the cable 22 in step S1 of the learning phase.

[0107] In step S22, the processor 64 of the robot system 80 activates the sensor 82 to obtain the detection value when the robot 20 performs the predetermined action in step S11. Fig.10 In the learning cycle shown, position data D is also obtained. p When learning the learning model LM, the sensor 82 detects the rotation angle θ' as a detection value at the time when the camera 50 photographs the cable 22 in step S21. Then, the processor 64 obtains the position data D of the robot 20 of the robot system 80 based on the rotation angle θ'. p '.

[0108] As another example, in Fig.10 When the learning model LM is learned by acquiring the ambient temperature T in the learning cycle shown, the sensor 82 detects the ambient temperature T' as the detection value during the execution of step S11 (or before and after the execution). Fig.10 When the current A or resistance R is also acquired in the learning cycle shown to learn the learning model LM, the sensor 82 detects the current A' or resistance R' as a detection value during the execution of step S11 (for example, the time point when the camera 50 photographs the cable 22 in step S21).

[0109] As another example, in Fig.10When the pressure P is also acquired in the learning cycle shown to learn the learning model LM, the sensor 82 detects the pressure P' as the detection value during the execution of step S11 (for example, the time point when the camera 50 photographs the cable 22 in step S12). As another example, in Fig.10 In the learning cycle shown, when the optical characteristic OP is acquired to perform learning of the learning model LM, the sensor 82 detects the optical characteristic OP′ as a detection value.

[0110] In step S23, the processor 64 of the robot system 80 determines the state S of the cable 22 based on the learning model LM. As an example, the position data D is obtained in the above step S22. p ', the processor 64 converts the image data D of the cable 22 obtained in step S21 into i 'With the position data D obtained in step S22 p 'Input to the learning model LM stored in the storage unit 66.

[0111] In this way, the learning model LM is based on the image data D i ' and position data D p 'To estimate the state data D s , determine the state S of the cable 22. In this way, the position data D is obtained p When learning the learning model LM, the image data D captured at any time in step S21 is used as the learning data set DS. i ' is input to the learning model LM, thereby determining the state S of the cable 22. In addition, the state S of the cable 22 can be determined with higher accuracy in accordance with the position and posture of the robot 20.

[0112] In addition, as another example, when the detected value of the ambient temperature T', the current A', the resistance R', the pressure P', or the optical characteristic OP' is obtained in the above step S22, the processor 64 converts the image data D of the cable 22 obtained in step S21 into i ', and the detection value T', A', R', P' or OP' obtained in step S22 are input to the learning model LM.

[0113] In this way, the learning model LM is based on the image data D i ' and the detected value T', A', R', P' or OP' estimated state data D s , and determine the state S of the cable 22. According to this structure, the state S of the cable 22 can be determined with higher accuracy in accordance with the ambient temperature T', the current A', the resistance R', the pressure P', or the optical characteristic OP'.

[0114] As described above, in the robot systems 60 and 80, when the processor 64 causes the robot 20 to perform a predetermined action, the state S of the cable 22 is determined based on the learning model LM learned by the machine learning device 10. According to this structure, it is possible to automatically and accurately determine whether the state S of the cable 22 is abnormal at a work site such as a factory.

[0115] In addition, in step S23 , in addition to using the learning model LM to determine the state S of the cable 22 , the processor 64 may further determine whether the current A′ or resistance R′, pressure P′, or optical characteristic OP′ detected by the sensor 82 is within a preset allowable range.

[0116] Then, when the current A' or the resistance R', the pressure P', or the optical characteristic OP' is outside the allowable range, the processor 64 can determine in step S23 that it is yes and transfer to step S14, assuming that even if the learning model LM determines that the state S of the cable 22 is not an abnormal state (or a normal state). According to this structure, the processor 64 can detect the abnormality of the cable 22 more reliably based on the detection values ​​such as the current A' or the resistance R', the pressure P', or the optical characteristic OP in addition to the determination of the state S by the learning model LM.

[0117] Next, refer to Fig.17 A network system 90 according to one embodiment is described. The network system 90 includes a plurality of machine learning devices 10A, 10B, 10C, and 10D, a storage device 94, and a network 92 that connects the machine learning devices 10A, 10B, 10C, and 10D and the storage device 94 so as to be communicable.

[0118] Each of the machine learning devices 10A, 10B, 10C, and 10D has the same structure as the machine learning device 10 described above. The machine learning devices 10A, 10B, 10C, and 10D may be installed in different factories or in different production lines in the same factory. The storage device 94 has a server and the like and can store data. The network 92 is a LAN or Ethernet, and data communication is performed bidirectionally between each of the machine learning devices 10A, 10B, 10C, and 10D and the storage device 94.

[0119] Each of the machine learning devices 10A, 10B, 10C, and 10D transmits the learning data set DS acquired by the learning data acquisition unit 12 to the storage device 94 via the network 92. The storage device 94 stores the learning data set DS received from each of the machine learning devices 10A, 10B, 10C, and 10D.

[0120] Furthermore, each of the machine learning devices 10A, 10B, 10C, and 10D obtains the learning data set DS stored in the storage device 94 from the storage device 94 via the network 92. And the learning unit 14 of each of the machine learning devices 10A, 10B, 10C, and 10D uses the learning data set DS obtained from the storage device 94 to learn the learning model LM.

[0121] According to this embodiment, the learning data sets DS obtained by the plurality of machine learning devices 10A, 10B, 10C, and 10D can be collected in the storage device 94, and the learning data sets DS can be shared among the plurality of machine learning devices 10A, 10B, 10C, and 10D. According to this structure, the machine learning devices 10A, 10B, 10C, and 10D can use more learning data sets DS to learn the learning model LM, and thus can effectively learn the learning model LM.

[0122] In addition, the learning data acquisition unit 12 can also acquire the position data D p , Type data D t , ambient temperature T, current A or resistance R, pressure P, and at least two (for example, all) of the optical characteristics OP' as the learning data set DS. In addition, the learning unit 14 may perform the image data D i With status data D s Add the location data D p , Type data D t , ambient temperature T, current A or resistance R, pressure P, and at least two of the optical characteristics OP' and the image data D i , to generate the learning model LM.

[0123] In addition, in the above step S2, the operator can use a sensor to detect the current A or the resistance R, the pressure P, or the optical characteristic OP, and confirm the state S of the cable 22 according to the current A or the resistance R, the pressure P, or the optical characteristic OP, and convert the state data D s The data is input to the learning data acquisition unit 12. In this case, the machine learning device 10 further includes a detection value determination unit that determines whether the current A, the resistance R, the pressure P, or the optical characteristic OP is within a preset allowable range.

[0124] In the above step S1, the camera 50 continuously photographs the cable 22 (photographs a moving image) for a predetermined period Δ, and obtains a set of image data D i_1 , D i_2 ,…D i_n (n is a positive integer), in step S2, the operator can i_1 ~D i_n(i.e., animation of period Δ) gives 1 state data D s (ie, label information) is input to the learning data acquisition unit 12.

[0125] In this case, the learning data acquisition unit 12 acquires a set of image data D i_1 ~D i_n Given to the set of image data D i_1 ~D i_n 1 state data D s The learning unit 14 uses a set of image data D i_1 ~D i_n With status data D s The learning dataset DS is used to generate a set of image data D i_1 ~D i_n A learning model LM of the dependency on the state S of the cable 22 .

[0126] Furthermore, in the above step S12 or S21, the camera 50 continuously photographs the cable 22 (photographs an animation) within a period Δ at the same time as in step S1, and obtains a set of image data D i_1 '、D i_2 ', ...D i_n '(n is a positive integer). In addition, in the above step S13 or S23, the processor 64 converts a set of image data D i_1 '~D i_n 'Input to the learning model LM, the learning model LM infers a set of image data D i_1 '~D i_n 'Status data D with correlation s According to this structure, the animation of the cable 22 (a set of image data D i_1 '~D i_n ') to determine the state S of the cable 22.

[0127] In addition, in the above embodiment, the camera 50 is provided to photograph the cable 22 at a position near the connection portion between the wrist 32 and the upper arm 30. However, the camera 50 is not limited thereto, and may be provided to photograph any position of the cable 22 that is bent when the robot 20 performs a predetermined action. In addition, a plurality of cameras 50 may be provided to photograph different positions of the cable 22.

[0128] In addition, the robot 20 is not limited to a vertical multi-joint robot, and can be any type of robot as long as it has movable elements and cables, such as a horizontal multi-joint type, a parallel link type, a loader, etc. The present disclosure is described above through the embodiments, but the above embodiments do not limit the invention related to the claims.

Claims

1. A machine learning device for learning the bending state of a cable set in a robot, characterized in that: The machine learning device has: a learning data acquisition unit that acquires, as a learning data set, image data of the cable bent in accordance with the prescribed action captured by a camera at a time point when the robot is arranged at a predetermined reference position when the robot performs a prescribed action, and data indicating whether the degree of bending of the cable when the robot performs the prescribed action is within an allowable range; as well as A learning unit generates a learning model representing a correlation between the image data and a bending state of the cable using the learning data set.

2. The machine learning device according to claim 1, characterized in that The learning data acquisition unit further acquires data on the position and posture of the robot when performing the predetermined action as the learning data set, The learning unit generates the learning model by adding the association between the image data and the position and posture to the correlation.

3. The machine learning device according to claim 1 or 2, characterized in that: The learning data acquisition unit further acquires data on the type of the cable or the robot as the learning data set. The learning unit generates the learning model by adding the association between the image data and the category to the correlation.

4. The machine learning device according to claim 1 or 2, characterized in that: The learning data acquisition unit further acquires the ambient temperature during the predetermined action as the learning data set, The learning unit generates the learning model by adding the relationship between the image data and the ambient temperature to the correlation.

5. The machine learning device according to claim 1 or 2, characterized in that: The cable has: a transmission line that transmits an electrical signal; The learning data acquisition unit further acquires the current flowing through the transmission line or the resistance of the transmission line as the learning data set. The learning unit generates the learning model by adding the relationship between the image data and the current or the resistance to the correlation.

6. A machine learning device for learning the bending state of a cable installed in a robot, characterized in that: The machine learning device has: a learning data acquisition unit that acquires, as a learning data set, image data of the cable bent in accordance with the prescribed action captured by a camera when the robot performs the prescribed action, and data indicating whether the degree of bending of the cable when the robot performs the prescribed action is within an allowable range; as well as a learning unit that generates a learning model representing a correlation between the image data and a bending state of the cable using the learning data set, The cable has: a supply pipe, which supplies fluid, The learning data acquisition unit further acquires the pressure of the fluid during the predetermined action as the learning data set, The learning unit generates the learning model by adding the relevance between the image data and the pressure to the correlation.

7. The machine learning device according to claim 1 or 2, characterized in that: The cable has: an optical fiber that transmits light, The learning data acquisition unit further acquires the optical characteristics of the light during the predetermined action as the learning data set, The learning unit generates the learning model by adding the relationship between the image data and the optical characteristics to the correlation.

8. The machine learning device according to claim 1 or 2, characterized in that: The machine learning device sends the learning data set to a storage device connected to the machine learning device so as to be communicable via a network, and stores the learning data set in the storage device.

9. A robot system, characterized in that: have: robot; The machine learning device according to any one of claims 1 to 8; a camera that photographs the cable at a time point when the robot is arranged at the reference position when the robot performs the predetermined action, and provides the camera to the learning data acquisition unit as the image data; as well as A state determination unit determines a bending state of the cable when the robot performs the predetermined action based on the image data captured at the time point and the learning model generated by the learning unit.

10. A machine learning method for learning the bending state of a cable set in a robot, characterized in that: The processor obtains, as a learning data set, image data of the cable bent in accordance with the prescribed action photographed by a camera at a time point when the robot is arranged at a predetermined reference position when the robot performs the prescribed action, and data indicating whether the degree of bending of the cable when the robot performs the prescribed action is within an allowable range, The processor generates a learning model representing a correlation between the image data and a bending state of the cable using the learning data set.

Images