Industrial Robot Debugging Method Based on the Combination of Natural Language and Computer Vision

Abstract

The present invention discloses an industrial robot debugging method based on the combination of natural language and computer vision. The present invention generates semantic information by passing natural language descriptions through a word2vec network and a linear layer. The industrial robot environmental features are extracted by a three-dimensional recurrent convolutional neural network. The environmental features are input into a long short-term memory network to generate intermediate context, and the intermediate context is used as the input of a GRU network to obtain API recommendations through a recurrent neural network RNN. The text embedding generated by passing natural language descriptions through a word2vec network is input into a long short-term memory network encoder and a long short-term memory network decoder to output an AST construction action sequence, and the API recommendations and the construction action sequence are combined to generate industrial robot debugging code. The debugging code is added to the robot program editor to complete the debugging. The present invention effectively improves the development efficiency of robot debugging and reduces the deployment time of robot production lines in industrial environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial robots, and particularly relates to an industrial robot debugging method based on the combination of natural language and computer vision. Background Art

[0002] In recent years, with the country's advocacy of intelligent factories, the manufacturing industry has begun to widely use robot technology to assist production, and the concept of intelligent manufacturing has entered the stage of comprehensive promotion from popularization. As a main component of an intelligent factory, robots can help improve the productivity of the factory, perform operations that workers cannot do, and can quickly adapt to new production requirements.

[0003] Currently, the debugging methods of industrial robots on the market can be divided into on-line debugging and off-line debugging. For on-line debugging, users need to control the robot to complete specified actions and save them, and the specified actions can be repeated by running them. For off-line debugging, users need to perform "virtual" programming on the robot through software tools. During the debugging period, the robot does not need to stop, which does not interfere with production operations. Whether it is on-line debugging or off-line debugging, these methods have corresponding disadvantages. For on-line debugging, users need to operate the robot on-site. Users need to have rich experience, and it takes a lot of time to program for complex tasks. Moreover, the robot needs to stop during the debugging process, which affects production operations. Off-line debugging also requires users to have rich experience and programming capabilities in the corresponding robot language, and after completion, users need to fine-tune the robot actions according to the actual scenario. For example, in a certain elevator company, the off-line debugging method was used to complete the debugging work of the robot. The engineer spent six months to implement a new welding plate handling production line, and the time spent on "virtual" programming was as high as five months. Debugging the robot in an industrial environment requires users to have professional domain knowledge, understand the industrial site environment and the movement trajectory of the robot. Even professional engineers need to spend a lot of time. Shortening the development cycle and quickly adapting to production requirements have become the main demands of enterprises today.

[0004] Both the industrial community and the academic community have shown strong interest in the fields related to robots, and important progress has been made in methods based on machine learning and deep learning. Learning natural language features and visual features through neural networks has become a hot topic in current research. However, the robot debugging methods based on the current deep learning models have disadvantages such as poor robustness of task codes and the inability of debugging results to be applied to the on-site environment. The debugging methods based on this approach cannot be applied to the production environment of factories. Summary of the Invention

[0005] In view of the deficiencies of the current technology, the present invention proposes an industrial robot debugging method based on the combination of natural language and computer vision.

[0006] The general idea of the method of the present invention is as follows:

[0007] To address the deficiencies of existing neural network-based debugging methods, this application incorporates the environmental characteristics around the robot into the neural network, and at the same time, converts the natural language describing the robot code into semantic features and adds them to the network to enhance robustness. The present invention mainly consists of four parts: 1) Generate semantic information (Semantic Information) that represents the semantics of the language through a word2vec network and a linear layer for the natural language description, and then generate a semantic vector of a specified dimension through a linear network. 2) Extract the features of the industrial robot environment captured through a three-dimensional recurrent convolutional neural network (3D-RCNN) model. 3) Input the features obtained above into a long short-term memory network (LSTM) encoder to generate intermediate context, then initialize the GRU network with the semantic vector generated in 1), and use the intermediate context output in 2) as the input of the GRU network to output API recommendations through a recurrent neural network (RNN). 4) Input the text embeddings generated by the word2vec network for the natural language description into a long short-term memory network encoder and a long short-term memory network decoder in sequence, output an abstract syntax tree (AST) construction action sequence, and combine the API recommendations and the construction action sequence to generate industrial robot debugging code. 5) Add the debugging code in 4) to the task module of the robot program editor, and let the user perform final processing on the debugging code to complete the debugging of the industrial robot; the specific implementation steps are as follows:

[0008] Step (1): Generate semantic information;

[0009] Step (2): Extract environmental features;

[0010] Step (3): Generate API recommendations;

[0011] Step (4): Program the target code of the industrial robot;

[0012] Step (5): Complete robot debugging.

[0013] Another object of the present invention is to provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed in a computer, the computer is made to execute the above method.

[0014] Still another object of the present invention is to provide a computing device, including a memory and a processor. An executable code is stored in the memory, and when the processor executes the executable code, the above method is implemented.

[0015] Advantages of the present invention: The present invention solves the problems of low robot debugging efficiency and long robot program online cycle in the actual industrial environment. The industrial robot debugging method based on the combination of natural language and computer vision provided by the present invention has several main innovations: 1) Using a neural network-based method to complete robot debugging; 2) Combining computer vision-based image feature extraction and natural language-based target code generation; 3) Using a three-dimensional convolutional network to extract image features in the industrial environment; 4) Using a neural network with additional semantic information to generate API recommendations; 5) Combining an encoder-decoder network and API recommendations to jointly complete the robot debugging work.

[0016] The present invention does not require the user to have a lot of robot programming knowledge. The user only needs to input the natural language description for generating the robot code to achieve the task, which can provide a feasible debugging method for novice users without experience and can also be used as an auxiliary means for code engineers. The present invention uses advanced neural network-based robot debugging technology and integrates the industrial site environment into the neural network. Considering that the existing neural networks do not incorporate environmental features and do not consider the actual scenario in the industrial environment, the present invention solves the problems of high time consumption and low accuracy in robot debugging in the industrial environment. The present invention can effectively improve the development efficiency of robot debugging and reduce the deployment time of the robot production line in the industrial environment. Brief Description of the Drawings

[0017] Figure 1 : Schematic diagram of the method flow of the present invention;

[0018] Figure 2 : Sequence diagram for action construction;

[0019] Figure 3 : Encoding schematic diagram. Detailed Embodiments

[0020] The following further analyzes the present invention in combination with specific embodiments.

[0021] An industrial robot debugging method based on the combination of natural language and computer vision, as Figure 1 including the following steps:

[0022] Step (1), generating semantic information

[0023] 1-1 Input the natural language description of the robot action code into the word2vec network to generate text embeddings; specifically as follows:

[0024] The natural language instruction X = {x i |i = 1, 2,..., n} composed of i natural language words is used to generate a text embedding vector matrix E = {ei |i = 1, 2, …, n} ∈ R L×C ;

[0025] E = word2vec(X) (1)

[0026] where x i represents the i-th natural language word, and e i represents the i-th text embedding vector, word2vec() represents the word2vec network function, L is the number of text embeddings, and C is the embedding dimension;

[0027] 1 - 2 Generate semantic information through two cascaded linear layers A using the text embedding vector matrix generated in step 1 - 1, and then input the semantic information into linear layer B to output a semantic vector of a specified dimension; specifically as follows:

[0028] 1 - 2 - 1 Convert the text embedding vector matrix E generated in step 1 - 1 into a K - dimensional vector representation I; where the dimension of K is L × C;

[0029] K = L × C (2)

[0030] 1 - 2 - 2 Use the vector representation I as the input of two cascaded linear layers A to obtain semantic information S;

[0031] S = W 2 σ(W 1 + b 1 ) + b 2 (3)

[0032] where W 1 , W 2 , b 1 , b 2 are the trainable weights and biases of the corresponding linear functions of two cascaded linear layers A respectively, and σ is the ReLU activation function;

[0033] 1 - 2 - 3 Convert the semantic information S into a semantic vector of a specified dimension using linear layer B.

[0034] Step (2), Environmental Feature Extraction

[0035] Take the image data of the industrial environment where the industrial robot is located as the input of a three - dimensional recurrent convolutional network, and thus output environmental visual features; specifically as follows:

[0036] Input the industrial robot on - site environment image Q into the three - dimensional recurrent convolutional network to generate environmental visual feature f;

[0037] f = 3D - RCNN(Q) (4)

[0038] Among them, f is the visual feature representation of the image environment, and 3D-RCNN() is the three-dimensional recurrent convolutional network function.

[0039] Step (3), API recommendation generation

[0040] 3-1 Input the environmental visual features generated in step (2) into the long short-term memory network encoder for encoding to generate an intermediate semantic vector; specifically as follows:

[0041] Use the environmental visual feature f as the input of the long short-term memory network to generate a hidden state vector g, and then convert the hidden state vector g into an intermediate semantic vector;

[0042] g t = LSTM(f t , g t-1 ) (5)

[0043] Among them, g t is the hidden state vector at time t, g t-1 represents the hidden state vector at time t-1, f t is the environmental visual feature at time t, and LSTM() is the long short-term memory network function;

[0044] 3-2 Initialize the GRU network according to the semantic vector S generated in step (1), then convert the intermediate semantic vector generated in step 3-1 into a hidden state vector, and then input the hidden state vector into the initialized GRU network to generate an intermediate vector; specifically as follows:

[0045] 3-2-1 Initialize the GRU network according to the semantic vector generated in step (1);

[0046] 3-2-2 Convert the intermediate semantic vector generated in step 3-1 into a hidden state vector r, use the hidden state vector r as the input of the GRU network, and generate an intermediate vector k;

[0047] k t = GRU(r t , k t-1 ) (6)

[0048] Among them, k t is the intermediate vector at time t, GRU() is the GRU function, k t-1 is the intermediate vector at time t-1, and r t is the hidden state vector at time t;

[0049] 3-3 Use the intermediate vector k generated in step 3-2 to generate an API recommendation list through a recurrent neural network; specifically as follows:

[0050] The intermediate vector generated according to Step 3-2 is used as the input of the Recurrent Neural Network (RNN), and then the probability distribution of the API recommendation list is obtained through normalization by the softmax layer;

[0051] P = softmax(RNN(k)) (7)

[0052]

[0053] where P is the probability distribution of the API recommendation list, softmax() is the normalization exponential function, RNN() is the recurrent neural network function, τ i is the predicted probability of the i-th API recommendation output by the recurrent neural network, y i is the API recommendation in the actual situation, is the loss rate;

[0054] Step (4), target code programming of the industrial robot

[0055] 4-1 According to formula (1), use the word2vec network to convert the natural language description of the robot action code into the text embedding vector E;

[0056] 4-2 Use the text embedding vector E in Step 4-1 as the input of the Long Short-Term Memory network encoder, and then input the output into the Long Short-Term Memory network decoder to generate the construction action sequence of the AST tree; specifically as follows:

[0057] 4-2-1 Use the text embedding vector E as the input of the Long Short-Term Memory network encoder (LSTM-Encoder) to generate the hidden state vector h, and then convert the hidden state vector into the intermediate semantic vector;

[0058] h t = LSTM e (e t , h t-1 ) (9)

[0059] where h t is the hidden state vector at time t; LSTM e () represents the Long Short-Term Memory network encoder function, e t represents the embedding vector at time t, h t-1 represents the hidden state vector at time t-1;

[0060] 4-2-2 Convert the intermediate semantic vector into the hidden state vector θ, and use the hidden state vector θ as the input of the Long Short-Term Memory network decoder (LSTM-Decoder);

[0061]

[0062]

[0063] where θ t is the hidden state at time t, and θ t-1 is the hidden state at time t-1. LSTM d () represents the long short-term memory network decoder function, [:] represents the vector joint feature, and a t-1 represents the AST tree construction action at time t-1, is the attention vector of the hidden state at time t-1, β t is the vector containing the father boundary information in the derivation process, is the attention vector of the hidden state at time t, h t is the context vector, and W c is the connection layer function, and tanh() is the hyperbolic tangent function;

[0064] The 4-2-3 generates the construction action sequence of the AST tree, specifically:

[0065]

[0066] where ApplyConstr[c] is one of the action types in the AST tree (abstract syntax tree) construction action. This action can apply the construction operation c to the boundary field of the same type as c, and this field can be used to fill the node; p(a t = ApplyConstr[c]|a <t , x) represents the probability of the action ApplyConstr[c] under the action information and natural language description before time t, a t is the AST tree construction action representing time t, a <t represents the AST tree construction action information before time t, x is the natural language word, softmax() is the normalized exponential function, and a c is the AST tree construction action of the construction operation c, T is the vector transpose, and W is the connection layer function, is the attention vector of the hidden state at time t;

[0067] p(a t = GenToken[v]|a <t , x)

[0068] = p(gen|a t , x)p(v|gen, a t , x)+

[0069] p(copy|a t , x)p(v|copy, a t,x) (13)

[0070] where GenToken[v] is another type of action in the AST tree construction action, which can fill the AST tree boundary field with code v; p(a t = GenToken[v] | a <t ,x) represents the probability of the action GenToken[v] under the action information and natural language description before the time slice, and a t represents the AST tree construction action at time t, and a <t represents the AST tree construction action information before time t, x is a natural language word, gen is the generation operation, and copy is the copy operation;

[0071] Finally, all the construction actions of the AST tree are obtained from formulas (12)-(13), and then the construction action sequence of the AST tree is obtained as Figure 2 ;

[0072] 4-2-4 Combine the API recommendation list generated in step (3) and the construction action sequence of the AST tree generated in step 4-2-3 to output the final robot code; specifically:

[0073] Sort according to the probability distribution of the API recommendation list generated in step (3) to obtain the API recommendation with the highest probability, and embed this API into the construction action sequence of the AST tree generated in step 4-2-3 to obtain the optimized construction action sequence of the AST tree; then generate an abstract syntax tree (AST) from the optimized construction action sequence of the AST tree, and then convert the abstract syntax tree into the debugging code of the industrial robot in the current industrial environment, such as Figure 3 ;

[0074] Step (5), complete robot debugging

[0075] Input the above debugging code into the task module of the robot program editor to complete the debugging of the industrial robot.

Claims

1. An industrial robot debugging method based on the combination of natural language and computer vision, characterized in that it includes the following steps: Step (1), generating semantic information from the natural language description of the robot action code; Step (2), environmental feature extraction Taking the image data of the industrial environment where the industrial robot is located as the input of a three-dimensional recurrent convolutional network, so as to output environmental visual features; Step (3), API recommendation generation 3-1 Using the environmental visual features generated in step (2) to be input into a long short-term memory network encoder for encoding to generate an intermediate semantic vector; specifically as follows: Taking the environmental visual feature f as the input of the long short-term memory network to generate a hidden state vector g, and then converting the hidden state vector g into an intermediate semantic vector; g t = LSTM(f t , g t-1 ) (5) where g t is the hidden state vector at time t, and g t-1 represents the hidden state vector at time t - 1, f t is the environmental visual feature at time t, and LSTM() is the long short-term memory network function; 3-2 Initializing a GRU network according to the semantic vector S generated in step (1), then converting the intermediate semantic vector generated in step (3-1) into a hidden state vector, and then inputting the hidden state vector into the initialized GRU network to generate an intermediate vector; specifically as follows: 3-2-1 Initializing a GRU network according to the semantic vector generated in step (1); 3-2-2 Converting the intermediate semantic vector generated in step 3-1 into a hidden state vector r, taking the hidden state vector r as the input of the GRU network, and generating an intermediate vector k; k t = GRU(r t , k t-1 ) (6) where k t is the intermediate vector at time t, GRU() is the GRU function, and k t-1 is the intermediate vector at time t-1, and r t is the hidden state vector at time t; 3-3 Using the intermediate vector k generated in step 3-2 to generate an API recommendation list through a recurrent neural network; specifically as follows: Taking the intermediate vector generated in step 3-2 as the input of a recurrent neural network RNN, and then obtaining the probability distribution of the API recommendation list through softmax layer normalization; P = softmax(RNN(k)) (7) where P is the probability distribution of the API recommendation list, softmax() is the normalized exponential function, RNN() is the recurrent neural network function, and τ i is the predicted probability of the i-th API recommendation output by the recurrent neural network, and y i is the API recommendation in the actual situation, is the loss rate; Step (4), target code programming of the industrial robot 4-1 Using the word2vec network to convert the natural language description of the robot action code into a text embedding vector E according to formula (1); 4-2 Using the text embedding vector E in step 4-1 as the input of a long short-term memory network encoder, and then inputting the output into a long short-term memory network decoder to generate a construction action sequence of an AST tree; specifically as follows: 4-2-1 Taking the text embedding vector E as the input of a long short-term memory network encoder (LSTM-Encoder) to generate a hidden state vector h, and then converting the hidden state vector into an intermediate semantic vector; h t = LSTM e (e t , h t-1 ) (9) where h t is the hidden state vector at time t; LSTM e () represents the long short-term memory network encoder function, e t represents the embedding vector at time t, h t-1 represents the hidden state vector at time t-1; 4-2-2 Converting the intermediate semantic vector into a hidden state vector θ, and taking the hidden state vector θ as the input of a long short-term memory network decoder (LSTM-Decoder); where θ t is the hidden state at time t, and θ t-1 is the hidden state at time t-1. LSTM d () represents the long short-term memory network decoder function, [:] represents the vector joint feature, and a t-1 represents the AST tree construction action at time t-1, represents the attention vector of the hidden state at time t-1, and β t is the vector containing the father boundary information in the derivation process, is the attention vector of the hidden state at time t, and h t is the context vector, W c is the connection layer function, and tanh() is the hyperbolic tangent function; 4-2-3 Generating a construction action sequence of an AST tree, specifically: where ApplyConstr[c] is one of the action types in the AST (Abstract Syntax Tree) construction action, which applies the construction operation c to the boundary field of the same type as c, and this field is used to fill the node; p(a t = ApplyConstr[c]|a <t , x) represents the probability of the action ApplyConstr[c] under the action information and natural language description before time t, a t represents the AST construction action at time t, a <t represents the AST construction action information before time t, x is the natural language word, softmax() is the normalized exponential function, a c is the AST construction action of the construction operation c, T is the vector transpose, W is the connection layer function, is the attention vector of the hidden state at time t; p(a t = GenToken[v]|a <t ,x) = p(gen|a t , x)p(v|gen, a t , x) + p(copy|a t ,x)p(v|copy,a t ,x) (13) Among them, GenToken[v] is another type of action in the AST tree construction action, which fills the AST tree boundary field with the code v; pa( t = GenToken[v] | a<t , x) represents the probability of the action GenToken[v] under the action information and natural language description before the time slice, a t is the AST tree construction action representing the t-th moment, a <t represents the AST tree construction action information before the t-th moment, x is the natural language word, gen is the generation operation, and copy is the copy operation; Finally, obtaining all construction actions of the AST tree from formulas (12)-(13), and further obtaining a construction action sequence of the AST tree; 4-2-4 Combining the API recommendation list generated in step (3) and the construction action sequence of the AST tree generated in step 4-2-3 to output the final robot code; Step (5), completing robot debugging Using the code obtained in step 4-2-4 to be input into the task module of the robot program editor to complete the debugging of the industrial robot.

2. The method according to claim 1, wherein step (1) is specifically as follows: 1-1 Input the natural language description of the robot action code into the word2vec network to generate text embeddings; 1-2 Use the text embedding vector matrix generated in step 1-1 to generate semantic information through two cascaded linear layers A, and then input the semantic information into linear layer B to output a semantic vector of a specified dimension.

3. The method according to claim 2, wherein step 1-1 is specifically as follows: The natural language instruction X = {x i | i = 1, 2, …, n} consisting of i natural language words is used to generate a text embedding vector matrix E = {e i | i = 1, 2, …, n} ∈ R L×C ; E = word2vec(X) (1) where x i represents the i-th natural language word, and e i represents the i-th text embedding vector, word2vec() represents the word2vec network function, L is the number of text embeddings, and C is the embedding dimension.

4. The method according to claim 2, wherein step 1-2 is specifically as follows: 1-2-1 Convert the text embedding vector matrix E generated in step 1-1 into a K-dimensional vector representation I; where the dimension of K is L×C; K = L×C (2) 1-2-2 Use the vector representation I as the input of two cascaded linear layers A to obtain semantic information S; S = W 2 σ(W 1 + b 1 ) + b 2 (3) where W 1 and W 2 , b 1 and b 2 are the trainable weights and biases of the linear functions corresponding to two cascaded linear layers A respectively, and σ is the ReLU activation function; 1-2-3 Convert the semantic information S into a semantic vector of a specified dimension using linear layer B.

5. The method according to claim 1, wherein step (2) is specifically as follows: Input the industrial robot on-site environment image Q into a three-dimensional recurrent convolutional network to generate environmental visual features f; f = 3D-RCNN(Q) (4) where f is the image environmental visual feature representation, and 3D-RCNN() is the three-dimensional recurrent convolutional network function.

6. The method according to claim 1, wherein step 4-2 is specifically: Sort according to the probability distribution of the API recommendation list generated in step (3) to obtain the API recommendation with the highest probability, embed this API into the construction action sequence of the AST tree generated in step 4-2-3 to obtain the optimized construction action sequence of the AST tree; then generate an abstract syntax tree (AST) from the optimized construction action sequence of the AST tree, and then convert the abstract syntax tree into the debugging code of the industrial robot in the current industrial environment through a conversion function.

7. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, the computer is made to execute the method according to any one of claims 1-6.

8. A computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1-6 is implemented.

Images