Human work intention recognition method for human-robot collaborative assembly

Abstract

The present invention discloses a method for identifying personnel's work intention in human-machine collaborative assembly, which is used to identify the work intention of personnel in the human-machine collaborative assembly process, and includes the following steps: collecting RGB-D images of assembly environment scenes; constructing an assembly scene perception network, and training the assembly scene perception network to detect various elements in the assembly scene; wherein the various elements include personnel, machines, and various workpieces; inputting the image into the assembly scene perception network for detection, and extracting the visual features and spatial features of each element; constructing a graph attention network, using visual features to initialize the features of each node, using spatial features to initialize the features of the edges between nodes, dynamically aggregating the features of each node, and inferring the work intention of personnel. The present invention improves the accuracy and generalization of work intention recognition in the work process of personnel by effectively utilizing contextual information in the assembly environment, including visual information and spatial information.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-robot collaboration and intelligent manufacturing, and in particular relates to a method for identifying personnel's work intentions in human-robot collaborative assembly. Background Art

[0002] In human-machine collaborative tasks, workers have higher flexibility and skilled assembly skills, while industrial robots have the characteristics of faster speed, higher load capacity, and more precise positioning. The complementary advantages of both parties can bring new possibilities for the efficient completion of complex assembly tasks.

[0003] Currently, one of the many challenges facing human-machine collaborative assembly is to enable robots to recognize the work intentions of humans, so that the robots can actively adjust their assembly tasks and provide timely assistance to humans during the assembly process, ultimately achieving smooth human-machine collaborative assembly and improving production efficiency. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a method for identifying personnel's work intention in human-machine collaborative assembly, so as to realize the identification of personnel's work intention in complex assembly scenarios.

[0005] To achieve the above object, the present invention provides a method for identifying personnel work intentions in human-machine collaborative assembly, which is used to identify personnel work intentions in the assembly process, and comprises the following steps:

[0006] 1) Through the Kinect infrared camera, the assembly environment scene information is collected, the visual information and depth information in the assembly environment are recorded, and the assembly scene picture is input into the assembly scene perception network in the form of RGB-D video stream;

[0007] 2) The assembly scene perception network detects the various elements existing in the assembly scene.

[0008] 3) The visual feature information of each element is extracted from the assembly scene perception network, and the bounding box of each element is detected and combined with the depth information to obtain the three-dimensional spatial feature.

[0009] 4) Using the detection results of the scene perception network, a graph attention network is constructed to dynamically aggregate the visual feature information and three-dimensional spatial feature information of people and work objects in the assembly environment.

[0010] 5) The attention mechanism is used to suppress the impact of the "noisy" detection results of the scene perception network on the node feature aggregation, and ultimately realize the recognition of personnel's work intentions.

[0011] In step 1), the environment scene perception network is assembled by the following:

[0012] Use the Kinect camera to collect visual information of the assembly scene and create a dataset.

[0013] A scene perception network is built and trained based on the Faster-RCNN neural network. In order to detect targets in the human-machine collaborative assembly environment, the categories of training target detection mainly include personnel, robot arms, various workpieces (bases, gears, shafts), etc.

[0014] In step 2), after using the Faster R-CNN neural network to obtain the bounding box of the personnel and the work object in the assembly scene, the pixel coordinates (u, v) at the center of the bounding box are calculated, and then combined with the RGB-D image to obtain the depth information d of the current pixel.

[0015] Use the built-in function of the Kinect camera to complete the conversion from the image plane coordinate system to the camera coordinate system, and obtain the coordinates of the pixel point in the Kinect camera coordinate system.

[0016] The spatial features can be divided into three-dimensional relative spatial features F rp and the three-dimensional relative distance feature F rd .

[0017] Three-dimensional relative spatial feature F rp :

[0018]

[0019] 3D relative distance feature F rd :

[0020]

[0021] Among them, (x i ,y i ,z i )、(x j ,y j ,z j ) represent the three-dimensional coordinates of the person and the work object in the Kinect camera coordinate system, X k , Y k , Z k are the maximum coordinate ranges in the camera coordinate system.

[0022] In step 3), the scene perception network performs target detection on personnel and various work objects in the assembly environment, obtains the bounding boxes of personnel and various work objects, and extracts the visual features of personnel and various work objects in the assembly environment. The 256-dimensional visual features of each detection result in the assembly scene are extracted from the last fully connected layer (FC7) in the neural network.

[0023] The spatial features will be used to initialize the features of the edges between two nodes in the graph attention network, while the visual features will be used to initialize the features of the nodes in the graph attention network.

[0024] In step 4), the graph G is defined as G = (V, E), where V is the set of nodes and E is the set of edges. The characteristics of the nodes and edges in the graph are h v and h e . Let the i-th node in the graph be v i ∈V,e i,j =(v i ,v j )∈E is the node v i and v j The directed edge of .

[0025] Note node v i The characteristics are To update the features of a node, we need to aggregate the features of the nodes around it, using the update function f update (·) Updated node features It can usually be defined as:

[0026]

[0027] in Represents node v i Aggregation result of the features of surrounding nodes, Can be defined as:

[0028]

[0029] Where N i Represents the node v i The set of adjacent nodes. At the same time, the aggregation function f aggregate (·) is to average the feature information of surrounding nodes:

[0030]

[0031] Use the edge function f edge (·) Encode the relationship features between two connected nodes:

[0032]

[0033] in, Represents node v j Features, Represents node v j to v i The characteristics of the directed edges.

[0034] In step 5), the attention mechanism is used to suppress the impact of irrelevant detection results on node feature aggregation:

[0035]

[0036] where α ij Represents node v j For node v i The soft weight after normalization, f attn (·) is the attention function, N i Represents the node v i Adjacent nodes v o Then apply the weighted sum using the update function f update (·) to update the features of each node, and finally obtain the i The potential relationship features of adjacent nodes are concatenated into

[0037]

[0038] Where N i Represents the node v i The set of adjacent nodes. The characteristics of the last updated node

[0039]

[0040] So far, we have obtained a graph attention network with an attention mechanism.

[0041] Through the above steps, the splicing features for prediction are finally constructed:

[0042]

[0043] in, For node v i Features, For node v j Features, For node v j To node v i The directed edge f ij characteristics.

[0044] To calculate the action classification score s a ∈R k , where k represents the total number of possible actions. Apply the edge reading function f readout (·) Read the feature vectors of the edges between nodes and apply a binary Sigmoid classifier s to each action category a =sigmoid(f readout (ζij )).

[0045] The final task intention classification can be obtained by chain multiplication:

[0046] S R =s h *s o *s a

[0047] where s a is the confidence of the operation action, s h is the confidence level of the detector and s o The confidence of the detected object is given by the detection results of the scene perception network above.

[0048] A multi-class cross entropy loss (BCE) is used in training so that the loss is proportional to the task intent score s. ij and the actual job intention label Minimize between:

[0049]

[0050] Where N is the number of all personnel-task pairs in a mini-batch, and k represents the number of task actions. During the network training process, relevant parameters such as weights and biases are continuously updated until the loss function L becomes small enough, so that the best model for personnel task intention recognition can be obtained.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] The present invention uses a scene perception network to extract contextual information of various elements in the production space, and dynamically aggregates the contextual information of personnel and work objects in the assembly environment based on a graph attention network, thereby inferring the work intention of personnel in complex assembly scenarios. Compared with the prior art, the present invention improves the accuracy and generalization of work intention recognition during the work of personnel by effectively utilizing the contextual information in the assembly environment, including visual information and spatial information, and provides a basis for the allocation and subsequent optimization of human-machine collaborative disassembly tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of the method for identifying personnel's work intention for human-machine collaborative assembly.

[0054] Figure 2 This is the structural diagram of the deep learning network.

[0055] Figure 3 It is the loss function curve of the graph attention network.

[0056] Figure 4 Multi-label mixture matrix (19 categories) for job intent recognition. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0058] The present invention proposes a method for identifying the work intention of personnel in the process of human-robot collaborative assembly, which is applicable to industrial robots equipped with depth cameras. This example is based on the industrial robot ABB IRB1200 7 / 0.7 and installs a depth camera Kinect V2 in the workspace of the industrial robot for perception. Its optimization goal is to accurately infer the work intention of personnel in the process of human-robot collaborative assembly and then cooperate with personnel to improve production efficiency. Figure 1 As shown, the method mainly includes the following steps:

[0059] 1) Based on the depth camera Kinect V2, the perception and monitoring of the industrial human-machine collaborative assembly space is realized, and the assembly environment scene information is input into the assembly scene perception network in the form of RGB-D video stream; RGB records the visual information in the assembly environment, and D records the depth information.

[0060] 2) The assembly scene perception network detects workers, robots, workpieces and other targets in the assembly scene, and outputs the bounding box (Bounding Box) and confidence score (Score) of each target.

[0061] Building a scene perception network based on Faster-RCNN neural network, such as Figure 2 As shown in the figure, the scene perception network is trained and assembled, and the threshold of target detection is set to 0.95. While detecting the target, the visual features of the target are extracted from the last fully connected layer (FC7) of the network.

[0062] 3) Sense data fusion processing, extract 3D spatial features, extract the target’s depth information d based on the bounding box detection results, and use the Kinect camera’s built-in function to complete the conversion from the image plane coordinate system to the pixel coordinate system. Obtain the coordinates of the pixel point in the Kinect camera coordinate system (xi, yi, zi). The 3D spatial features can be divided into 3D relative spatial features F rp and the three-dimensional relative distance feature F rd :

[0063]

[0064] Among them, (x i ,y i ,z i )、(x j ,y j ,z j ) represent the three-dimensional coordinates of the person and the work object in the Kinect camera coordinate system. k , Y k , Z k are the maximum coordinate ranges in the camera coordinate system.

[0065] 4) Use graph attention to dynamically aggregate visual information and spatial information, dynamically update the weights of the graph attention network, and the graph attention network loss function curve is as follows: Figure 3 shown.

[0066] 5) The attention mechanism is used to suppress the "noisy" detection results in the scene perception network, and finally the graph attention network is used to infer the work intention of the personnel.

[0067] The above steps are steps for realizing personnel work intention recognition in an embodiment of the present invention.

[0068] Among them, in step 2), the visual and depth information is collected mainly by using a Kinect camera at a sampling rate of 30 frames per second to collect the visual information and depth information of the assembly environment in the RGB-D format during the assembly process.

[0069] In the feature extraction process based on the scene perception network in step 3), the visual information extracted by the scene perception network will be used to initialize the features of the nodes in the graph attention network, and the relative spatial features will be used to initialize the features of the edges between the nodes.

[0070] In calculating the intention score S R The calculation formula is as follows:

[0071] S R =s h *s o *s a

[0072] Among them, s h is the score of detected people, s o is the score of detected objects, s a is the score of the detected action.

[0073] The following is an analysis and explanation of a certain type of gear assembly. The example contains 19 assembly intentions. For this assembly task, the assembly workpieces are inspected and classified according to the scheme shown in step 1). The specific assembly work intentions of personnel are shown in Table 1.

[0074] Table 1 Gear collaborative assembly personnel work intention chart

[0075]

[0076] Note: “√” means that the operation object has the operation action, and vice versa without “√”.

[0077] The assembly scene perception network detects the personnel, robots, and workpieces (gears, bases, etc.) in the assembly scene and extracts the corresponding visual features and relative spatial features.

[0078] The object detection results together with the extracted visual features and spatial features are fed into the graph attention network for reasoning.

[0079] The present invention is based on graph attention network to identify personnel work intention in human-machine collaborative environment. Its optimization goal is to improve the accuracy and generalization of personnel work intention recognition in complex assembly scenarios. The multi-label mixed matrix of work intention recognition is as follows: Figure 4 shown.

[0080] It will be easily understood by those skilled in the art that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying personnel work intentions in human-machine collaborative assembly, used to identify personnel work intentions in the human-machine collaborative assembly process, characterized in that: The following steps are involved: Collect RGB-D images of assembly environment scenes; Constructing an assembly scene perception network and training the assembly scene perception network to detect various elements in the assembly scene, including personnel, machines, and various workpieces; The image is input into the assembly scene perception network for detection, and the visual features and spatial features of each element are extracted; Construct a graph attention network, use visual features to initialize the features of each node, use spatial features to initialize the features of the edges between nodes, dynamically aggregate the features of each node, and infer the operator's work intention; Among them, the dynamic aggregation of node features and the inference of personnel's work intentions include: First, aggregate nodes v i The features of the surrounding nodes are used to update the node; Use the edge function fedge(·) to connect two nodes v i and v j Encode the relationship features between: In the formula, h v and h e Represent the characteristics of nodes and edges respectively, Represents node v j to v i The characteristics of the directed edges of Then the attention mechanism is used to suppress the impact of irrelevant detection results on node feature aggregation: In the formula, α ij Represents node v j For node v i The soft weight after normalization, f attn (·) is the attention function, N i Represents the node v i Adjacent nodes v o A collection of; Then apply the weighted sum using the update function f update (·) to update the characteristics of each node, All connections to node v i The potential relationship features of adjacent nodes are concatenated into: Then update the node features Get the graph attention network with attention mechanism: Finally, the updated features are used to construct the concatenated features for prediction: In the formula, For node v i Features, For node v j Features, For node v j To node v i The directed edge f ij Features; The binary Sigmoid classifier is applied to calculate the action classification score and obtain the personnel's work intention.

2. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 1, characterized in that: The RGB-D images of the assembly environment scene are collected by the Kinect infrared camera.

3. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 1, characterized in that: Build a scene perception network based on Faster-RCNN neural network.

4. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 3, characterized in that: The visual features of each element are extracted from the last fully connected layer.

5. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 1, characterized in that: The assembly scene perception network is used to obtain the bounding box of each element in the assembly scene, and the pixel coordinates of the center of the bounding box of each element are calculated; combined with the depth information of the image, the image plane coordinate system is converted to the camera coordinate system, thereby obtaining the spatial characteristics of each element.

6. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 5, characterized in that: The spatial features are divided into three-dimensional relative spatial features F rp and the three-dimensional relative distance feature F rd : In the formula, (x i ,y i ,z i ) represents the three-dimensional coordinates of the person in the camera coordinate system, (x j ,y j ,z j ) represents the three-dimensional coordinates of the machine and various workpieces in the camera coordinate system, X k , Y k , Z k are the maximum coordinate ranges in the camera coordinate system.

7. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 1, characterized in that: The task intention classification is obtained by chain multiplication: S R =s h *s o *s a Among them, s a is the classification score of the operation action, s h is the confidence of the inspector, s o To test the confidence of machines and various workpieces.

8. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 7, characterized in that: Use multi-class cross entropy loss BCE so that the loss L is in the task intent score s ij and the actual job intention label Minimize between: In the formula, N represents the number of all pairs of personnel, machines and various types of workpieces, and K represents the number of operating actions.

9. The method for identifying personnel work intentions for human-machine collaborative assembly according to claim 1, characterized in that: The images are input into the assembly environment perception network in the form of video streams to infer the operator's work intentions in real time.

Images