Human-computer collaboration method and human-computer collaboration system based on neural network
Through multi-view RGB video data and bone key point recognition algorithm, combined with HRI-GCN action recognition network, the problem of single-channel camera incomplete viewing angle and robots being unable to predict the next task is solved, efficient human-machine collaboration is achieved, and the recognition accuracy is improved. Sex and collaborative efficiency.
Patent Information
- Application Number
- CN202310168810.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-02-27
AI Technical Summary
The existing neural network-based visual human-computer collaboration system has a single-channel depth camera with limited perspective angle and is easily obstructed, and has low recognition accuracy; the robot cannot predict the worker's next action intention and has low collaboration efficiency; the robot and human tasks are relatively independent, It is difficult to build efficient collaboration scenarios.
Using multi-view RGB video data and bone key point recognition algorithm, action prediction and recognition are carried out through the HRI-GCN action recognition network, combined with graph convolution network, long and short memory network and attention mechanism, a complete spatial coordinate of the key point of the human body skeleton is constructed. Sequence, formulate a human-machine task correspondence table to achieve active assistance from robots.
It improves the accuracy and collaboration efficiency of action recognition, reduces environmental information interference, realizes the robot's prediction and task allocation of workers' next action intentions, and improves the flexibility and production efficiency of human-machine collaborative assembly.
Smart Images

Figure CN116189054B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a human-machine collaboration method and a human-machine collaboration system based on a neural network, and belongs to the technical field of motion recognition of human-machine collaborative operation. Background Art
[0002] Human-machine collaboration is a way of working that achieves collaborative operation through the cooperation between humans and robots to complete related tasks. The main concept of human-machine collaboration is to combine human capabilities with the advantages of robots. Existing robots generally have defects such as long teaching cycles, difficulty in understanding abstract commands, and lack of online perception capabilities, which restrict the intelligent application of robots in the industrial field. With the diversification of products in high-end manufacturing and the development of artificial intelligence, human-machine collaboration will become the development trend of the next generation of industrial applications of robots. Human-machine collaboration is committed to allowing robots to actively collaborate. Under the condition that robots have not yet reached a high level of intelligence, the most effective method is to first perceive and identify human behavior, and then provide assistance to humans in a preset way.
[0003] In recent years, human-machine collaboration has become a hot topic in the field of intelligent manufacturing. At present, research on robot applications is mainly concentrated in the fields of handling, inspection and assembly. In traditional manufacturing scenarios, for safety reasons, human operators and robots are separated in different work areas, and each completes their assigned tasks independently. In an ideal human-machine collaboration system, the high precision, strength and repeatability of robots can be combined with the high flexibility and adaptability of human operators to improve production efficiency and reduce the burden on workers. In order to achieve human-machine collaboration, the robot needs to recognize the worker's actions, and then infer the worker's operating intentions, and provide corresponding assistance based on the correspondence between the operating intentions and the robot's actions; thereby improving the initiative and work efficiency of people in human-machine collaboration.
[0004] The invention patent of Northwestern Polytechnical University, "Action Recognition Method Based on Segmented Human Body Model for Human-Machine Collaboration" (authorization number CN109086706B), discloses an action recognition method based on segmented human body model for human-machine collaboration. The method uses Microsoft Kinect V2 sensor to collect the spatial coordinates of human skeletal joints, extracts key frames respectively using Boolean feature matrix, and uses feature vector and dynamic time warping algorithm (DTW algorithm) to perform template matching on the input respectively to complete the recognition of human action. However, the system only uses a single-channel depth camera as input for target detection, and the action is easily blocked, resulting in incomplete viewing angle and reduced recognition accuracy; in terms of recognition algorithm, a template matching method based on DTW algorithm is used, and the algorithm is not generalizable and accurate enough. It is necessary to collect action of a specific operator before recognition, and the transferability of the recognition model is poor.
[0005] The invention patent of South China University of Technology, "Robot Perception and Understanding Method Based on Human-Machine Collaboration" (grant number CN107150347B), discloses a robot perception and understanding method based on human-machine collaboration, which realizes intention recognition in a multimodal manner through gesture recognition and natural language processing. Human-machine communication and assistance are carried out through natural and efficient human interaction to improve the flexibility, intelligence and adaptability of robot perception and understanding. However, this method requires multimodal information fusion as input, and the input data of the recognition model is voice and single-view RGB video. This data is susceptible to noise or background interference in industrial scenarios, which is not conducive to accurate recognition of the operator's intention. In terms of robot task allocation, it relies on reinforcement learning and natural language processing, and inputs the robot's task content and environmental information through oral explanation. This requires a large amount of empirical data and the learning process is uncontrollable. It is difficult to accurately construct the correspondence between the tasks that the robot needs to handle and the tasks handled by humans in human-machine collaboration.
[0006] In summary, the existing neural network-based visual human-machine collaboration system has the following three main problems: First, the human motion recognition part usually uses the data of a single-channel depth camera as the input of the motion module, which has a limited field of view and is easily blocked, and cannot effectively recognize the worker's movements. Second, under the current conditions, the robot's assistance to workers is limited to receiving fixed instructions and completing the current task under the worker's operation. The robot cannot predict the worker's next action intention and actively assist in the next task. Third, the robot's tasks and the human's tasks are usually independent of each other, and are limited to completing their respective tasks separately and then combining them. There is a lot of waiting time in the process. It is difficult to build a correspondence between the tasks that the robot needs to handle and the tasks that the worker needs to handle in an efficient collaboration scenario. Summary of the invention
[0007] In order to solve the problems existing in the above technologies, the present invention provides a human-machine collaboration method and a human-machine collaboration system based on a neural network, and the technical solution is as follows:
[0008] The first object of the present invention is to provide a human-machine collaboration method based on a neural network, comprising:
[0009] Step 1: Obtain video data of human operation in human-machine collaborative operation tasks from multiple perspectives, and use the 2D skeleton key point recognition model openpose to complete skeleton key point recognition for multiple sets of video data from different perspectives;
[0010] Step 2: The skeleton space coordinates of multiple sets of different viewpoints obtained by recognition are fused and reconstructed to obtain a complete and accurate sequence of human skeleton key point space coordinates;
[0011] Step 3: Input the spatial coordinate sequence of the key points of the human skeleton into the trained custom HRI-GCN action recognition network for action prediction and recognition;
[0012] Step 4: Develop a workflow and a human-machine task correspondence table based on collaborative tasks;
[0013] Step 5: According to the identified action category, the set collaborative task process and the human-machine task correspondence table, specify the robot to transfer the workpiece or tool required for the next step;
[0014] The HRI-GCN action recognition network is constructed based on graph convolutional network GCN, long short-term memory network LSTM and attention mechanism.
[0015] Optionally, the construction process of the HRI-GCN action recognition network includes:
[0016] Step 21: Use the multidimensional array of spatial coordinates of the skeleton key points as the original input of the graph convolutional layer GCN network, use the skeleton key points as the nodes of the graph, and convert them into the adjacency matrix A and the degree matrix D as the input of the graph convolution for extracting spatial features;
[0017] Step 22: Add the attention mechanism to the graph convolutional layer GCN network to form the AGCN module;
[0018] Step 23: Using the spatial feature matrix sequence obtained by extracting single-frame spatial features through the AGCN module as the input of the LSTM module to extract the temporal features of the action;
[0019] Step 24: alternately stack the spatial graph convolution AGCN module and the temporal gated unit LSTM module to form a spatiotemporal convolution unit STunit. The HRI-GCN action recognition network includes 10 spatiotemporal convolution units STunit1-Stunit10 and a final fully connected layer FC1, and finally connects the softmax classifier to form an action recognition network.
[0020] The input of the AGCN module is a 4-dimensional structure matrix [N*M, C, T, V], and the input of the LSTM module is also a 4-dimensional structure matrix [N*M, C, T, V], but only the time series information of T is extracted; the outputs of the AGCN module and the LSTM module both maintain the original structure, which is a 4-dimensional structure matrix [N*M, C, T, V]; finally, the fully connected layer FC1 and the softmax classifier are connected to output the action label;
[0021] Among them, in the 4-dimensional structure matrix [N*M,C,T,V], N is the batch size, M is the number of people, C is the number of input channels, T is the number of time steps, and V is the number of skeleton key points.
[0022] Optionally, the input of the HRI-GCN action recognition network is an action matrix sequence composed of human skeleton points, and a 25×3 matrix is used to represent the spatial coordinates of 25 key points of the human skeleton; the output of the HRI-GCN action recognition network uses an N×1 matrix to represent the probability that the input belongs to a certain action.
[0023] Optionally, the spatiotemporal convolution unit STunit in the HRI-GCN action recognition network adopts a Relu function as an activation function.
[0024] Optionally, the training process of the HRI-GCN action recognition network includes:
[0025] Step 1: Use an RGB camera to capture video data of human operations in human-machine collaborative operation tasks, standardize the number of captured video frames, and save them with action labels;
[0026] Step 2: For the human action video collected in step 1, use the 2D skeleton key point recognition model openpose to complete the skeleton key point recognition, generate the spatial coordinate data of each key point, store it as an array, and complete the preliminary establishment of the data set;
[0027] Step 3: constructing the spatial coordinate data of the skeleton key points corresponding to the different actions obtained in step 2 into a human action skeleton dataset;
[0028] Step 4: Take pictures and identify the coordinates of each skeleton key point when there is no action as a reference, subtract the spatial coordinates in the data set from the reference coordinates when there is no action, obtain the feature vector array, and select a reasonable threshold to use the threshold to eliminate interference data;
[0029] Step 5: Use the adjacency matrix A k Sum degree matrix D k The spatial position of each key point representing the skeleton key point and the topological relationship between nodes are constructed into a data set by constructing the spatial position information and the topological connection relationship of the skeleton key points representing the movement;
[0030] Step 6: The data set is divided into a training set and a test set in a ratio of 8:2;
[0031] Step 7: Using the training set obtained in step 6, train the HRI-GCN action recognition network until the model converges, and then test the model performance on the test set.
[0032] Optionally, the loss function of the HRI-GCN action recognition network is:
[0033]
[0034] Among them, N represents the total number of samples used in the training process, M represents the type of action that needs to be predicted and recognized, and y ik is the symbolic function representing the target action, p ik Represents the probability that sample i output by the final fully connected layer belongs to action category k.
[0035] Optionally, the accuracy function of the HRI-GCN action recognition network is:
[0036]
[0037]
[0038] Among them, T P Represents the number of correct recognitions of a certain type of action, F P represents the number of errors in the recognition of a certain type of action, M represents the type of action that needs to be predicted and recognized, and P k Represents the accuracy of a certain type of action recognition, W k Represents the weight of this type of action data in the overall data.
[0039] Optionally, the process of formulating a workflow and a human-machine task correspondence table according to the collaborative task in step 4 includes:
[0040] Step 41: For the human-machine collaborative task, formulate a workflow to divide the human work tasks and the robot collaborative tasks;
[0041] Step 42: Match the actions in the human work task with the actions that the robot needs to collaborate with, such as transferring workpieces or tools, and match the identification tags corresponding to the actions with the robot collaborative tasks to form a human-machine task allocation table.
[0042] The second object of the present invention is to provide a human-machine collaboration system based on a neural network, which is used to implement the above-mentioned human-machine collaboration method based on a neural network, comprising:
[0043] Multi-view action acquisition module, used to capture operator action videos from multiple angles;
[0044] The action intention recognition module obtains the spatial coordinate sequence of the skeleton key points based on the action video of the operator, inputs the sequence into the HRI-GCN action recognition network, and outputs the final predicted recognition feature vector value, wherein the feature vector value includes the probability that the input action belongs to each type of action;
[0045] The intention prediction and task assignment module is used to specify the workpiece or tool required for the robot to transfer the next step based on the intention recognition results and the human-machine task correspondence table;
[0046] The human-machine collaboration module transfers the tool or workpiece to the operator according to the instructions issued by the intention prediction and task allocation module.
[0047] Optionally, the multi-view motion acquisition module collects motion data of the worker during operation from the front and side respectively.
[0048] The beneficial effects of the present invention are:
[0049] (1) The present invention realizes the fusion reconstruction of multi-view skeleton information through multi-view RGB video data and skeleton key point estimation algorithm, solves the problem that single-channel camera is easily blocked, resulting in incomplete angle of view of collected action signals, and reduces the interference of environmental information. By using a multi-view RBG camera to collect video data, and cooperating with the skeleton key point recognition algorithm, the skeleton information of multi-angle video data is recognized and fused and reconstructed to obtain the complete 3D spatial coordinates of the human skeleton. At the same time, the present invention can not only effectively avoid the occlusion caused by the incomplete angle of view of the single-channel camera and the interference of ambient light in the industrial environment, but also generate a sequence of skeleton key point spatial coordinates, reducing the redundant information of the input model.
[0050] (2) The present invention solves the problem that the robot cannot predict and actively assist the worker's next task by customizing the HRI-GCN action recognition network. The HRI-GCN action recognition network uses the position changes of the key points of the skeleton during continuous actions as input to achieve dynamic recognition of operation actions, and uses the timing characteristics of the recurrent neural network to complete the prediction of the early actions of the worker during the operation, identify the worker's next action intention, and further improve the accuracy of action recognition and the efficiency of collaboration.
[0051] (3) The present invention solves the problem that in collaborative tasks, the tasks of robots and humans are relatively independent, making it difficult to give full play to the advantages of human-machine collaboration by creating a comparison table of human actions and robot tasks. The assembly process is decomposed into sub-assembly tasks and single-step operation tasks, and they are allocated according to work characteristics. A comparison table of worker operation tasks and robot task division is created to achieve the purpose of the robot actively assisting in the next assembly after recognizing the operator's intention, thereby improving the flexibility of collaboration and production efficiency.
[0052] The method for human-machine collaborative system motion recognition based on neural network proposed in the present invention can significantly improve the accuracy of motion recognition and the efficiency of human-machine collaborative assembly, and can be extended to more human-machine collaborative operation fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0054] Figure 1 This is an overall flow chart of the human-machine collaborative system action recognition method based on neural network of the present invention.
[0055] Figure 2 It is a schematic diagram of the implementation of the method for action recognition of a human-machine collaborative system based on a neural network according to the present invention.
[0056] Figure 3 This is a schematic diagram of the arrangement of multi-view action acquisition cameras in the human-machine collaborative system action recognition method based on neural network of the present invention.
[0057] Figure 4 The structural diagram of the HRI-GCN model in the human-machine collaborative system action recognition method based on neural network of the present invention.
[0058] Figure 5 Structural diagram of the network model AGCN and LSTM modules in the neural network-based human-machine collaborative system action recognition method of the present invention.
[0059] Figure 6 A schematic diagram of the execution flow of the method for action recognition of a human-machine collaborative system based on a neural network of the present invention. DETAILED DESCRIPTION
[0060] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0061] Embodiment 1:
[0062] This embodiment provides a human-machine collaboration method based on a neural network. Figure 2 ,include:
[0063] Step 1: Obtain video data of human operation in human-machine collaborative operation tasks from multiple perspectives. For two sets of video data from different perspectives, use the 2D skeleton key point recognition model openpose to complete skeleton key point recognition.
[0064] Step 2: The two sets of skeleton space coordinates obtained under different viewpoints are fused and reconstructed to obtain a complete and accurate sequence of human skeleton key point space coordinates;
[0065] Step 3: Input the spatial coordinate sequence of the key points of the human skeleton into the trained custom HRI-GCN action recognition network for action prediction and recognition;
[0066] Step 4: Develop a workflow and a human-machine task correspondence table based on collaborative tasks;
[0067] Step 5: According to the identified action category, the set collaborative task process and the human-machine task correspondence table, specify the robot to transfer the workpiece or tool required for the next step;
[0068] The HRI-GCN action recognition network is built based on graph convolutional network GCN, long short-term memory network LSTM and attention mechanism.
[0069] Embodiment 2:
[0070] This embodiment provides a human-machine collaboration method based on a neural network. Figure 1 and Figure 2 The method of this embodiment includes a neural network training process and a human-machine collaboration process using the trained neural network. The specific implementation steps are as follows:
[0071] Step 1: Use multiple RGB cameras to capture video data of human operations in human-machine collaborative operation tasks from different perspectives, standardize the number of captured video frames, and save them with action labels;
[0072] Step 2: For the human action videos from different perspectives collected in step 1, use the 2D skeleton key point recognition model openpose to complete the skeleton key point recognition respectively. Finally, use the key point correspondence to reconstruct the spatial three-dimensional information of the human skeleton and store it as an array to complete the preliminary establishment of the data set.
[0073] Step 3: Construct the spatial coordinate data of the skeleton key points corresponding to the different actions obtained in step 2 into a human action skeleton dataset, and divide it into a training set and a test set in a ratio of 8:2;
[0074] Step 4: Take pictures and identify the coordinates of each skeleton key point when there is no action as a reference, subtract the spatial coordinates in the data set from the reference coordinates when there is no action, obtain the feature vector array, and select a reasonable threshold to use the threshold to eliminate interference data;
[0075] Step 5: Use graph convolutional networks (GCN), long short term memory (LSTM) and attention mechanism to build a custom HRI-GCN action recognition network model for predicting and recognizing human actions.
[0076] The HRI-GCN network model in this embodiment has a main structure as shown in Table 1:
[0077] Table 1 Main structure of HRI-GCN network model
[0078]
[0079] As shown in Table 1, the main features of the HRI-GCN action recognition network model designed in this embodiment are as follows: First, in order to effectively extract the spatiotemporal features of the action, the 10 STunit units are connected in sequence to achieve the deepening of the model to extract deeper spatiotemporal features.
[0080] Secondly, the attention mechanism is added to the graph convolutional network to realize the extraction of spatial features between non-directly adjacent joints, and enhance the model's ability to extract spatial change features of joints corresponding to different actions.
[0081] Finally, since the number of channels is reduced when the model is built, in order to ensure the performance of the model, a batch normalization layer is used in the input part to normalize the data. The feature extraction part uses 10 STunit units in succession, each of which contains a graph convolution layer and a pooling layer. The output stage network maps the abstract feature vector to the predicted category label through a fully connected layer, and the output dimension is the category.
[0082] Step 6: Use the processed training set obtained in step 3 to train the HRI-GCN neural network model obtained in step 5 until the model converges. The training process includes:
[0083] Step 61: Use the adjacency matrix A k Sum degree matrix D k The spatial position information and topological connection relationship of the key points of the skeleton representing the motion are constructed into a training data set by the spatial position information and topological connection relationship of the key points of the skeleton representing the motion;
[0084] Step 62: Based on the HRI-GCN action recognition network, establish an AGCN module for extracting spatial variation features of skeletal key points and an LSTM module for predicting and extracting temporal features. The input of the AGCN module is a 4-dimensional structure matrix [N*M,C,T,V], and the input of the LSTM module is also a 4-dimensional structure matrix [N*M,C,T,V], but only the temporal information of T is extracted; the outputs of the AGCN and LSTM modules maintain the original structure, which is a 4-dimensional structure matrix [N*M,C,T,V]; finally, connect the fully connected layer FC1 and the softmax classifier to output the action label.
[0085] Among them, in the 4-dimensional structure matrix [N*M,C,T,V], N is the batch size, M is the number of people, C is the number of input channels, T is the number of time steps, and V is the number of skeleton key points;
[0086] Step 63: Divide the training set and test set for the HRI-GCN action recognition network: divide the data set into a training set and a test set in a ratio of 8:2;
[0087] Step 64: define the performance representation during the training process, namely, the loss function, accuracy function and recall function of the HRI-GCN action recognition network;
[0088] Step 65: Use the training set obtained in step 63 to train the HRI-GCN action recognition network defined in step 62 until the model converges, and then test the model performance on the test set; during the training process, use the loss function and accuracy function defined in step 64 to evaluate the performance of the model.
[0089] Furthermore, the HRI-GCN model uses the cross entropy loss function as the loss function, including:
[0090]
[0091] Among them, N represents the total number of samples used in the training process, M represents the type of action that needs to be predicted and recognized, and y ik is the symbolic function representing the target action, p ik Represents the probability that sample i output by the final fully connected layer belongs to action category k.
[0092] Furthermore, the accuracy function of the HRI-GCN model is:
[0093]
[0094]
[0095] Among them, T P Represents the number of correct recognitions of a certain type of action, F P represents the number of errors in the recognition of a certain type of action, M represents the type of action that needs to be predicted and recognized, and P k Represents the accuracy of a certain type of action recognition, W k Represents the weight of this type of action data in the overall data.
[0096] Furthermore, the AGCN module and the LSTM module use the ReLU function as the activation function.
[0097] Step 7: Use the camera to collect operator movements in real time, convert them into skeleton data, and input them into the trained HRI-GCN network for movement prediction and recognition;
[0098] Step 8: Based on the collaborative assembly tasks, formulate the assembly workflow and human-machine task allocation table:
[0099] Furthermore, the human-machine task correspondence table in step 8 includes:
[0100] Step 81: For the human-machine collaborative task, formulate a workflow to divide the human work tasks and the robot collaborative tasks;
[0101] Step 82: Match the actions in the human work task with the actions that the robot needs to collaborate with, such as transferring workpieces or tools, and match the identification tags corresponding to the actions with the robot collaborative tasks to form a human-machine task correspondence table.
[0102] Step 9: Predict the next intention based on the identified action category, and then specify the robot to deliver the workpiece or tool required for the next step based on the set human-machine task allocation table.
[0103] This embodiment adopts a dual-view RBG video data acquisition method, and cooperates with the skeleton key point recognition algorithm to fuse and reconstruct the skeleton key point information, so as to solve the problem that the single-channel camera collects the action signal with incomplete viewing angle and is easily blocked, and reduce the interference of environmental information. Video data is collected by two RGB cameras with different viewing angles, and the skeleton key point recognition algorithm is used to identify the skeleton information of the video data from two angles and fuse and reconstruct to obtain the complete 3D spatial coordinates of the human skeleton. It can not only effectively avoid the occlusion caused by the incomplete viewing angle of the single-channel camera and the interference of ambient light in the industrial environment, but also generate the skeleton key point spatial coordinate sequence to reduce the redundant information of the input model.
[0104] This embodiment solves the problem that the robot is limited to completing the current task under the operation of the worker and cannot predict or actively assist in the next task through a customized HRI-GCN action recognition network. The HRI-GCN action recognition network uses the position changes of the key points of the skeleton during continuous actions as input to achieve dynamic recognition of operation actions, and uses the timing characteristics of the recurrent neural network to complete the prediction of the early actions of the worker during the operation, identify the worker's next action intention, and further improve the accuracy of action recognition and the efficiency of collaboration.
[0105] This embodiment solves the problem that in collaborative tasks, the tasks of robots and humans are relatively independent, making it difficult to achieve the human-machine collaborative task allocation required for efficient collaboration by creating a comparison table of human actions and robot tasks. By decomposing the assembly process into sub-assembly tasks and single-step operation tasks, and creating a comparison table of worker operation tasks and robot tasks, the robot can actively assist in the next assembly step after recognizing the operator's intention, thereby improving the flexibility of collaboration and the efficiency of production.
[0106] The neural network-based human-machine collaborative system action recognition method proposed in this embodiment can significantly improve the accuracy of action recognition and the efficiency of human-machine collaborative assembly, and can be extended to more human-machine collaborative operation fields.
[0107] Embodiment three:
[0108] This embodiment provides a human-machine collaboration system based on a neural network. Figure 2 As shown in the figure, it includes: a multi-view action acquisition module, which uses a dual-view RGB camera to collect and fuse the skeleton information of the worker's operation action video data; a motion intention recognition module, which is composed of a data processing part and an HRI-GCN motion recognition network; the intention prediction and task allocation module is composed of three parts: action prediction, task allocation and action planning; through the information processing of the three modules of multi-view action acquisition module, action intention recognition module, intention prediction and task allocation module, the control instructions of the robot are obtained, and finally input into the human-machine collaboration module for execution, so as to complete efficient human-machine collaboration.
[0109] The multi-view action acquisition module is specifically used to: obtain the worker's operation video from multiple perspectives to avoid action occlusion. It further includes: collecting the worker's motion data from the front and side respectively during operation; using the spatial position relative relationship of the main view acquisition camera and the auxiliary view acquisition camera reference coordinate system obtained during the camera calibration process to complete the coordinate system reconstruction.
[0110] The data processing part in the action intention recognition module is specifically used for: obtaining the spatial coordinates of the key points of the skeleton during the worker's movement. It further includes: inputting the obtained dual-channel action signals into the 2D human skeleton key point recognition algorithm, respectively obtaining the spatial positions of the key points of the worker's skeleton in each camera reference coordinate system under two different perspectives, and converting the two sets of coordinates to the same world coordinate system through the key point correspondence and the relative position relationship of the two reference coordinates in space, completing the points lost or misjudged due to occlusion, and finally performing mean filtering to obtain relatively pure spatial coordinates of the skeleton key points.
[0111] The HRI-GCN action recognition network in the action intention recognition module is specifically used for: identifying the worker's actions and predicting the next action intention. It further includes: inputting the acquired skeleton key point spatial coordinate sequence into the HRI-GCN network, extracting the spatiotemporal features of the input sequence through the STunit unit, and outputting the final predicted recognition feature vector value, wherein the feature vector value described includes the probability of the input action belonging to each type of action, and the intention of the current action and the next action can be obtained.
[0112] The intention prediction and task allocation module is specifically used to: predict the worker's next action intention based on the early characteristics of the worker's action process, and complete the robot task allocation according to the human-machine task allocation table. It further includes: according to the work characteristics of each step in the assembly step, it is decomposed into the worker's operation tasks and the corresponding robot-assisted tasks to form a human-machine task allocation table. After obtaining the current action of the human operator and the next work intention, according to the human-machine task allocation table, select the robot task that best suits the current working conditions, and plan the robot's motion trajectory according to the task content.
[0113] The human-machine collaboration module is specifically used to: generate robot motion instructions according to the planned robot motion trajectory, and the robot transfers tools or workpieces to the operator to complete human-machine collaboration.
[0114] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.
[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A human-machine collaboration method based on neural network, It is characterized in that The method comprises: Step 1: Obtain video data of human operation in human-machine collaborative operation tasks from multiple perspectives, and use the 2D skeleton key point recognition model openpose to complete skeleton key point recognition for multiple sets of video data from different perspectives; Step 2: The skeleton space coordinates of multiple sets of different viewpoints obtained by recognition are fused and reconstructed to obtain a complete and accurate sequence of human skeleton key point space coordinates; Step 3: Input the spatial coordinate sequence of the key points of the human skeleton into the trained custom HRI-GCN action recognition network for action prediction and recognition; Step 4: Develop a workflow and a human-machine task correspondence table based on collaborative tasks; Step 5: According to the identified action category, the set collaborative task process and the human-machine task correspondence table, specify the robot to transfer the workpiece or tool required for the next step; The HRI-GCN action recognition network is constructed based on graph convolutional network GCN, long short-term memory network LSTM and attention mechanism; The construction process of the HRI-GCN action recognition network includes: Step 21: Use the multidimensional array of spatial coordinates of the skeleton key points as the original input of the graph convolutional layer GCN network, use the skeleton key points as the nodes of the graph, and convert them into the adjacency matrix A and the degree matrix D as the input of the graph convolution for extracting spatial features; Step 22: Add the attention mechanism to the graph convolutional layer GCN network to form the AGCN module; Step 23: Using the spatial feature matrix sequence obtained by extracting single-frame spatial features through the AGCN module as the input of the LSTM module to extract the temporal features of the action; Step 24: alternately stack the spatial graph convolution AGCN module and the temporal gated unit LSTM module to form a spatiotemporal convolution unit STunit. The HRI-GCN action recognition network includes 10 spatiotemporal convolution units STunit1-Stunit10 and a final fully connected layer FC1, and finally connects the softmax classifier to form an action recognition network. The input of the AGCN module is a 4-dimensional structure matrix [N*M, C, T, V], and the input of the LSTM module is also a 4-dimensional structure matrix [N*M, C, T, V], but only the time series information of T is extracted; the outputs of the AGCN module and the LSTM module both maintain the original structure, which is a 4-dimensional structure matrix [N*M, C, T, V]; finally, the fully connected layer FC1 and the softmax classifier are connected to output the action label; Among them, in the 4-dimensional structure matrix [N*M,C,T,V], N is the batch size, M is the number of people, C is the number of input channels, T is the number of time steps, and V is the number of skeleton key points.
2. The human-machine collaboration method based on neural network according to claim 1, It is characterized in that The input of the HRI-GCN action recognition network is an action matrix sequence composed of human skeleton points, and a 25×3 matrix is used to represent the spatial coordinates of 25 key points of the human skeleton; the output of the HRI-GCN action recognition network uses an N×1 matrix to represent the probability that the input belongs to a certain action.
3. The human-machine collaboration method based on neural network according to claim 1, It is characterized in that The spatiotemporal convolution unit STunit in the HRI-GCN action recognition network adopts the Relu function as the activation function.
4. The human-machine collaboration method based on neural network according to claim 1, It is characterized in that The training process of the HRI-GCN action recognition network includes: Step 1: Use an RGB camera to capture video data of human operations in human-machine collaborative operation tasks, standardize the number of captured video frames, and save them with action labels; Step 2: For the human action video collected in step 1, use the 2D skeleton key point recognition model openpose to complete the skeleton key point recognition, generate the spatial coordinate data of each key point, store it as an array, and complete the preliminary establishment of the data set; Step 3: constructing the spatial coordinate data of the skeleton key points corresponding to the different actions obtained in step 2 into a human action skeleton dataset; Step 4: Take pictures and identify the coordinates of each skeleton key point when there is no action as a reference, subtract the spatial coordinates in the data set from the reference coordinates when there is no action, obtain the feature vector array, and select a reasonable threshold to use the threshold to eliminate interference data; Step 5: Use the adjacency matrix A k Sum degree matrix D k The spatial position of each key point representing the skeleton key point and the topological relationship between nodes are constructed into a data set by the spatial position information and the topological connection relationship of the skeleton key points representing the movement; Step 6: The data set is divided into a training set and a test set in a ratio of 8:2; Step 7: Using the training set obtained in step 6, train the HRI-GCN action recognition network until the model converges, and then test the model performance on the test set.
5. The human-machine collaboration method based on neural network according to claim 4, It is characterized in that The loss function of the HRI-GCN action recognition network is: Among them, N represents the total number of samples used in the training process, M represents the type of action that needs to be predicted and recognized, and y ik is the symbolic function representing the target action, p ik Represents the probability that sample i output by the final fully connected layer belongs to action category k.
6. The human-machine collaboration method based on neural network according to claim 4, It is characterized in that The accuracy function of the HRI-GCN action recognition network is: Among them, T P Represents the number of correct recognitions of a certain type of action, F P represents the number of errors in the recognition of a certain type of action, M represents the type of action that needs to be predicted and recognized, and P k Represents the accuracy of a certain type of action recognition, W k Represents the weight of this type of action data in the overall data.
7. The human-machine collaboration method based on neural network according to claim 1, It is characterized in that The process of formulating the workflow and the human-machine task correspondence table according to the collaborative task in step 4 includes: Step 41: For the human-machine collaborative task, formulate a workflow to divide the human work tasks and the robot collaborative tasks; Step 42: Match the actions in the human work task with the actions that the robot needs to collaborate with, such as transferring workpieces or tools, and match the identification tags corresponding to the actions with the robot collaborative tasks to form a human-machine task allocation table.
8. A human-machine collaborative system based on neural network, It is characterized in that The method for implementing the human-machine collaboration method based on a neural network according to any one of claims 1 to 7 comprises: Multi-view action acquisition module, used to capture operator action videos from multiple angles; The action intention recognition module obtains the spatial coordinate sequence of the skeleton key points based on the action video of the operator, inputs the sequence into the HRI-GCN action recognition network, and outputs the final predicted recognition feature vector value, wherein the feature vector value includes the probability that the input action belongs to each type of action; The intention prediction and task assignment module is used to specify the workpiece or tool required for the robot to transfer the next step based on the intention recognition results and the human-machine task correspondence table; The human-machine collaboration module transfers the tool or workpiece to the operator according to the instructions issued by the intention prediction and task allocation module.
9. The human-machine collaboration system based on neural network according to claim 8, It is characterized in that The multi-view motion acquisition module collects motion data of workers during operation from the front and side respectively.
Citation Information
Patent Citations
Robot perception and understanding methods based on human-robot collaboration
CN107150347B
Action recognition method based on human body segmentation model applied to human-computer collaboration
CN109086706B
Bone action recognition method based on learnable PL-GCN and ECLSTM
CN114529984A
Attention mechanism-based motion recognition method of graph convolutional neural network
CN114821804A
Skeleton behavior recognition method based on spatial-temporal feature enhanced graph convolutional network
CN114882421A
