A dynamic human-machine collaborative assembly method and system based on limb motion prediction

Abstract

The present invention proposes a dynamic human-machine collaborative assembly method and system based on limb motion prediction, comprising an inertial measurement unit, a matching controller, an industrial robot and a computer. The matching controller controls the inertial measurement unit to collect the operator's motion information, the computer extracts the signal characteristics of the inertial measurement unit, and uses a support vector machine algorithm optimized by a particle swarm to perform limb motion recognition. A parameter-variable hidden Markov model is proposed to predict the action sequence in real time, which can infer the operator's future intentions and realize the dynamic variability of the assembly sequence. The proposed method can effectively predict the operator's next action and realize dynamic human-machine collaborative assembly with variable assembly sequence by cooperating with humans to complete the key steps of the operation through the industrial robot. The industrial robot of the present invention makes corresponding assembly actions according to the different choices of the operator, realizes a dynamically variable human-machine collaborative assembly solution, realizes flexible production and improves production efficiency.

Description

Technical Field

[0001] The present invention relates to the field of limb motion recognition and prediction, and in particular to a dynamic human-machine collaborative assembly method and system based on limb motion prediction. Background Art

[0002] Human-robot collaborative assembly is a key development in the current robotics field. Although robots are widely used in workshop production, they lack cognitive capabilities. In some complex assembly environments or flexible production workshops, humans and robots still need to work together to complete key steps. Human actions are complex, diverse, and random, and humans can subjectively choose different feasible operation methods and routes. During the collaborative process, robots should recognize and predict human intentions in real time and provide corresponding assistance. This is of great significance for improving production efficiency and achieving flexible production. Current human-robot collaborative processes generally require a pre-specified operation sequence. Robots cannot yet respond differently to different human operation sequences, effectively addressing the dynamic variability of assembly sequences.

[0003] Currently, there are many methods for dynamic human-machine collaborative assembly, such as the patent number CN114445741A, which discloses a method for identifying human work intentions for human-machine collaborative assembly. The method comprises the following steps: collecting RGB-D images of the assembly environment scene; constructing and training an assembly scene perception network to detect various elements in the assembly scene, including humans, machines, and various workpieces; inputting the image into the assembly scene perception network for detection, and extracting the visual and spatial features of each element; constructing a graph attention network, using visual features to initialize the features of each node, using spatial features to initialize the features of the edges between nodes, dynamically aggregating the features of each node, and inferring the human work intention. Although this invention can effectively utilize contextual information in the assembly environment, the robot cannot yet respond differently to different human operation sequences, that is, it cannot effectively solve the problem of dynamic variability in the assembly sequence. Summary of the Invention

[0004] The present invention proposes a dynamic human-machine collaborative assembly method and system based on limb movement prediction, which can solve the problem of a robot making corresponding assembly movements according to different choices of the operator and realize dynamic human-machine collaborative assembly with variable assembly sequence.

[0005] To this end, the present invention proposes a dynamic human-machine collaborative assembly method based on limb motion prediction, comprising an inertial measurement unit, a supporting controller, an industrial robot, and a computer. The supporting controller controls the inertial measurement unit to collect the operator's motion information, and the computer extracts the signal features of the inertial measurement unit and uses a support vector machine algorithm optimized by a particle swarm to perform limb motion recognition. A parameter-variable hidden Markov model is proposed to predict the action sequence in real time, which can infer the operator's future intentions and achieve dynamic variability of the assembly sequence. The proposed method realizes dynamic human-machine collaborative assembly with variable assembly sequence by effectively predicting the operator's next action through the industrial robot cooperating with humans to complete the key steps.

[0006] The robot of the present invention performs corresponding assembly actions according to the operator's different choices, realizing a dynamic and variable human-machine collaborative assembly solution, achieving flexible production and improving production efficiency. The technical solution adopted by the present invention to solve its technical problems is:

[0007] A dynamic human-machine collaborative assembly system based on limb motion prediction includes an inertial measurement unit, a matching controller, an industrial robot and a computer.

[0008] The inertial measurement unit is worn on the wrist;

[0009] The supporting controller controls the inertial measurement unit to collect the operator's motion information;

[0010] The computer extracts the signal features of the inertial measurement unit and uses a particle swarm optimization support vector machine algorithm of time domain and time-frequency domain features for limb movement recognition, and uses the limb recognition action sequence as the input of a hidden Markov model with variable parameters.

[0011] A dynamic human-machine collaborative assembly method based on body motion prediction, employing the above-described dynamic human-machine collaborative assembly system based on body motion prediction, uses a parameter-variable hidden Markov model to predict action sequences in real time, infers the operator's future intentions under supervised training and zero-training conditions, and achieves dynamic variability in assembly sequences.

[0012] The industrial robot cooperates with humans to complete key steps, performs corresponding assembly actions according to the operator's different choices, predicts the operator's next action and realizes dynamic human-machine collaborative assembly with variable assembly sequence;

[0013] The assembly method utilizes a particle swarm optimization support vector machine algorithm using both time domain and time-frequency domain features during limb movement recognition, and includes the following steps:

[0014] Step A1: the supporting controller controls the inertial measurement unit to collect the operator's motion information, performs zero-point correction on the raw data, and eliminates gravity acceleration and zero-point drift in the signal;

[0015] Step A2: The computer extracts the signal characteristics of the inertial measurement unit and extracts the mean value Standard deviation σ x , maximum value x p , peak factor C, waveform factor W time domain characteristics, decompose the db1 wavelet packet into three layers of coefficients, and extract the low-frequency energy value E of the three-layer coefficient decomposition j,k As the time-frequency domain features of the signal, the six feature formulas are shown in the following formulas (1)-(6):

[0016]

[0017]

[0018] x p =max{|x1|,|x2|,…,|x N |} (Formula 3)

[0019]

[0020]

[0021]

[0022] Where x i is the time domain sequence of the signal, where i = 1, 2, ..., N, and N is the number of sample points; The decomposition signal at node (j, k) after the wavelet packet is applied to the representative signal, and l is the node signal The discrete point number, m represents the total number of discrete points of the node signal;

[0023] Step A3: Based on MATLAB software, using the LIBSVM software package, selecting the RBF kernel function, and using the particle swarm optimization algorithm to optimize the parameters of the constructed support vector machine classifier to find the optimal parameters c and g, that is, the global optimal solution;

[0024] Step A4: After preprocessing and feature extraction of the collected data, 80% of the data is used as a training set and the remaining 20% ​​of the data is used as a test set. The particle swarm optimized support vector machine algorithm is used for body movement recognition.

[0025] The specific steps of the assembly method using a variable parameter hidden Markov model to predict action sequences in real time include:

[0026] Step B1: In the Hidden Markov Model (HMM) modeling phase, the observation layer is considered as T behaviors in a random order. Specifically, the hidden layer represents the assembly task; the state set Q represents the set of different task states of the assembly step, i.e., the hidden state set; the observation set V represents the set of body movement recognition, i.e., the explicit state set; the state sequence I represents a sequence of different assembly tasks of length T, and the observation sequence O is the corresponding body movement recognition sequence, as shown in Formula 7:

[0027]

[0028] Where N is the number of possible states and M is the number of possible observations;

[0029] Step B2: The actions identified by the support vector machine algorithm optimized by particle swarm optimization are used as observation sequences, and the observation sequences o1, o2, ..., o T As input to the Hidden Markov Model;

[0030] Step B3: In the hidden Markov model prediction stage, the product assembly task is modeled as a series of body movements, and the Baum-Welch algorithm is improved. The model inputs the single observation sequence sample O = {o1, o2, ..., o T}, in order to prevent the zero probability situation, after each re-estimation of the model parameters A, B, π, the λ parameter is improved, as shown in formula 8, that is:

[0031] λ=(A,B,π)+(A',B',π') (Formula 8)

[0032] A', B', π' and A, B, π have the same rows and columns and all elements are 1;

[0033] Step B4: Iterate the Baum-Welch algorithm to obtain the re-estimated model parameters The parameters As the parameters of the hidden Markov model of the next human-machine collaborative assembly, the parameters of the state transfer matrix and the parameters of the initial state matrix can be changed in real time, thereby realizing dynamic human-machine collaborative assembly with variable assembly order;

[0034] Step B5: The steps of Baum-Welch algorithm are as follows: Initialize the model, for n=0, select b j (k) (0) , Q={q1,q2,…,q N}, the model formula is shown in Formula 9 and Formula 10:

[0035] λ (0) =(A (0) ,B (0) ,π (0)) (Formula 9)

[0036] Then recursively set the number of iterations n = 1, 2...,

[0037]

[0038] Get the termination model parameter: λ (n+1) =(A (n+1) ,B (n+1) ,π (n+1) ),Right now

[0039] Step B6: Expand the forward algorithm in the hidden Markov model to predict the person's next intention; the model input is the real-time body movement recognition sequence O and parameters Initialize the model's first action forward variable α1(i), and then iteratively calculate α t+1 (j), and then calculate P(O|λ). The algorithm steps are shown in Formula 11:

[0040]

[0041] Where 1≤t≤T-1, 1≤j≤N, a ij is an element in the state transfer matrix, b j (o t ) is an element in the emission matrix;

[0042] Step B7: Extend the forward algorithm of the model to achieve the prediction effect and calculate the joint probability S of each hidden state in time step t+1 t+2 , where the hidden state with the highest probability represents the operator's intention, thereby predicting the operator's future intention, as shown in Formula 12:

[0043]

[0044] Step B8: The input of the Viterbi algorithm is the real-time body movement recognition sequence O and parameters The assembly sequence order is inferred by the Viterbi algorithm;

[0045] Step B9: The industrial robot cooperates with humans to complete the key steps. The robot makes corresponding assembly actions according to the operator's different choices and realizes dynamic human-machine collaborative assembly with variable assembly sequence by effectively predicting the operator's next action.

[0046] Compared with the prior art, the industrial robot of the present invention performs corresponding assembly actions according to different choices of the operator, realizes a dynamically variable human-machine collaborative assembly solution, achieves flexible production and improves production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0048] Attachment Figure 1 This is a schematic diagram of a dynamic human-machine collaborative assembly method and system based on limb motion prediction according to the present invention;

[0049] Attachment Figure 2 It is a flow chart of the particle swarm optimization support vector machine algorithm of the present invention;

[0050] Attachment Figure 3 is a flow chart of the improved hidden Markov model of the present invention;

[0051] Attachment Figure 4 This is a flow chart of the body movement sequence prediction of the present invention;

[0052] In the figure, 1- inertial measurement unit; 2- matching controller; 3- industrial robot; 4- computer. DETAILED DESCRIPTION

[0053] To make the features and advantages of this patent more clearly understood, the following embodiments are specifically described in detail as follows:

[0054] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art to which this application belongs.

[0055] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0056] The following is a further detailed description of this embodiment with reference to the accompanying drawings:

[0057] like Figure 1 As shown, a dynamic human-machine collaborative assembly system based on limb motion prediction includes an inertial measurement unit 1, a matching controller 2, an industrial robot 3 and a computer 4.

[0058] The inertial measurement unit is worn on the wrist;

[0059] The supporting controller controls the inertial measurement unit to collect the operator's motion information;

[0060] The computer extracts the signal features of the inertial measurement unit and uses a particle swarm optimization support vector machine algorithm of time domain and time-frequency domain features for limb movement recognition, and uses the limb recognition action sequence as the input of a hidden Markov model with variable parameters.

[0061] A dynamic human-machine collaborative assembly method based on body motion prediction, employing the above-described dynamic human-machine collaborative assembly system based on body motion prediction, uses a parameter-variable hidden Markov model to predict action sequences in real time, infers the operator's future intentions under supervised training and zero-training conditions, and achieves dynamic variability in assembly sequences.

[0062] The industrial robot cooperates with humans to complete key steps, performs corresponding assembly actions according to the operator's different choices, predicts the operator's next action and realizes dynamic human-machine collaborative assembly with variable assembly sequence;

[0063] like Figure 2 The embodiment 1 for limb movement recognition is shown. The assembly method utilizes a particle swarm optimization support vector machine algorithm using both time domain and time-frequency domain features during limb movement recognition, and includes the following steps:

[0064] Step A1: the supporting controller controls the inertial measurement unit to collect the operator's motion information, performs zero-point correction on the raw data, and eliminates gravity acceleration and zero-point drift in the signal;

[0065] Step A2: The computer extracts the signal characteristics of the inertial measurement unit and extracts the mean value Standard deviation σ x , maximum value x p , peak factor C, waveform factor W time domain characteristics, decompose the db1 wavelet packet into three layers of coefficients, and extract the low-frequency energy value E of the three-layer coefficient decomposition j,k As the time-frequency domain features of the signal, the six feature formulas are shown in the following formulas (1)-(6):

[0066]

[0067]

[0068] x p =max{|x1|,|x2|,…,|x N |} (Formula 3)

[0069]

[0070]

[0071]

[0072] Where x i is the time domain sequence of the signal, where i = 1, 2, ..., N, and N is the number of sample points; The decomposition signal at node (j, k) after the wavelet packet is applied to the representative signal, and l is the node signal The discrete point number, m represents the total number of discrete points of the node signal;

[0073] Step A3: Based on MATLAB software, using the LIBSVM software package, selecting the RBF kernel function, and using the particle swarm optimization algorithm to optimize the parameters of the constructed support vector machine classifier to find the optimal parameters c and g, that is, the global optimal solution;

[0074] Step A4: After preprocessing and feature extraction of the collected data, 80% of the data is used as a training set and the remaining 20% ​​of the data is used as a test set. The particle swarm optimized support vector machine algorithm is used for body movement recognition.

[0075] like Figure 3 and Figure 4 The second embodiment for body movement prediction is shown in FIG. Figure 3 The flowchart of the improved hidden Markov model algorithm is shown in Figure 4 Shown is the flow chart of limb motion sequence prediction.

[0076] The specific steps of the assembly method using a variable parameter hidden Markov model to predict action sequences in real time include:

[0077] Step B1: In the Hidden Markov Model (HMM) modeling phase, the observation layer is considered as T behaviors in a random order. Specifically, the hidden layer represents the assembly task; the state set Q represents the set of different task states of the assembly step, i.e., the hidden state set; the observation set V represents the set of body movement recognition, i.e., the explicit state set; the state sequence I represents a sequence of different assembly tasks of length T, and the observation sequence O is the corresponding body movement recognition sequence, as shown in Formula 7:

[0078]

[0079] Where N is the number of possible states and M is the number of possible observations;

[0080] Step B2: The actions identified by the support vector machine algorithm optimized by particle swarm optimization are used as observation sequences, and the observation sequences o1, o2, ..., o T As input to the Hidden Markov Model;

[0081] Step B3: In the hidden Markov model prediction stage, the product assembly task is modeled as a series of body movements, and the Baum-Welch algorithm is improved. The model inputs the single observation sequence sample O = {o1, o2, ..., o T}, in order to prevent the zero probability situation, after each re-estimation of the model parameters A, B, π, the λ parameter is improved, as shown in formula 8, that is:

[0082] λ=(A,B,π)+(A',B',π') (Formula 8)

[0083] A', B', π' and A, B, π have the same rows and columns and all elements are 1;

[0084] Step B4: Iterate the Baum-Welch algorithm to obtain the re-estimated model parameters The parameters As the parameters of the hidden Markov model of the next human-machine collaborative assembly, the parameters of the state transfer matrix and the parameters of the initial state matrix can be changed in real time, thereby realizing dynamic human-machine collaborative assembly with variable assembly order;

[0085] Step B5: The steps of Baum-Welch algorithm are as follows: Initialize the model, for n=0, select b j (k) (0) , Q={q1,q2,…,q N}, the model formula is shown in Formula 9 and Formula 10:

[0086] λ (0) =(A (0) ,B (0) ,π (0) ) (Formula 9)

[0087] Then recursively set the number of iterations n = 1, 2...,

[0088]

[0089] Get the termination model parameter: λ (n+1) =(A (n+1) ,B (n+1) ,π (n+1) ),Right now

[0090] Step B6: Expand the forward algorithm in the hidden Markov model to predict the person's next intention; the model input is the real-time body movement recognition sequence O and parameters Initialize the model's first action forward variable α1(i), and then iteratively calculate α t+1(j), and then calculate P(O|λ). The algorithm steps are shown in Formula 11:

[0091]

[0092] Where 1≤t≤T-1, 1≤j≤N, a ij is an element in the state transfer matrix, b j (o t ) is an element in the emission matrix;

[0093] Step B7: Extend the forward algorithm of the model to achieve the prediction effect and calculate the joint probability S of each hidden state in time step t+1 t+2 , where the hidden state with the highest probability represents the operator's intention, thereby predicting the operator's future intention, as shown in Formula 12:

[0094]

[0095] Step B8: The input of the Viterbi algorithm is the real-time body movement recognition sequence O and parameters The assembly sequence order is inferred by the Viterbi algorithm;

[0096] Step B9: The industrial robot cooperates with humans to complete the key steps. The robot makes corresponding assembly actions according to the operator's different choices and realizes dynamic human-machine collaborative assembly with variable assembly sequence by effectively predicting the operator's next action.

[0097] This patent is not limited to the above-mentioned optimal implementation method. Anyone can derive various other forms of dynamic human-machine collaborative assembly methods and systems based on limb motion prediction under the inspiration of this patent. All equal changes and modifications made within the scope of the patent application of this invention should be covered by this patent.

Claims

1. A dynamic human-machine collaborative assembly method based on limb motion prediction, using a dynamic human-machine collaborative assembly system based on limb motion prediction, characterized by: The assembly method uses a parameter-variable hidden Markov model to predict action sequences in real time, infers the operator's future intentions under supervised training and zero training conditions, and realizes dynamic variability of assembly sequences. Industrial robots work with humans to complete key steps, make corresponding assembly movements based on the operator's choices, predict the operator's next move, and achieve dynamic human-machine collaborative assembly with variable assembly order. A dynamic human-robot collaborative assembly system based on limb motion prediction, including an inertial measurement unit, a supporting controller, an industrial robot, and a computer; The inertial measurement unit is worn on the wrist; The supporting controller controls the inertial measurement unit to collect the operator's motion information; The computer extracts the signal features of the inertial measurement unit and uses a particle swarm optimization support vector machine algorithm based on time domain and time-frequency domain features for limb movement recognition, using the limb recognition action sequence as input to a hidden Markov model with variable parameters; The specific steps of the assembly method using a variable parameter hidden Markov model to predict action sequences in real time include: Step B1: In the Hidden Markov Model (HMM) modeling phase, the observation layer is considered as T behaviors in a random order. Specifically, the hidden layer represents the assembly task; the state set Q represents the set of different task states of the assembly step, i.e., the hidden state set; the observation set V represents the set of body movement recognition, i.e., the explicit state set; the state sequence I represents a sequence of different assembly tasks of length T, and the observation sequence O is the corresponding body movement recognition sequence, as shown in Formula 7: Where N is the number of possible states and M is the number of possible observations; Step B2: The actions identified by the support vector machine algorithm optimized by particle swarm optimization are used as observation sequences, and the observation sequences o1, o2, ..., o T As input to the Hidden Markov Model; Step B3: In the hidden Markov model prediction stage, the product assembly task is modeled as a series of body movements, and the Baum-Welch algorithm is improved. The model inputs the single observation sequence sample O = {o1, o2, ..., o T }, in order to prevent the zero probability situation, after each re-estimation of the model parameters A, B, π, the λ parameter is improved, as shown in formula 8, that is: λ=(A,B,π)+(A',B',π') (Formula 8) A', B', π' and A, B, π have the same rows and columns and all elements are 1; Step B4: Iterate the Baum-Welch algorithm to obtain the re-estimated model parameters The parameters As the parameters of the hidden Markov model of the next human-machine collaborative assembly, the parameters of the state transfer matrix and the parameters of the initial state matrix can be changed in real time, thereby realizing dynamic human-machine collaborative assembly with variable assembly order; Step B5: The steps of Baum-Welch algorithm are as follows: Initialize the model, for n=0, select Q={q1,q2,…,q N }, the model formulas are shown in Formula 9 and Formula 10: λ (0) (A (0) ,B (0) ,π (0) ) (Picture 9) Then recursively set the number of iterations n = 1, 2..., Get the termination model parameter: λ (n+1) =(A (n+1) ,B (n+1) ,π (n+1) ),Right now Step B6: Expand the forward algorithm in the hidden Markov model to predict the person's next intention; the model input is the real-time body movement recognition sequence O and parameters Initialize the model's first action forward variable α1(i), and then iteratively calculate α t+1 (j), and then calculate P(O|λ). The algorithm steps are shown in Formula 11: Where 1≤t≤T-1, 1≤j≤N, a ij is an element in the state transfer matrix, b j (o t ) is an element in the emission matrix; Step B7: Extend the forward algorithm of the model to achieve the prediction effect and calculate the joint probability S of each hidden state in time step t+1 t+2 , where the hidden state with the highest probability represents the operator's intention, thereby predicting the operator's future intention, as shown in Formula 12: Step B8: The input of the Viterbi algorithm is the real-time body movement recognition sequence O and parameters The assembly sequence order is inferred by the Viterbi algorithm; Step B9: The industrial robot cooperates with humans to complete the key steps. The robot makes corresponding assembly actions according to the operator's different choices and realizes dynamic human-machine collaborative assembly with variable assembly sequence by effectively predicting the operator's next action.

2. The dynamic human-machine collaborative assembly method based on limb motion prediction according to claim 1, characterized in that: The assembly method utilizes a particle swarm optimization support vector machine algorithm using both time domain and time-frequency domain features during limb movement recognition, and includes the following steps: Step A1: the supporting controller controls the inertial measurement unit to collect the operator's motion information, performs zero-point correction on the raw data, and eliminates gravity acceleration and zero-point drift in the signal; Step A2: The computer extracts the signal characteristics of the inertial measurement unit and extracts the mean value Standard deviation σ x , maximum value x p , peak factor C, waveform factor W time domain characteristics, decompose the db1 wavelet packet into three layers of coefficients, and extract the low-frequency energy value E of the three-layer coefficient decomposition j,k As the time-frequency domain features of the signal, the six feature formulas are shown in the following formulas (1)-(6): x p =max{|x1|,|x2|,…,|x N |} (Formula 3) Where x i is the time domain sequence of the signal, where i = 1, 2, ..., N, and N is the number of sample points; The decomposition signal at node (j, k) after the wavelet packet is applied to the representative signal, and l is the node signal The discrete point number, m represents the total number of discrete points of the node signal; Step A3: Based on MATLAB software, using the LIBSVM software package, selecting the RBF kernel function, and using the particle swarm optimization algorithm to optimize the parameters of the constructed support vector machine classifier to find the optimal parameters c and g, that is, the global optimal solution; Step A4: After preprocessing and feature extraction of the collected data, 80% of the data is used as a training set and the remaining 20% ​​of the data is used as a test set. The support vector machine algorithm with particle swarm optimization is used for body movement recognition.

Images