Multimodal data acquisition method, system and device based on robot assembly learning

Abstract

The present invention relates to a multimodal data acquisition method, system and device based on robot assembly learning, which is applied to robot assembly motion planning, wherein the method comprises: collecting the assembly trajectory and assembly video of the target object and human hands through a multimodal data acquisition device, and associating the assembly trajectory and the assembly video, based on the assembly action unit library, according to the assembly video and the assembly trajectory, the assembly operation of the target object is identified and segmented through a multimodal fused continuous action recognition and segmentation model to obtain a series of continuous single action elements, based on the motion planning strategy library, according to the type of each single action element, the corresponding motion planning strategy is obtained to generate the robot assembly motion planning of the entire assembly operation of the target object. The present invention can improve the success rate and efficiency of robot assembly by fusing the assembly video and the assembly trajectory, performing single action element recognition and semantic segmentation on the continuous assembly action.

Description

Technical Field

[0001] The present invention relates to a multimodal data acquisition method, system and device based on robot assembly learning, and in particular to a method for manual assembly action recognition and segmentation combining multimodal fusion of video and precise posture, which belongs to the technical field of action recognition and segmentation and can be applied to robot assembly motion planning. Background Art

[0002] With the proposal of "Industry 4.0" and "Made in China 2025", China has made intelligent manufacturing an important national research direction. Assembly tasks are a key process in the manufacturing process, while 3C products are still mainly assembled manually. With the shortage of labor and the gradual increase in labor costs, the demand for automation in precision assembly production lines is increasing. The assembly process is complex, the precision requirements are high, and the working space is small, which brings high difficulty to robot programming for robot technicians. In response to such problems, traditional robot programming control methods are too time-consuming and inflexible, which not only limits the speed at which robots adapt to new tasks, but also limits the application of robots in the field of assembly. A technology that can quickly and flexibly transfer the assembly skills of workers on assembly lines to robots is of great significance to expanding the scope of application of robots and improving the automation level of precision assembly production lines.

[0003] However, the current research on demonstration learning in the field of assembly lacks research on complex, high-precision, and dexterous robot assembly operations, and has not solved the problem of low efficiency of robot assembly trajectories. In addition, the manual assembly demonstration data itself is not optimal. The data information obtained by relying on a single motion capture system is insufficient, and it is impossible to clearly know the true meaning of each part of the trajectory to meet the constraints and optimize the trajectory in a reasonable manner. When applied to industrial robots, the assembly efficiency and success rate are average. Summary of the invention

[0004] The present invention provides a multimodal demonstration method and system for robot assembly skill learning, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of the present invention relates on the one hand to a multimodal demonstration method for robot assembly skill learning, comprising the following steps:

[0006] S100, collecting assembly trajectories and assembly videos of a target object and a human hand through a multimodal data collection device, and associating the assembly trajectories and the assembly videos;

[0007] S200, based on the assembly action unit library, according to the assembly video and the assembly trajectory, the assembly operation of the target object is identified and segmented by a multimodal fusion continuous action recognition and segmentation model to obtain a series of continuous single action units;

[0008] S300. Based on a motion planning strategy library, and according to the type of each single action element, obtain the corresponding motion planning strategy to generate a robot assembly motion plan for the entire assembly operation of the target object.

[0009] Further, in step S200:

[0010] The continuous action recognition and segmentation model includes an action recognition model based on a 3D convolutional neural network model and a continuous action segmentation model; wherein the action recognition module is trained by a single action data set, and the continuous action segmentation module is trained by a continuous action data set.

[0011] Further, the step S200 includes:

[0012] S210, based on the assembly action unit library, the trajectory recognition module based on the LSTM network model of the action recognition model performs feature extraction on the assembly trajectory to realize trajectory action recognition, the video action recognition module based on the C3D network model of the action recognition model performs video action recognition on the assembly video, and the trajectory action recognition result and the video action recognition result are merged to obtain a single action recognition result of the target object;

[0013] S220, based on the assembly action unit library, according to the single action recognition result, and according to the assembly trajectory and the assembly video, segmenting the continuous assembly operation of the target object by the continuous action segmentation model to obtain a series of continuous single action units;

[0014] The types of single action elements included in the assembly action unit library include transport action elements, alignment action elements, plug-in action elements, fastening action elements and tearing action elements.

[0015] Furthermore, the network model of the action recognition module includes:

[0016] A first feature extraction submodule for extracting assembly trajectory features; a first LSTM network for processing trajectory features;

[0017] A plurality of single-action video processing submodules for processing assembly videos, wherein the single-action video processing submodule is composed of a first 3D convolution layer and a first 3D pooling layer, and the zero padding in the time dimension of the first 3D convolution layer is replaced by part of the information in the previous time step;

[0018] a first fully connected layer for processing outputs of the plurality of single-action video processing submodules;

[0019] Two second fully connected layers for fusing the output of the first LSTM network with the output of the first fully connected layer;

[0020] A softmax layer is arranged behind the two second fully connected layers.

[0021] Furthermore, the network model of the continuous action segmentation model includes:

[0022] A second feature extraction submodule for extracting assembly trajectory features; a second LSTM network for processing trajectory features;

[0023] A plurality of continuous action video processing submodules for processing assembly videos, wherein the continuous action video processing submodule is composed of a second 3D convolution layer and a third 3D convolution layer, the third 3D convolution layer is a bias item of the second 3D convolution layer, and a ReLU activation function is set after the third 3D convolution layer;

[0024] a third fully connected layer for processing outputs of the plurality of continuous action video processing submodules;

[0025] Two fourth fully connected layers for fusing the output of the second LSTM network with the output of the third fully connected layer;

[0026] A softmax layer is arranged behind the two fourth fully connected layers.

[0027] Further, for the trajectory recognition module:

[0028] The data of the assembly trajectory includes the three-dimensional space position coordinates p and the unit quaternion q: {(p x ,p y ,p z ),(q0,q1,q2,q3)}, the extracted trajectory features include velocity, acceleration, angular velocity, angular acceleration, curvature and torsion;

[0029] The curvature c and the torsion τ are calculated as follows:

[0030]

[0031]

[0032] Here, the points above the vector represent the time derivative.

[0033] Further, the training of the neural network model includes the following steps:

[0034] The recognition and segmentation results are optimized through the weight function, which is calculated as follows:

[0035]

[0036]

[0037] The input of the neural network model is 16 frames of video data from frame i to frame i+15. k_j represents the probability that the kth frame in the model input data is recognized as action j, W(ik) represents the weight function, and L i represents the optimal weight.

[0038] Furthermore, the network model training of the continuous action recognition segmentation model includes the following steps:

[0039] Obtaining an assembly trajectory and an assembly video of a single assembly operation of the single action element, and collecting them into a single action dataset;

[0040] The assembly trajectory and assembly video of the entire continuous assembly operation of the target object are obtained and collected into a continuous action dataset.

[0041] Another aspect of the technical solution of the present invention relates to a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement the above-mentioned method when executed by a processor.

[0042] Another aspect of the technical solution of the present invention relates to a system for robot assembly skill learning, comprising: a computer device, wherein the computer device includes the above-mentioned computer-readable storage medium.

[0043] The beneficial effects of the present invention are as follows.

[0044] The multimodal assembly demonstration method and system for robot assembly skill learning in the technical solution of the present invention is a multimodal action acquisition, recognition, segmentation and planning method, system and device based on a manual assembly action demonstration platform. Different assembly action units (single action units) are identified by fusing assembly video and six-dimensional assembly trajectory, thereby semantically segmenting continuous assembly trajectory, providing semantic support for assembly skill learning, and then formulating different robot assembly motion planning strategies to achieve the goal of improving the success rate and efficiency of robot assembly. It is proposed to segment the assembly process into a starting state, a state to be aligned, a state to be assembled and a completed state, and an assembly action unit library is designed. It effectively decomposes industrial assembly tasks; in assembly skill learning, the video dimension information and precise six-dimensional trajectory information are integrated for action recognition, which can not only make up for the shortcoming that pure video action recognition is difficult to obtain the precise trajectory of the target, but also make up for the low accuracy of semantic segmentation of pure trajectory action recognition, and is suitable for most industrial assembly; based on the action recognition segmentation results, different motion planning strategies are adopted for different action designs, which can improve the success rate and efficiency of robot assembly, and can be applied to robot programming situations with complex assembly processes and high precision requirements. It can quickly and flexibly transfer workers' assembly skills to robot assembly, which is conducive to expanding the scope of application of robot assembly. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG. 1 is a schematic structural diagram and a real photograph of a multimodal data acquisition device according to an embodiment of the present invention.

[0046] FIG. 2 is a schematic structural diagram and a real photograph of a human-machine interface of a multimodal data acquisition device according to an embodiment of the present invention.

[0047] FIG3 is a schematic structural diagram and a real photograph of a marking device of a multimodal data acquisition device according to an embodiment of the present invention.

[0048] Figure 4 is a basic flow chart of the method according to the present invention.

[0049] Figure 5 Schematic diagram of the network structure of the action recognition model according to the method of the present invention.

[0050] Figure 6 Schematic diagram of the network structure of the continuous action segmentation model according to the method of the present invention.

[0051] Figure 7 It is a schematic diagram of recognition and segmentation results of the continuous action recognition and segmentation model according to the method of the present invention.

[0052] Figure 8 It is a flow chart of network model training of continuous action recognition and segmentation model according to the method of the present invention.

[0053] Reference numerals:

[0054] 100. Multimodal data acquisition device; 110. Optical motion capture platform; 111. Capture camera; 112. Human-machine interface; 113. Connecting part; 114. Supporting part; 115. First reflective marking point; 116. Connecting rod; 120. Video acquisition camera; 130. Data marking device; 131. Active luminous marking point; 132. Second reflective marking point; 133. Light bracket; 140. Marking point switch.

[0055] It should be understood that the text contents presented in the above-mentioned drawings have become part of the text contents of this specification, and any combination with the embodiments in the specific implementation part of this specification does not violate the technical solution principle of the present invention. DETAILED DESCRIPTION

[0056] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in conjunction with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

[0057] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0058] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0059] 1 to 3, the present invention is a multimodal data acquisition device 100 based on robot assembly skill learning, wherein the target object is set on a manual assembly action demonstration platform. The multimodal data acquisition device 100 includes an optical motion capture platform 110, a video acquisition camera 120, a data marking device 130 and a marking point switch 140. The optical motion capture platform 110 includes eight capture cameras 111 and a human-machine interface 112.

[0060] Eight capture cameras 111 are arranged at two heights in a row, and the eight capture cameras 111 are arranged in pairs on the four sides (four corners) of the learning platform. Referring to FIG1 , the eight capture cameras 111 are installed on four camera brackets. Two capture cameras 111 are installed at different heights of the same camera bracket, wherein the higher capture camera 111 is installed at the top of the camera bracket, and the lower capture camera 111 is installed in the middle of the camera bracket and below the higher capture camera 111. The four lower capture cameras 111 are on the same horizontal plane, that is, the four lower capture cameras 111 are all at a first height, and at the same time, the four higher capture cameras 111 are on the same horizontal plane, that is, the four higher capture cameras 111 are all at a second height, wherein the first height is lower than the second height. The four camera brackets are evenly distributed on the four corners of the manual assembly action demonstration platform, and the lenses of the eight capture cameras 111 are all facing the parts to be assembled (target objects). Furthermore, the embodiment of the present invention may adopt the Optitrack optical motion capture platform 110, and the capture camera 111 adopts the Optitrack Prime X41. The above embodiment is only for illustration and is not specifically limited. By setting multiple lenses at different heights and orientations, it is helpful to obtain a diversified perspective, and the object motion trajectory of the target object in the workspace can be obtained in all directions. At the same time, by adjusting the angle of the capture camera 111, it is beneficial to make all the capture cameras 111 cover the entire workspace, and make the field of view center of the capture camera 111 coincide with the center of the workspace, so as to obtain more accurate assembly trajectory data.

[0061] The video acquisition camera 120 is arranged on one side of the manual assembly action demonstration platform for collecting the assembly operation video, and the angle of the video acquisition camera 120 is adjusted so that it can cover the working area of ​​the manual assembly action demonstration platform. Further, the video acquisition camera 120 of the embodiment of the present invention can adopt an RGB camera, and the above embodiment is only used as an example and is not specifically limited. The data marking device 130 is arranged between the video acquisition camera 120 and the target object, and the data marking device 130 includes a lamp bracket 133, an active light-emitting mark point 131 and two second reflective mark points 132, and the active light-emitting mark point 131 is arranged within the shooting field of view of the video acquisition camera 120. As shown in FIG. 2, the lamp bracket 133 is arranged on the learning platform, the active light-emitting mark point 131 is arranged at the top of the lamp bracket 133, and the two second reflective mark points 132 are respectively connected to the lamp bracket 133 through two connecting rods 116, and the connecting rods 116 of the two second reflective mark points 132 are perpendicular to each other, and the two second reflective mark points 132 are arranged on the same horizontal plane. Furthermore, the active light-emitting mark point 131 of the data marking device 130 of the embodiment of the present invention is set as an infrared lamp, and the wavelength of the infrared lamp is set to 850mm, so as to ensure that the optical movement of the active light-emitting mark point 131 after the light is turned on can be collected. The turning on and off (lighting on and off) of the active light-emitting mark point 131 is controlled by the mark point switch 140. Further, the mark point switch 140 can be a foot switch or an electric switch electrically connected to the control system.

[0062] Referring to FIG. 3 , the human-machine interface 112 includes a support portion 114, a connection portion 113 and a plurality of first reflective marking points 115, wherein the support portion 114 is disposed on one side of the connection portion 113, and the other side of the connection portion 113 is connected to the target object. It should be noted that the connection portion 113 can also be used for connecting the target object to the robotic arm. A plurality of first reflective marking points 115 are disposed on the support portion 114 to obtain multi-degree-of-freedom posture data of the target object. Further, the human-machine interface 112 of the embodiment of the present invention is provided with four first reflective marking points 115, and the four first reflective marking points 115 are not coplanar, so that the first reflective marking points can constitute an asymmetric rigid body by setting four first reflective marking points 115. By connecting the human-machine interface 112 to the target object, the six-degree-of-freedom posture data (assembly trajectory) of the target object can be obtained, and at the same time, occlusion can be reduced and the signal can be amplified, thereby facilitating the acquisition of more subtle and sensitive object motion data.

[0063] Further, in some specific embodiments of the present invention, the connecting portion 113 is a flat plate, the target object and the supporting portion 114 are respectively arranged on two opposite planes of the connecting portion 113, the supporting portion 114 is an elongated strip, one end of the supporting portion 114 is connected to the connecting portion 113, and the other end is connected to the first reflective marking point 115 through a connecting rod 116. One of the first reflective marking points 115 is arranged on the center line of the supporting portion 114 and on the side of the supporting portion 114 away from the connecting portion 113. The remaining three first reflective marking points 115 are all arranged on the first plane, and the first plane is perpendicular to the center line of the supporting portion 114. Two of the first reflective marking points 115 on the first plane are both on the first straight line, and the connecting rod 116 of the remaining first reflective marking point 115 on the first plane is perpendicular to the first straight line. The above embodiments are merely illustrative and are not specifically limited.

[0064] Reference Figure 4 The technical solution of the present invention is a multi-modal demonstration method (i.e., assembly demonstration method) and system applied to robot assembly skill learning. Based on the manual assembly action demonstration platform, the multi-modal assembly demonstration method applied to the robot assembly motion trajectory generally includes the following steps:

[0065] S100, collecting assembly trajectories and assembly videos of target objects and human hands through a multimodal data collection device 100, and associating the assembly trajectories and the assembly videos;

[0066] S300, based on the assembly action unit library, according to the assembly video and the assembly trajectory, the assembly operation of the target object is identified and segmented by a multimodal fusion continuous action recognition and segmentation model to obtain a series of continuous single action units;

[0067] S400 , based on the motion planning strategy library, according to the type of each single action element, obtain its corresponding motion planning strategy to generate a robot assembly motion plan for the entire assembly operation of the target object.

[0068] Specific implementation of step S100

[0069] The data acquisition of the embodiment of the present invention is acquired and processed by a multimodal data acquisition device 100 to obtain the assembly trajectory and assembly video of the target object, and the assembly trajectory and the assembly video are associated. Specifically, the target object is placed on a manual assembly action demonstration platform, the human-machine interface 112 is connected to the target object, and capture cameras 111 and video acquisition cameras 120 are arranged around the learning platform. Among them, the six-dimensional posture data (i.e., assembly trajectory) of the target object is obtained by setting eight capture cameras 111 around the target object (four corners) and installing the first reflective marking point 115 on the target object, while the RGB camera (video acquisition camera) is used to synchronously collect the video data (i.e., assembly video) of the assembly operation of the target object and the human hand, and the data marking device 130 is used to generate the marking data of the assembly action of the object to be assembled (target object). Its data acquisition and processing steps include:

[0070] S110, collecting assembly trajectory data, assembly video data and marking data of the target object during the assembly process through the multimodal data collection device 100, wherein the data collection includes the following steps:

[0071] S111, turning on the capture camera 111 and the video acquisition camera 120 to perform continuous data acquisition, wherein the capture camera 111 is used to collect assembly trajectory data and marking data, and the video acquisition camera 120 is used to collect assembly video data;

[0072] S112, when the target object begins to be assembled, the action head mark is obtained through the active light-emitting mark point 131, the second reflective mark point 132 and the first reflective mark point 115. Specifically, when the target object is assembled, the active light-emitting mark point 131 is turned on by the mark point switch 140 to make it light up, and when the lighting time reaches about 0.2 seconds, the active light-emitting mark point 131 is turned off by the mark point switch 140 to turn off the light.

[0073] S113, during the assembly process of the target object, the active light-emitting mark point 131 is controlled to emit light and extinguish to obtain the action mid-stage mark. Specifically, the active light-emitting mark point 131 maintains a fixed frequency of light emission and extinguishing, and the capture camera 111 and the video acquisition camera 120 continuously collect data.

[0074] S114, when the target object is finished being assembled, the action end mark is obtained through the active luminous marking point 131, the second reflective marking point 132 and the first reflective marking point 115. Specifically, after the assembly action is completed, the active luminous marking point 131, the second reflective marking point 132 and the first reflective marking point 115 are turned on through the marking point switch 140, and when the lighting time reaches about 0.2 seconds, the active luminous marking point 131, the second reflective marking point 132 and the first reflective marking point 115 are turned off through the marking point switch 140.

[0075] S120: Process the assembly trajectory data and the assembly video data according to the marking data to align the assembly trajectory data and the assembly video data. The data processing operation includes the following steps:

[0076] S121, according to the mid-segment mark of the action, the assembly trajectory data is cropped to obtain multiple trajectory segments. Specifically, the six-dimensional trajectory data (i.e., assembly trajectory data) captured by the optical motion capture platform 110 is processed offline, and according to the mid-segment mark of the action obtained in step S110, the assembly trajectory data of the entire assembly operation of the target object is cropped to obtain multiple desired trajectory segments.

[0077] S122, label the assembly video data frame by frame according to the on and off of the active light-emitting mark point 131, and black out the area where the data labeling device 130 is located in each frame of the video. Specifically, the assembly video data collected by the video acquisition camera 120 is processed offline, and firstly, labels are added to the assembly video data frame by frame according to the brightness of the active light-emitting mark point 121 (i.e., the light-on and off of the active light-emitting mark point 131), and then black out the area where the data labeling device 130 is located in each frame of the assembly video.

[0078] S123, perform trajectory compensation on the assembly trajectory data, and align the assembly trajectory data and the assembly video data according to the action head mark and the action tail mark. Specifically, according to the action head mark, align the assembly trajectory data and the assembly video data at the time when the target object starts to assemble, and at the same time, according to the action tail mark, align the assembly trajectory data and the assembly video data at the time when the target object completes assembly. Furthermore, because the assembly trajectory data captured by the optical motion capture platform 110 and the assembly video data collected by the video acquisition camera 120 cannot be completely consistent or multiple, after the video frames at the start and end of the assembly are aligned with the assembly trajectory, it cannot be guaranteed that each frame of assembly video data can correspond to the original motion capture trajectory data. In view of the above problems, the method of the embodiment of the present invention adopts a trajectory compensation strategy based on Gaussian process regression to compensate the assembly trajectory data.

[0079] Specific implementation of step S200

[0080] The continuous action recognition and segmentation model in the method of the present invention integrates assembly video data and accurate six-dimensional assembly trajectory data, and designs an effective tooling assembly task decomposition by designing an assembly action unit library, and realizes semantic segmentation of continuous assembly actions by identifying continuous action elements of different assembly actions. The multimodal fusion continuous action recognition and segmentation model includes an action recognition model based on a 3D convolutional neural network model and a continuous action segmentation model, and the recognition and segmentation of assembly actions includes the following steps:

[0081] S210, based on the assembly action unit library, the trajectory recognition module based on the LSTM network model of the action recognition model performs feature extraction on the assembly trajectory to realize trajectory action recognition, the video action recognition module based on the C3D network model of the action recognition model performs video action recognition on the assembly video, and the trajectory action recognition result and the video action recognition result are merged to obtain a single action recognition result of the target object;

[0082] S220, based on the assembly action unit library, according to the single action recognition result, and according to the assembly trajectory and the assembly video, the continuous assembly operation of the target object is segmented by a continuous action segmentation model to obtain a series of continuous single action units (see Figure 7 ).

[0083] In some specific embodiments of the present invention, the types of single action elements included in the assembly action unit library include transport action elements, alignment action elements, plug-in action elements, fastening action elements, and tearing action elements. Specifically, in the setting of the assembly action unit library, the entire assembly operation process is first divided into four states, namely, the starting state, the state to be aligned, the state to be assembled, and the completed state. The entire assembly process of the target item is divided into the starting state, the state to be aligned, the state to be assembled, and the completed state, and the assembly action is divided into different single action elements such as "transport action element, alignment action element, plug-in action element, fastening action element, and tearing action element".

[0084] Among them, the action from the starting state to the state to be aligned is named "transport action element", which is specifically defined as the action of moving two objects to be assembled from the completely separated starting state to the state to be aligned in free space. Next, the action from the state to be aligned to the state to be assembled is defined as "alignment action element", and the "alignment action element" is specifically defined as the action of finely adjusting the relative posture between the parts to be assembled so that the connection and matching parts of the two parts to be assembled are in a relatively parallel relative posture that is convenient for assembly. Finally, the actions from the state to be assembled to the completed state are defined as "plug-in action element", "fastening action element" and "tearing action element" respectively.

[0085] Further, the "plug-in action element" can be subdivided into "translation insertion action element", "rotation insertion action element" and "spiral insertion action element". Among them, the "translation insertion action element" is specifically defined as the translation of the component to be assembled along the axis into the hole, key or groove to a certain depth, the "rotation insertion action element" is specifically defined as rotating the component to be assembled around the axis into the hole, key or groove to a certain depth, and the "spiral insertion action element" is specifically defined as rotating the component to be assembled around the central axis and translating it along the central axis into the hole, key or groove to a certain depth. Further, the "fastening action element" can be subdivided into "pressing action element" and "flat pressing action element". The "pressing action element" is specifically defined as applying an axial force perpendicular to the contact surface to move the object to be assembled along the direction of pressure, which includes pressing to buckle and pressing to fit. The "flat pressing action element" is specifically defined as applying a downward oblique force perpendicular to the contact surface, and the force applicator moves along a plane (perpendicular to the direction of pressure). Further, the action of the "tearing action element" in the assembly state is specifically defined as the alignment of the finger and the protruding part of the film, and the finger and the film are completely separated from the attachment in the completed state.

[0086] In some specific embodiments of the present invention, the action recognition model includes a trajectory recognition module based on an LSTM network model for performing trajectory action recognition on the assembly trajectory and a video action recognition module based on a C3D network model for performing video action recognition on the assembly video. The network model of the action recognition model is shown in Figure 5 . Among them, the trajectory action recognition includes a first feature extraction submodule for extracting assembly trajectory features and a first LSTM network for processing trajectory features. The video action recognition module includes multiple single-action video processing submodules for processing assembly videos and a first fully connected layer for processing the outputs of multiple single-action video processing submodules, wherein the single-action video processing submodule consists of a first 3D convolutional layer and a first 3D pooling layer, and the zero padding on the time dimension of the first 3D convolutional layer is replaced with part of the information in the previous time step. The two second fully connected layers fuse the output of the first LSTM network (trajectory action recognition result) with the output of the first fully connected layer (video action recognition result), and then enter the softmax layer.

[0087] For further information, see Figure 8 ,The action recognition module is trained by a single action dataset. A single assembly operation of a single action element is performed on the artificial assembly action demonstration platform, and the operation trajectory data and operation video data of the single operation process are collected through the multimodal data acquisition module. The obtained data are collected into a single action dataset, and the action recognition module is trained for action recognition through the single action dataset.

[0088] In a specific embodiment of the present invention, for the trajectory recognition module, the original data of the assembly trajectory obtained includes the three-dimensional space position coordinates p and the unit quaternion q: {(p x ,py ,p z ),(q0,q1,q2,q3)}, the trajectory features extracted by preprocessing include velocity, acceleration, angular velocity, angular acceleration, curvature and torsion. The velocity, acceleration, angular velocity and angular acceleration are calculated by differential method, while the curvature c and torsion τ are calculated as follows:

[0089]

[0090]

[0091] In the formula, the point above the vector represents the derivative with respect to time. The vector cross multiplication operation rule is as follows:

[0092]

[0093] After feature extraction, the trajectory recognition module adopts LSTM network, takes three-dimensional spatial position coordinates, unit quaternion, velocity, acceleration, angular velocity, angular acceleration, curvature, and torsion as the input of the trajectory recognition module, and pre-trains the trajectory module so that it can recognize single assembly actions.

[0094] In a specific embodiment of the present invention, for a video action recognition module, a public video data set is used to pre-train the video action recognition module in the network, and then the video data in the constructed single action data set is used to train the pre-trained video action recognition module. It should be noted that the video action recognition module can be based on a C3D network or can be replaced by other video action recognition networks.

[0095] In some specific embodiments of the present invention, the network model of the continuous action segmentation module is shown in Figure 6 , whose network architecture includes: a second feature extraction submodule for extracting assembly trajectory features; a second LSTM network for processing trajectory features; multiple continuous action video processing submodules for processing assembly videos, wherein the continuous action video processing submodule consists of a second 3D convolution layer and a third 3D convolution layer, the third 3D convolution layer is a bias item of the second 3D convolution layer, and a ReLU activation function is set after the third 3D convolution layer; a third fully connected layer for processing the outputs of multiple continuous action video processing submodules; two fourth fully connected layers for fusing the output of the second LSTM network with the output of the third fully connected layer; a softmax layer is set after the two fourth fully connected layers.

[0096] In an application scenario, the data size of the input 3D convolution layer is set to n1×n2×n3, the number of channels is c, and the size of the 3D convolution kernel is f×f×f. Then, after the convolution operation, the output data size N is:

[0097] N=(n1-f+1)×(n2-f+1)×(n3-f+1)

[0098] In the neural network of the action recognition model in step S210 (see Figure 5 The convolution layer in the video stream has a 3D convolution kernel size of 3×3×3 and a 3D convolution layer with zero padding. The zero padding in the time dimension of the 3D convolution layer is removed and replaced with part of the information of the same layer in the previous time step. In this case, the number of axes removed and added is 2, which does not affect the network size and facilitates the migration of the training parameters of step S210 to the continuous action segmentation module.

[0099] Furthermore, the recognition and segmentation results are optimized through the weight function, which is calculated as follows:

[0100]

[0101]

[0102] The input of the neural network model is 16 frames of video data from the i-th frame to the i+15th frame, P k_j represents the probability that the kth frame in the model input data is recognized as action j, W(ik) represents the weight function, and L i Represents the optimal weight. A variety of weight functions are proposed as follows, where x is ik, and the optimal weight function is selected through experiments.

[0103] W0=1

[0104]

[0105]

[0106]

[0107] In some specific embodiments of the present invention, see Figure 8 ,The continuous action segmentation module is trained by the continuous action dataset. The continuous assembly operation of the entire assembly process of the target object is performed on the artificial assembly action demonstration platform, and the operation trajectory data and operation video data of the continuous operation process are collected by the multimodal data acquisition module. The obtained data are collected into the continuous action dataset, and the continuous action segmentation module is trained for action segmentation through the continuous action dataset.

[0108] Specific implementation of step S300

[0109] Based on the assembly action unit library, different motion planning strategies are designed for different types of assembly actions (single action elements) according to their accuracy requirements, motion characteristics, etc., to form a motion planning strategy library. Then, according to the series of continuous single action elements obtained in step 200, according to the type of each single action element, the motion planning strategy corresponding to the type to which it belongs is found in the motion planning measurement library, so a series of continuous single action elements can obtain a series of motion planning strategies, and the obtained different motion planning strategies are connected to complete the robot assembly motion planning of the entire assembly process of the target object.

[0110] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in an assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed ASIC for this purpose.

[0111] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0112] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicates with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether it is removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, RS1M, ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0113] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0114] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A multimodal demonstration method for robot assembly skill learning, characterized in that: The method comprises the following steps: S100, collecting assembly tracks and assembly videos of a target object and a human hand through a multimodal data collection device (100), and associating the assembly tracks and the assembly videos; S200, based on the assembly action unit library, according to the assembly video and the assembly trajectory, the assembly operation of the target object is identified and segmented by a multimodal fusion continuous action recognition and segmentation model to obtain a series of continuous single action units; S210, based on the assembly action unit library, the trajectory recognition module based on the LSTM network model of the action recognition model performs feature extraction on the assembly trajectory to realize trajectory action recognition, the video action recognition module based on the C3D network model of the action recognition model performs video action recognition on the assembly video, and the trajectory action recognition result and the video action recognition result are merged to obtain a single action recognition result of the target object; S220, based on the assembly action unit library, according to the single action recognition result, and according to the assembly trajectory and the assembly video, segmenting the continuous assembly operation of the target object by the continuous action segmentation model to obtain a series of continuous single action units; The types of single action elements included in the assembly action unit library include transport action elements, alignment action elements, plug-in action elements, fastening action elements and tearing action elements; S300. Based on a motion planning strategy library, and according to the type of each single action element, obtain the corresponding motion planning strategy to generate a robot assembly motion plan for the entire assembly operation of the target object.

2. The method according to claim 1, wherein: In step S200: The continuous action recognition and segmentation model includes an action recognition model based on a 3D convolutional neural network model and a continuous action segmentation model; wherein the action recognition module is trained by a single action data set, and the continuous action segmentation module is trained by a continuous action data set.

3. The method according to claim 2, characterized in that The network model of the action recognition module includes: A first feature extraction submodule for extracting assembly trajectory features; a first LSTM network for processing trajectory features; A plurality of single-action video processing submodules for processing assembly videos, wherein the single-action video processing submodule is composed of a first 3D convolution layer and a first 3D pooling layer, and the zero padding in the time dimension of the first 3D convolution layer is replaced by part of the information in the previous time step; a first fully connected layer for processing outputs of the plurality of single-action video processing submodules; Two second fully connected layers for fusing the output of the first LSTM network with the output of the first fully connected layer; A softmax layer is arranged behind the two second fully connected layers.

4. The method according to claim 2, characterized in that: The network model of the continuous action segmentation model includes: A second feature extraction submodule for extracting assembly trajectory features; a second LSTM network for processing trajectory features; A plurality of continuous action video processing submodules for processing assembly videos, wherein the continuous action video processing submodule is composed of a second 3D convolution layer and a third 3D convolution layer, the third 3D convolution layer is a bias item of the second 3D convolution layer, and a ReLU activation function is set after the third 3D convolution layer; a third fully connected layer for processing outputs of the plurality of continuous action video processing submodules; Two fourth fully connected layers for fusing the output of the second LSTM network with the output of the third fully connected layer; A softmax layer is arranged behind the two fourth fully connected layers.

5. The method according to claim 2, characterized in that: For the trajectory recognition module: The data of the assembly trajectory includes the three-dimensional space position coordinates p and the unit quaternion q: {(p x ,p y ,p z ),(q0,q1,q2,q3)}, the extracted trajectory features include velocity, acceleration, angular velocity, angular acceleration, curvature and torsion; The curvature c and the torsion τ are calculated as follows: Here, the points above the vector represent the time derivative.

6. The method according to claim 2, wherein: The training of the neural network model includes the following steps: The recognition and segmentation results are optimized through the weight function, which is calculated as follows: The input of the neural network model is 16 frames of video data from frame i to frame i+15. k_j represents the probability that the kth frame in the model input data is recognized as action j, W(ik) represents the weight function, and L i represents the optimal weight.

7. The method according to claim 2, wherein: The network model training of the continuous action recognition segmentation model includes the following steps: Obtaining an assembly trajectory and an assembly video of a single assembly operation of the single action element, and collecting them into a single action dataset; The assembly trajectory and assembly video of the entire continuous assembly operation of the target object are obtained and collected into a continuous action dataset. 8 . A computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 1 to 7.

9. A multimodal demonstration system for robot assembly skill learning, characterized in that: include: A computer device comprising the computer-readable storage medium according to claim 8.

Images