Assembly robot system and method based on multimodal information fusion and apprentice learning

Abstract

The present invention relates to the field of intelligent robot technology, and specifically to an assembly robot system and method based on multimodal information fusion and apprentice learning, wherein a teaching task is determined, a demonstrator demonstrates the assembly process through body movements, a sensor records the demonstrator's movement information, and a multimodal information fusion algorithm is used to fuse the multimodal data acquired by the sensor network. The robot autonomously learns relevant information, performs the assembly task, verifies the learning effect, and applies the learned skills to actual production to realize autonomous planning of the assembly process. The robot can quickly learn and adapt to new tasks or environments. The learning speed and accuracy of the robot for complex assembly skills are significantly improved, and the dependence on manpower is reduced. The adaptability and autonomy of the robot in an unstructured environment are enhanced, and the success rate and efficiency of the assembly task are improved. Through intelligent planning and autonomous adjustment, the uncertainty in the assembly process can be effectively dealt with.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to an assembly robot system and method based on multimodal information fusion and apprentice learning. Background Art

[0002] Robots are not only key supporting equipment for advanced manufacturing, but also an important entry point for improving human lifestyles. In today's industrial production field, the popularity of automated assembly lines has greatly improved production efficiency and product quality. However, traditional industrial robots still face some technical challenges when performing tasks on assembly lines, which limits their application scope and further improvement of efficiency.

[0003] Traditional industrial robots usually learn tasks through teaching methods, which requires technicians to personally guide the robot to complete the action. This process is not only time-consuming and laborious, but also in the case of complex tasks, the operator's operation level will greatly affect the assembly accuracy. More importantly, this control process is highly coupled with the production task, resulting in the robot usually only being able to perform pre-programmed actions, lacking flexibility and difficulty in handling new or non-standard assembly requirements. Although drag teaching can improve programming efficiency in some cases, it does not enable the robot to truly have the ability to identify the precise position of the workpiece. Therefore, if the workpiece moves, the robot will still operate along the preset path, which will have a negative impact on the accuracy and production efficiency of the operation.

[0004] In view of the above situation, it has become an urgent need to develop a new generation of intelligent assembly robot technology that can quickly learn complex skills, autonomously plan assembly paths and adapt to changing assembly objects. Summary of the invention

[0005] The purpose of the present invention is to provide an assembly robot system and method based on multimodal information fusion and apprentice learning, which solves the problems that traditional industrial robots have difficulty learning complex skills, have poor adaptability to unknown assembly objects, and have a strong dependence on manual teaching.

[0006] To achieve the above object, the present invention provides a method based on multimodal information fusion and apprenticeship learning, in which the sensor network of the robot equipment has been deployed during production, the demonstrator is familiar with the skills required for the assembly task, and the deep learning network has been pre-trained and deployed before the assembly robot leaves the factory, including the following steps:

[0007] S101: Clarify the assembly objectives and specific requirements of the teaching task, and analyze the required assembly skills;

[0008] S102: The demonstrator demonstrates the assembly process through body movements, and the sensor network records the demonstrator's movement information;

[0009] S103: fusing the multimodal data acquired by the sensor network through a multimodal information fusion algorithm;

[0010] S104: The robot makes autonomous decisions and plans routes for complex assembly;

[0011] S105: The robot performs assembly tasks to verify the learning effect;

[0012] S106: The robot applies the learned skills to actual production and realizes autonomous planning of the assembly process.

[0013] Among them, the assembly objectives and specific requirements of the teaching task are clarified, and the required assembly skills are analyzed. The steps also include:

[0014] (1) Determine the design specifications, materials, dimensions and functional requirements of the products or components to be assembled;

[0015] (2) Clarify the shapes, sizes and relationships of all parts that need to be assembled;

[0016] (3) Determine the assembly order of the parts;

[0017] (4) Determine the optimal speed of the assembly process based on production efficiency requirements.

[0018] The demonstrator demonstrates the assembly process through body movements, and the sensor network records the demonstrator's movement information. The steps also include:

[0019] The demonstrator wears a capture tool that captures force feedback data on the demonstrator's hands, and uses a depth camera to record the complex assembly process demonstrated by the demonstrator through body movements.

[0020] Wherein, the multimodal data acquired by the sensor network is fused by a multimodal information fusion algorithm, and the step further includes:

[0021] (1) Collect force feedback data, color video data, and depth video data during the demonstrator’s assembly demonstration;

[0022] (2) performing denoising and normalization processing on the collected data, wherein the force feedback data is denoised using a wavelet transform method, and the color video data and the depth video data are denoised using a Gaussian filter method. After denoising, normalization processing is performed to scale the data to a uniform range;

[0023] (3) Feature extraction is performed on the processed color video data and depth video data respectively, and then the features of the two modalities are directly spliced, feature extraction is performed on the force feedback data, and finally the video features and force feedback features are fused using a multimodal fusion module.

[0024] The robot makes autonomous decisions and plans the route for complex assembly, and the steps also include:

[0025] (1) Accurately estimate the spatial position and orientation of the target object through the 6D pose estimation algorithm;

[0026] (2) Perform robot motion path planning;

[0027] (3) Establish a kinematic model of the robot and calculate the rotation of the robot's joints to ensure that the robot can reach the precise position of the target object;

[0028] (4) Build a deep convolutional neural network to calculate the location information of the accessory, such as the grasping point, angle, and width;

[0029] (5) Check the gripping point and gripping force;

[0030] (6) The robot arm moves to the position of the object to be assembled;

[0031] (7) The robot autonomously learns the assembly relationship of parts;

[0032] (8) Calculate the deviations between parts and accessories to autonomously plan the matching method to achieve accurate matching.

[0033] Wherein, the spatial position and direction of the target object are accurately estimated by a 6D pose estimation algorithm, and the steps further include:

[0034] ①Use the scale-invariant feature conversion algorithm to extract features from each frame in the video;

[0035] ② Perform feature matching between consecutive video frames to track the movement of objects, and then use the feature matching results to estimate the movement of objects between video frames;

[0036] ③Build a semantic segmentation model;

[0037] ④Build a pose estimation network to calculate the position parameters, rotation parameters and confidence parameters of each object except the background.

[0038] The robot performs the assembly task to verify the learning effect, and the steps further include:

[0039] (1) Training in a simulated environment to reduce errors and risks that may occur in actual assembly;

[0040] (2) The robot performs the assembly task for the first time in a real environment and makes preliminary attempts;

[0041] (3) Closely monitor the robot's performance, including accuracy, speed, and stability, while it performs its tasks;

[0042] (4) Collect data while the robot performs tasks;

[0043] (5) If the robot does not perform as expected, diagnose the problem;

[0044] (6) Adjust the robot’s learning algorithm based on the evaluation results and perform iterative optimization.

[0045] An assembly robot system based on multimodal information fusion and apprentice learning, including a color video frame encoder, a depth video frame encoder, a video data fusion module, a force feedback data encoder and a multimodal feature fusion module;

[0046] The color video frame encoder and the depth video frame encoder are respectively connected to the video data fusion module; the multimodal feature fusion module is respectively connected to the force feedback data encoder and the video data fusion module.

[0047] The present invention discloses an assembly robot system and method based on multimodal information fusion and apprentice learning, wherein a teaching task is determined, including specific goals and requirements of the assembly task, a demonstrator demonstrates the assembly process through body movements, a sensor records the demonstrator's movement information, and a multimodal information fusion algorithm is used to fuse the multimodal data obtained by the sensor network to build a unified perception of the environment and movement. The robot autonomously learns relevant information, including pose estimation, semantic segmentation, path planning, establishment of kinematic models, grasping methods, use of tools, assembly relationships, etc., through a robot autonomous planning method, and the robot performs the assembly task and verifies the learning effect. The robot applies the learned skills to actual production to realize autonomous planning of the assembly process. The robot can quickly learn and adapt to new tasks or environments. In the present invention, the robot can actively identify the precise position of the workpiece and operate it, which significantly improves the robot's learning speed and accuracy for complex assembly skills and reduces dependence on manpower. The adaptability and autonomy of the robot in an unstructured environment are enhanced, and the success rate and efficiency of the assembly task are improved. Through intelligent planning and autonomous adjustment, the uncertainty in the assembly process is effectively dealt with, and the intelligence level of the overall assembly system is improved. Promote the development of intelligent assembly technology and provide key technical support for the transformation and upgrading of the manufacturing industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0049] Figure 1 It is a connection block diagram of the multimodal feature fusion module of the present invention.

[0050] Figure 2 It is a schematic diagram of a single-frame image pose estimation algorithm of the present invention.

[0051] Figure 3 It is a flow chart of the A* algorithm adopted by the present invention.

[0052] Figure 4 It is a step flow chart of the multimodal information fusion and apprentice learning method of the present invention.

[0053] In the figure: 201-color video frame encoder, 202-depth video frame encoder, 203-video data fusion module, 204-force feedback data encoder, 205-multimodal feature fusion module. DETAILED DESCRIPTION

[0054] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0055] The first embodiment of the present application is:

[0056] See also Figures 1 to 4 ,in, Figure 1 It is a connection block diagram of the multimodal feature fusion module of the present invention. Figure 2 It is a schematic diagram of a single-frame image pose estimation algorithm of the present invention. Figure 3 It is a flow chart of the A* algorithm adopted by the present invention. Figure 4 The present invention provides a method based on multimodal information fusion and apprenticeship learning, wherein the sensor network of the robot equipment has been deployed during production, the demonstrator is familiar with the skills required for the assembly task, and the deep learning network has been pre-trained and deployed before the assembly robot leaves the factory, including the following steps:

[0057] S101: Clarify the assembly objectives and specific requirements of the teaching task, and analyze the required assembly skills;

[0058] Specifically, the premise of the present invention is that the sensor network of the robot equipment has been deployed during production, the demonstrator is familiar with the skills required for the assembly task, and the deep learning network has been pre-trained and deployed when the assembly robot leaves the factory. Determine the products or components that need to be assembled, including their design specifications, materials, dimensions and functional requirements. Identify all parts that need to be assembled, including their shapes, sizes and relationships to each other. Determine the order in which the parts are assembled, which may involve complex sequences that need to be followed in specific steps. Determine the optimal speed of the assembly process based on production efficiency requirements.

[0059] S102: The demonstrator demonstrates the assembly process through body movements, and the sensor network records the demonstrator's movement information;

[0060] Specifically, the demonstrator wore a scalable tactile glove (STAG) developed by MIT, which has 550 micro sensors arranged on the entire hand, which can capture as complete force feedback data on the demonstrator's hand as possible. The Microsoft Kinect depth camera was used to record the demonstrator's complex assembly process through body movements for subsequent robot learning.

[0061] S103: fusing the multimodal data acquired by the sensor network through a multimodal information fusion algorithm;

[0062] Specifically, during the teaching process, the pressure sensor collects force feedback data during the demonstrator's assembly demonstration, which is expressed as P = [P1, P2, ..., P n-1 ,P n ], P i =[p1,p2,...,p 549 ,p 550 ], where P1~P n Represented as the force feedback data corresponding to each frame of video, p1~p 550 Represents the force feedback component data of the entire hand in the pressure sensor. The color video data is represented as V = [v1, v2, ..., v n ], where v i Represents pictures of different frames in color video, and the depth video data is represented by D = [d1, d2, ..., d n ],d i Indicates the color video v i The image of the corresponding depth video frame, i represents the sequence number of the frame in the video.

[0063] First, the collected data is denoised and normalized. The force feedback data is denoised using the wavelet transform method. The color video frame and the depth video frame are denoised using the Gaussian filter method.

[0064] ① Wavelet transform

[0065] Discrete Wavelet Transform (DWT) is a mathematical transform used for signal processing. It decomposes the signal into components of different scales on the time-frequency plane. The formula is as follows:

[0066] W j,k =∑ n p n ·h j,k (n)

[0067] Among them, W j,kis the wavelet coefficient; p n is the original force feedback data sequence; h j,k (n) is the Daubechies wavelet function, which is obtained by scaling and translating the original wavelet function; j represents the scale parameter, which controls the scaling of the wavelet; k represents the position parameter, which controls the translation of the wavelet.

[0068] ② Inverse wavelet transform

[0069] Inverse Discrete Wavelet Transform (IDWT) is used to reconstruct the original signal from the wavelet coefficients. j，k To restore the original force feedback data P. The formula is as follows:

[0070]

[0071] Where P′(n) is the reconstructed signal; W j,k is the wavelet coefficient; h j,k (n) is the Daubechies wavelet function; J max and J min are the minimum and maximum sizes of decomposition respectively; k is the position parameter that controls the translation of the wavelet.

[0072] ③ Normalization

[0073] After denoising, normalization is performed to scale the data to a uniform range, which helps to speed up the convergence of the algorithm while ensuring that data from different modalities are comparable when fused.

[0074] The min-max normalization method is used here:

[0075]

[0076] Among them, P i is each element in the reconstructed dataset, min(P) and max(P) are the minimum and maximum values ​​in the reconstructed dataset, respectively.

[0077] ④Gaussian filtering

[0078] The Gaussian filter is based on the Gaussian function, which is a linear filter that smoothes the image spatially. The Gaussian filter is particularly effective in removing Gaussian noise in images while being able to preserve edge information to a certain extent. For a color video frame v i For each pixel point (x,y) in (x,y), its pixel value v after Gaussian filtering i ′(x,y) can be calculated by the convolution kernel G:

[0079]

[0080] Where m and n in the Gaussian kernel G(m,n) represent the indexes along the rows and columns of the Gaussian kernel. The Gaussian kernel G(m,n) is defined as:

[0081]

[0082] For the depth video frame d i In (x,y), the Gaussian formula is the same:

[0083]

[0084] For touch data, color video data and depth video data, we design corresponding encoders to extract features from them. Then we use the video data fusion module to fuse the video frame features. Then we use the multimodal feature fusion module to fuse the pressure features and video features. Finally, we get the fused feature vector of the input data. The framework structure of this part is shown in the figure below. Figure 1 The present invention uses Video Vision Transformer to extract features from color video frame data and depth video frame data, then directly splices the features of the two modalities, uses one-dimensional CNN to extract features from force feedback data, and finally uses a multimodal fusion module to fuse video features and force feedback features. The calculation process can be expressed as:

[0085] F fusion =f(cat(ViViT1(V),ViViT2(D)),CNN(P))

[0086] Among them, F fusion It represents the feature vector after the fusion of video features and force feedback features, cat represents the tensor concatenation operation, ViViT represents the use of Video Vision Transformer to calculate the attention score of the data, CNN represents the use of one-dimensional convolution to extract features from the data, and f represents the feature fusion function.

[0087] S104: The robot makes autonomous decisions and plans routes for complex assembly;

[0088] (1) Accurately estimate the spatial position and orientation of the target object, i.e., the 6D pose (including the position of the three coordinate axes X, Y, and Z and the rotation around these three axes) through the 6D pose estimation algorithm. The flowchart of this step is as follows: Figure 2 shown.

[0089] ① Use the Scale Invariant Feature Transform (SIFT) algorithm to extract features from each frame in the video.

[0090] ② Perform feature matching between consecutive video frames to track the movement of objects. Then use the feature matching results to estimate the movement of objects between video frames.

[0091] ③ Build a semantic segmentation model. For real-time applications, the present invention uses a lightweight SCTNet to distinguish various objects from the background. While maintaining high segmentation accuracy, the network framework also significantly improves the processing speed, making it suitable for application in such real-time scenarios. This step will separate the semantic information of the demonstrator's hands, tools, parts, accessories and background in the video for the robot to make subsequent autonomous decisions on the grasping strategy. The semantically annotated color video frame [V, C] = [[v1, c1], [v2, c2], [v3, c3], ..., [v n ,c n ]], where c represents the semantic category of the pixel points in the color video frame, which is input into SCTNet for training to obtain the model weight that can better segment this type of data. In actual use, the network can segment the color image in the "eye" in real time based on the color video frame transmitted by the "eye" on the robotic arm to distinguish the five types of pixels: the demonstrator's hand, the demonstrator's tools, parts, accessories and background, and provide semantic information for subsequent robot decision-making and grasping.

[0092] ④ Construct a pose estimation network to calculate the pose estimation parameters such as position parameters, rotation parameters and confidence of each object except the background. In order to improve the accuracy of pose estimation, the present invention uses the PoseCNN network as the framework, and replaces the feature extraction part of the network with the multimodal feature fusion module proposed by the present invention to perform 6D pose estimation. When performing video frame analysis, the present invention first annotates the target object in each frame at the pixel level, records the category information of each pixel point and the attributes such as the size and position of the object. These annotation data are organized in the form of a list, in which each element contains the number of the video frame and the corresponding annotation information, and the format is: I = [v i ,c i ], where c i A two-dimensional array representing the category label of each pixel in the i-th frame. By taking these annotated video frame data as input and sending them to the PoseCNN network for training, we get the post i , the 6D pose information of the objects corresponding to these pixel category labels. In actual use, the network can calculate the 6D pose of objects except the background in the visual image transmitted by the "eye" on the robot arm in real time based on the color video frames transmitted by the "eye", and provide 6D pose information for subsequent robot decision-making and grasping.

[0093] (2) Robot motion path planning. In the present invention, the robot needs to autonomously perform two-stage path planning. The first stage is the motion planning from the current position of the robot to the position of the accessory, and the second stage is the path planning from the position of the accessory to the position of the part. According to the semantic information provided by the semantic segmentation network and the pose information provided by the pose estimation network in step (1), the robot position and the position of the accessory are used as the starting points, and the position of the accessory and the position of the part are used as the end points, respectively, to perform two-stage motion path planning.

[0094] Commonly used path planning algorithms for robots include A* algorithm, Dijkstra algorithm, RRT (rapid random tree), etc. In the invention, A* algorithm is used for path planning.

[0095] A* algorithm (A-Star Search Algorithm) is the most effective direct search method for solving the shortest path in a static road network. It is also an effective algorithm for solving many search problems. It is widely used in indoor robot path search, game animation path search, etc. A* algorithm is a heuristic search algorithm. It is a fusion algorithm based on the depth first search algorithm (Depth First Search, DFS) and the breadth first search algorithm (Breadth First Search, BFS). It guides the search direction through the heuristic function (Heuristic Function) to find the optimal path from the starting point to the end point.

[0096] The A* algorithm needs to maintain two state tables during the search process, called the Open List and the Close List. The Open List consists of nodes to be evaluated, and the Close List consists of nodes that have been evaluated. Each node has an associated cost function f(n), which is defined as:

[0097] f(n)=g(n)+h(n)

[0098] Among them, f(n) is the estimated cost from the initial state via state n to the target state, g(n) is the actual cost (path length) from the initial state to state n, and h(n) is the estimated cost of the best path from state n to the target state (usually Euclidean distance or Manhattan distance). The algorithm flow chart is as follows Figure 3 shown.

[0099] Following are the steps of A* algorithm:

[0100] ① Initialize two sets: Open List and Closed List. The Open List is used to store nodes to be evaluated, and initially only contains the starting node; the Closed List is used to store nodes that have been evaluated, and initially is empty.

[0101] ② Set the g value of the starting node to 0, because the cost from it to itself is 0, and store the g value, h value, and f value in the open list. The heuristic function h can be selected according to the specific problem, such as Manhattan distance, Euclidean distance, etc.

[0102] ③When the open list is not empty, repeat the following steps:

[0103] a. Select the node with the smallest f value in the open list, called the current node (CurrentNode).

[0104] b. Move the current node from the open list to the closed list.

[0105] c. For each neighbor node of the current node, perform the following operations:

[0106] i. If the neighbor node is already in the closed list, it is ignored.

[0107] ii. Calculate the cost from the current node to the neighbor node and update the g value and f value of the neighbor node.

[0108] iii. If the neighbor node is not in the open list, add it to the open list.

[0109] iv. If the g value of the neighbor node is already the minimum value, or the path from the current node to the neighbor node is not the optimal path, it is ignored.

[0110] ④If the current node is the target node (GoalNode), the path is found; otherwise, continue to step 3.

[0111] ⑤ Path backtracking: Starting from the target node, trace back along the parent node chain to the starting node to obtain the final path.

[0112] (3) Establish the kinematic model of the robot, calculate the rotation amount of the robot joints, and ensure that the robot can reach the precise position of the target object. This invention takes the use of a 6-degree-of-freedom robot for intelligent assembly tasks as an example, and selects the DH (Denavit-Hartenberg parameters method) parameter method to establish the robot kinematic model. The modeling steps are as follows:

[0113] ①Establish DH coordinate system

[0114] Four parameters are used to describe the kinematic relationship between adjacent links. When the four parameters are determined, the position relationship between adjacent links is uniquely determined. The robot arm has 6 degrees of freedom, so 7 coordinate systems need to be established. Starting from the base of the robot, select a local coordinate system for each joint and link. For each joint, define the joint variable θ i, represents the angle of the joint. For each link, determine the link length d along the z-axis of the previous coordinate system i , the offset a along the z-axis of the current coordinate system i , the rotation angle α between the x-axis of the current coordinate system and the x-axis of the previous coordinate system i , so each joint link will have [θ,a i ,d i ,α i ]Four parameters.

[0115] ② Establish a forward kinematics model

[0116] When the HD parameter list of the robot is known, the unique correspondence between the joint angle in the joint space and the end operation space can be obtained. The position relationship of any adjacent link can be expressed by a homogeneous matrix:

[0117] The inheritance of coordinate system i+1 to i can be decomposed into a rotation θ around zi i , rotate around xi i , use Rot(Z,θ i ) indicates. Get the coordinate system F i+1 to F i Rotation matrix:

[0118]

[0119] The position of coordinate system i+1 in coordinate system i:

[0120]

[0121] Establish the adjacent connecting rod transfer matrix:

[0122]

[0123] Establish the forward kinematics model of the robot:

[0124] fkine(θ)= 0 T n = 0 T1 1 T2... n-1 T n

[0125] (4) Construct a deep convolutional neural network to calculate the location information of the accessory, such as the grasping point, angle, and width.

[0126] ① Use the public Cornell University grasping dataset to train the grasping point detection network. The dataset contains 10 groups of objects, 100 objects in each group, and a total of 1,000 objects.

[0127] ② Select SE-ResUNet as the grasping detection network for training. The grasping posture of the fixture can be expressed as:

[0128] g r = {P,θ r ,q,ω}

[0129] Where P = {x, y, z} is the center of the fixture, θ i is the angle of the gripper around the Z axis, ω is the opening width of the gripper, and q is a measure of the grip quality, with a value of [0,1]. When the value of a position is close to 1, θ i Represents the gripping angle of the robot gripper at position i, and the angle value is

[0130] ③ Create a grabbing data set for industrial assembly parts. For common tools and parts in the assembly process, create a commonly used assembly data set, which contains grabbing information such as grabbing points, angles, and widths of grabbing tools and parts.

[0131] ④Use the model trained on the public dataset and use the transfer learning method to fine-tune the network on the industrial assembly parts grasping dataset produced in this project, so that the network can be applied to the grasping point detection task of the intelligent assembly robot. The purpose of this step is to calculate the grasping points of the parts to be assembled, and calculate the 5 grasping points that the network believes have the best grasping quality for each object.

[0132] (5) Check the gripping point and gripping force.

[0133] First, the 6D pose information and semantic information calculated in step (1) are used to locate the position of the parts to be assembled.

[0134] The robot moves to the accessory position. According to the planned path calculated in step (2), the robot moves to the accessory position to calculate the friction coefficient. Assume that the glove sensor data includes the pressure data P of each area. i , where i = 1, 2, ..., 550. Assume that the mass of the object is m and the normal pressure it is subjected to is N. Then the friction coefficient can be calculated by the following steps:

[0135] ① Calculate the positive pressure N: Add the pressure data of each area of ​​the hand to get the total positive pressure N:

[0136]

[0137] ② Calculate the friction force F: According to Newton's second law, the friction force is equal to the friction coefficient μ multiplied by the normal pressure N.

[0138] F=μ·N

[0139] ③Estimate the friction coefficient μ: The friction coefficient μ can be estimated by dividing the friction force F by the normal pressure N.

[0140]

[0141] Calculate the gripping force. F grip is the gripping force of each gripper, m is the mass of the gripped object (in kilograms), g is the acceleration due to gravity (usually 9.81 m / s2), and μ is the coefficient of friction between the object surface and the gripper.

[0142]

[0143] (6) The robot autonomously learns the assembly relationship of spare parts. The robot records the demonstrator’s assembly process through a depth camera, and uses the semantic information in step (1) to locate tools and spare parts information, and imitates the assembly process.

[0144] Based on the posture information and semantic information in step (1), the robot detects the assembly tool (such as a screwdriver) used by the demonstrator, helps decide which tool to use for the assembly task, and calculates the posture state of the tool.

[0145] Based on the posture information and semantic information in step (1), the robot detects the posture information and semantic information of the parts and accessories, which helps the robot learn the assembly relationship and assembly method of the parts and accessories.

[0146] (7) Using the grasping points of the assembly parts calculated in step (4), control the fixture's drop position, rotation angle, clamp width, and the calculated grasping force to control the fixture to pick up the parts. The grasping points and grasping force are not necessarily optimal. The grasping points calculated in step (4) are sorted according to the grasping quality and grasping is attempted in sequence until the grasping is successful. If grasping fails in all five attempts, return to step (4) and recalculate the grasping points.

[0147] (8) The robot arm moves to the position of the object to be assembled. According to the planned path calculated in step (2), the robot clamps the accessories and moves to the position of the parts together.

[0148] (9) Calculate the deviations between parts and accessories to autonomously plan the matching method and achieve accurate matching.

[0149] Taking the assembly of screws and nuts as an example, the process of calculating the deviation between parts and assembly parts to autonomously plan the matching method:

[0150] ① Data collection and mathematical modeling. Use the robot depth camera to measure screws and nuts and record their diameter, thread length, pitch and other dimensions;

[0151] ②Use CAD software to create three-dimensional models of screws and nuts;

[0152] ③ Determine the assembly constraints. Determine the type of fit between the screw and the nut. For example, a common thread fit requires the outer diameter of the screw to match the inner diameter of the nut.

[0153] ④ Deviation calculation:

[0154] The measured outer diameter of the screw is 5.00mm, and the actual outer diameter is 4.95mm.

[0155] The measured inner diameter of the nut is 5.00 mm, and the actual inner diameter is 5.05 mm.

[0156] Calculation deviation: The outer diameter deviation of the screw is +0.05mm, and the inner diameter deviation of the nut is -0.05mm.

[0157] ⑤ Deviation analysis. Analyze the deviation data to determine whether they are within the allowable tolerance range. In this example, the deviation is within the tolerance range, but close to the limit value.

[0158] ⑥ Matching degree evaluation. Use assembly constraints and deviation data to evaluate the matching degree between screws and nuts. Since the deviation is small, the matching degree is high, but it still needs attention.

[0159] ⑦ Assembly feasibility assessment. Evaluate the impact of deviation on assembly feasibility. For example, if the deviation causes the thread fit to be too tight or too loose, adjustment may be required.

[0160] ⑧Simulated assembly: Simulate the assembly process in a virtual environment and predict the assembly effects under different deviation conditions.

[0161] ⑨Optimize assembly parameters. Optimize assembly parameters such as tightening speed, torque, etc. according to simulation results.

[0162] ⑩ Real-time monitoring and feedback. During the actual assembly process, sensors are used to monitor the assembly status in real time and adjust the assembly actions based on the feedback.

[0163] Assembly quality inspection: After assembly is completed, quality inspection is carried out, such as using a torque wrench to check the tightening torque to ensure assembly quality.

[0164] Deviation Correction. If assembly quality issues are detected, such as thread damage or poor fit, deviation correction is performed, such as remachining or selecting parts with appropriate specifications.

[0165] Learning and adaptation: Machine learning algorithms learn deviation patterns in the assembly process and adaptively adjust assembly strategies.

[0166] Continuous improvement. Based on assembly results and quality inspection data, continuously improve assembly technology and quality control processes.

[0167] Reporting and Recording. Generate assembly deviation analysis reports, record all relevant data and adjustment measures, and provide reference for future quality control and process optimization.

[0168] S105: The robot performs assembly tasks to verify the learning effect;

[0169] Specifically, training is first conducted in a simulation environment to reduce possible errors and risks in actual assembly.

[0170] The robot performed an assembly task in a real environment for the first time as a preliminary attempt.

[0171] As the robot performs its tasks, its performance, including accuracy, speed, and stability, is closely monitored.

[0172] Collect data when the robot performs tasks, such as joint angles, speed, execution time, assembly quality, etc.

[0173] If the robot does not perform as expected, diagnose the problem, such as programming errors, sensor errors, or insufficient learning algorithms.

[0174] Adjust the robot’s learning algorithm based on the evaluation results and perform iterative optimization.

[0175] S106: The robot applies the learned skills to actual production and realizes autonomous planning of the assembly process.

[0176] Specifically, the robot applies the learned skills to actual production and realizes autonomous planning of the assembly process. For new tasks or environments, the robot can quickly learn and adapt, significantly improving the learning speed and accuracy of complex assembly skills and reducing dependence on manpower. Enhance the adaptability and autonomy of robots in unstructured environments, and improve the success rate and efficiency of assembly tasks. Through intelligent planning and autonomous adjustment, effectively deal with uncertainties in the assembly process and improve the intelligence level of the overall assembly system. Promote the development of intelligent assembly technology and provide key technical support for the transformation and upgrading of the manufacturing industry.

[0177] The second embodiment of the present application is:

[0178] Based on the first embodiment, please refer to Figure 1 ,in, Figure 1 The assembly robot system based on multimodal information fusion and apprentice learning of this embodiment includes a color video frame encoder, a depth video frame encoder, a video data fusion module, a force feedback data encoder and a multimodal feature fusion module.

[0179] According to this specific implementation, the color video frame encoder and the depth video frame encoder are respectively connected to the video data fusion module; the multimodal feature fusion module is respectively connected to the force feedback data encoder and the video data fusion module.

[0180] For tactile data, color video data and depth video data, corresponding encoders are designed to extract features from them. Then, the video data fusion module is used to fuse the video frame features. Then, the multimodal feature fusion module is used to fuse the pressure features and video features, and finally the fused feature vector of the input data is obtained.

[0181] An assembly robot system based on multimodal information fusion and apprentice learning using this embodiment, the framework structure diagram of this part is as follows Figure 1 As shown. This embodiment uses Video Vision Transformer to extract features from color video frame data and depth video frame data, then directly splices the features of the two modalities, uses one-dimensional CNN to extract features from force feedback data, and finally uses a multimodal fusion module to fuse video features and force feedback features. The calculation process can be expressed as:

[0182] F fusion =f(cat(ViViT1(V),ViViT2(D)),CNN(P))

[0183] Among them, F fusion It represents the feature vector after the fusion of video features and force feedback features, cat represents the tensor splicing operation, ViViT represents the use of Video Vision Transformer to calculate the attention score of the data, CNN represents the use of one-dimensional convolution to extract features from the data, and f represents the feature fusion function.

[0184] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. Based on multimodal information fusion and apprenticeship learning methods, the sensor network of the robot equipment has been deployed during production, the demonstrator is familiar with the skills required for the assembly task, and the deep learning network has been pre-trained and deployed before the assembly robot leaves the factory. It is characterized by: The following steps are involved: Clarify the assembly objectives and specific requirements of the teaching task, and analyze the required assembly skills; The demonstrator demonstrates the assembly process through body movements, and the sensor network records the demonstrator's movement information; The multimodal data obtained by the sensor network are fused through the multimodal information fusion algorithm; The robot makes autonomous decisions and plans routes for complex assembly; The robot performs assembly tasks to verify the learning effect; The robot applies the skills it has learned to actual production and can autonomously plan the assembly process; The multimodal data acquired by the sensor network is fused by a multimodal information fusion algorithm, and the step further includes: Collect force feedback data, color video data and depth video data during the demonstrator's assembly demonstration; De-noising and normalizing the collected data, wherein the force feedback data is de-noised using a wavelet transform method, and the color video data and the depth video data are de-noised using a Gaussian filter method. After de-noising, normalization is performed to scale the data to a uniform range; The processed color video data and depth video data are respectively subjected to feature extraction, and then the features of the two modalities are directly spliced, and feature extraction is performed on the force feedback data. Finally, the video features and force feedback features are fused using a multimodal fusion module. The robot makes autonomous decisions and plans the route for complex assembly, and the steps also include: Accurately estimate the spatial position and direction of the target object through the 6D pose estimation algorithm; Perform robot motion path planning; Establish the robot's kinematic model and calculate the rotation of the robot's joints to ensure that the robot can reach the precise position of the target object; Build a deep convolutional neural network to calculate the grasping point, angle and width position information of the accessories; Check the gripping point and gripping strength; The robot arm moves to the position of the object to be assembled; The robot autonomously learns the assembly relationship of parts and components; Calculate the deviation between parts and accessories to autonomously plan the matching method to achieve accurate matching; The spatial position and direction of the target object are accurately estimated by a 6D pose estimation algorithm, and the steps further include: Use the scale-invariant feature transformation algorithm to extract features from each frame in the video; Perform feature matching between consecutive video frames to track the movement of objects, and then use the feature matching results to estimate the movement of objects between video frames; Build a semantic segmentation model; Construct a pose estimation network to calculate the position parameters, rotation parameters, and confidence parameters of each object except the background.

2. The method based on multimodal information fusion and apprenticeship learning as claimed in claim 1, characterized in that: The assembly objectives and specific requirements of the teaching task are clarified, and the required assembly skills are analyzed. The steps also include: Determine the design specifications, materials, dimensions and functional requirements of the products or components to be assembled; Clarify the shapes, sizes and relationships of all parts that need to be assembled; Determine the order in which the parts should be assembled; Determine the optimal speed of the assembly process based on production efficiency requirements.

3. The method based on multimodal information fusion and apprenticeship learning as claimed in claim 1, characterized in that: The demonstrator demonstrates the assembly process through body movements, and the sensor network records the demonstrator's movement information. The steps further include: The demonstrator wears a capture tool that captures force feedback data on the demonstrator's hands, and uses a depth camera to record the complex assembly process demonstrated by the demonstrator through body movements.

4. The method based on multimodal information fusion and apprenticeship learning as claimed in claim 1, characterized in that: The robot performs the assembly task to verify the learning effect, and the steps further include: Training in a simulated environment to reduce errors and risks that may occur in actual assembly; The robot performed the assembly task for the first time in a real environment as a preliminary attempt; Closely monitor the robot’s performance, including accuracy, speed, and stability, as it performs its tasks; Collect data while the robot performs tasks; If the robot does not perform as expected, diagnose the problem; Adjust the robot’s learning algorithm based on the evaluation results and perform iterative optimization.

5. An assembly robot system based on multimodal information fusion and apprentice learning, applicable to the method based on multimodal information fusion and apprentice learning as claimed in any one of claims 1 to 4, characterized in that: It includes a color video frame encoder, a depth video frame encoder, a video data fusion module, a force feedback data encoder and a multimodal feature fusion module; The color video frame encoder and the depth video frame encoder are respectively connected to the video data fusion module; the multimodal feature fusion module is respectively connected to the force feedback data encoder and the video data fusion module; The multimodal data acquired by the sensor network is fused by a multimodal information fusion algorithm, and the step further includes: Collect force feedback data, color video data and depth video data during the demonstrator's assembly demonstration; De-noising and normalizing the collected data, wherein the force feedback data is de-noised using a wavelet transform method, and the color video data and the depth video data are de-noised using a Gaussian filter method. After de-noising, normalization is performed to scale the data to a uniform range; The processed color video data and depth video data are respectively subjected to feature extraction, and then the features of the two modalities are directly spliced, and feature extraction is performed on the force feedback data. Finally, the video features and force feedback features are fused using a multimodal fusion module. The robot makes autonomous decisions and plans the route for complex assembly, and the steps also include: Accurately estimate the spatial position and direction of the target object through the 6D pose estimation algorithm; Perform robot motion path planning; Establish the robot's kinematic model and calculate the rotation of the robot's joints to ensure that the robot can reach the precise position of the target object; Build a deep convolutional neural network to calculate the grasping point, angle and width position information of the accessories; Check the gripping point and gripping strength; The robot arm moves to the position of the object to be assembled; The robot autonomously learns the assembly relationship of parts and components; Calculate the deviation between parts and accessories to autonomously plan the matching method to achieve accurate matching; The spatial position and direction of the target object are accurately estimated by a 6D pose estimation algorithm, and the steps further include: Use the scale-invariant feature transformation algorithm to extract features from each frame in the video; Perform feature matching between consecutive video frames to track the movement of objects, and then use the feature matching results to estimate the movement of objects between video frames; Build a semantic segmentation model; Construct a pose estimation network to calculate the position parameters, rotation parameters, and confidence parameters of each object except the background.

Images