A complex equipment human-machine collaboration assembly guiding and robot accompanying operation method based on multi-modal intention understanding

Abstract

The application faces the discrete manufacturing assembly scene, and proposes a complex equipment human-machine collaborative assembly guidance and robot accompanying operation method based on multi-modal intention understanding, solves the problems of weak robot cognition, large interactive intention understanding deviation and difficult inheritance of expert implicit process and the like. When the method is constructed, a space-time heterogeneous graph neural network multi-modal perception model is constructed, the operator's attention is captured by fusing vision, voice and eye movement trajectory; an embodied intelligent large model is introduced to combine CoT to perform post-prediction of action intention, and safety gating is realized by prediction entropy; variable guide-nad compliance control based on intention driving is used to adaptively adjust the stiffness and damping according to the intention confidence and the human-machine distance, so that the accompanying operation of 'fast distance and soft near distance' is realized; the implicit knowledge of experienced workers is quantitatively extracted through dynamic time warping, and the model is fed back. The method promotes the cooperation from passive response to active accompanying, and improves the assembly efficiency, safety and first-time yield.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method for guiding human-machine collaborative assembly and robot-accompanied operation of complex equipment based on multimodal intent understanding.

[0002] This invention belongs to the field of interdisciplinary technology of intelligent manufacturing and next-generation artificial intelligence, and in particular relates to human-machine collaboration and embodied intelligence technology in the context of Industry 4.0.

[0003] Specifically, this invention relates to an intelligent auxiliary system applied to the final assembly stage of discrete manufacturing. This system integrates the following key technologies: Multimodal perception and intent understanding technology: This involves the acquisition, alignment, and semantic understanding of multi-source heterogeneous data such as first-person perspective video, natural language speech, and eye-tracking trajectories from industrial sites.

[0004] Vertical domain large model technology: This involves using a large language model as a cognitive center to perform structured parsing of complex assembly process documents, and combining real-time perception data for thought reasoning and action prediction.

[0005] Virtual-real fusion interaction technology: This involves dynamic information overlay and spatial registration technology based on augmented reality.

[0006] Robot adaptive control technology: This involves flexible motion planning and safe human-robot interaction control for intention-driven collaborative robots (Cobot).

[0007] This invention is particularly applicable to assembly scenarios with high non-standardization, complex processes, and high dependence on human skills, such as precision instruments and large-scale energy equipment. It aims to solve technical problems in existing technologies, such as weak robot cognitive ability, large deviation in understanding human-computer interaction intentions, and difficulty in digitally inheriting implicit assembly knowledge. Background Technology

[0008] In the fields of discrete high-end equipment manufacturing, such as industrial manufacturing and high-end medical devices, product assembly processes are characterized by "multiple varieties, small batches," complex structures, and numerous non-standardized operations. Unlike highly automated automotive welding lines, these final assembly scenarios still heavily rely on manual operation and flexible decision-making by highly skilled workers. Although electronic work instructions (E-SOPs), collaborative robots (Cobots), and basic machine vision technologies have been introduced into industrial settings, the following key technological bottlenecks still urgently need to be addressed in practical applications: First, there is a disconnect between the "static presentation" of assembly information and the "dynamic operation". Traditional work instructions (SOPs) are mostly in the form of PDF documents, two-dimensional drawings or pre-recorded videos on tablets or workstation monitors.

[0009] Excessive cognitive load: When workers perform complex operations (such as laying engine pipes), they need to frequently switch their gaze and attention between "the work at hand" and "screen information." This not only interrupts flow and reduces assembly efficiency, but also makes it easy for them to make mistakes or omissions due to memory bias.

[0010] Lack of context awareness: Existing AR-assisted technologies are mostly static information "rigid superpositions" (such as displaying text in a fixed position), which cannot intelligently push the key torque value or installation techniques needed "at this moment" based on the worker's current gaze focus or hand movements, resulting in information redundancy or missing information.

[0011] Second, the "passive response" mode of collaborative robots limits the efficiency of human-robot collaboration. Although current collaborative robots (such as UR and Franka) have torque sensors and collision detection functions, which can ensure the safety of human-robot coexistence, they are still essentially based on pre-programmed trajectories or simple teaching and reproduction for movement.

[0012] Lack of intent understanding: Robots cannot proactively anticipate worker needs by observing the operator's eye contact, gestures, or voice commands (such as "hand me that wrench"), unlike human assistants.

[0013] Lack of interactive means: To control the robot, workers usually need to put down their tools and operate the teach pendant or physical buttons. This "interruption of assembly in order to use the robot" mode makes the robot a burden rather than an assistant.

[0014] Third, there is the loss of implicit assembly process knowledge and its difficulty in digitization. In the assembly of complex equipment, a large number of core process know-how (such as the tactile feedback during the assembly of precision bearings and the force control during wire harness combing) are mastered by experienced "master craftsmen" and belong to implicit knowledge that is difficult to describe in words.

[0015] Data silos and insufficient understanding: Existing industrial AI models are mostly small, specialized models designed for specific tasks (such as defect detection), lacking multimodal generalization capabilities. They cannot simultaneously process first-person video streams, audio narration, and force data, thus failing to effectively capture, understand, and pass on these implicit experiences.

[0016] Lack of adaptive error correction capability: When a new employee's operation deviates from the standard process but has not yet caused consequences, the existing system lacks real-time early warning capability based on "process behavior analysis," and can often only conduct post-event traceability after a quality incident occurs. In summary, how to build an intelligent system with embodied cognition capabilities, enabling it to understand the multimodal intentions of workers like a "skilled apprentice," and proactively provide AR information enhancement and robot companion services, is a key challenge in realizing the transition from "human-machine coexistence" to "human-machine integration" in the assembly of complex equipment. Summary of the Invention

[0017] The purpose of this invention is to provide a method for guiding human-machine collaborative assembly and robot-accompanied operation of complex equipment based on multimodal intent understanding, so as to solve the above-mentioned problems existing in the prior art.

[0018] A method for human-machine collaborative assembly guidance and robot-accompanied operation of complex equipment based on multimodal intent understanding, characterized by the following steps: Multi-source perception steps: Simultaneously collect operator eye-tracking data, first-person perspective video data, voice command data, and robot end effector force data in the assembly scenario; Intent prediction steps: A dynamic scene graph is constructed using a spatiotemporal heterogeneous graph neural network and input into the embodied intelligence large model. The posterior probability distribution of the operator's specific assembly intention at the next moment is calculated through thought chain reasoning. Safety decision-making steps: Calculate the information entropy of the posterior probability distribution as an uncertainty index. When the index is lower than the safety threshold, generate a definite action instruction. Two-way execution steps: Based on the action instructions, on the one hand, virtual assembly guidance is superimposed on the augmented reality device, and on the other hand, the collaborative robot is driven to perform accompanying tool delivery or auxiliary support actions.

[0019] 2. The method according to claim 1, characterized in that, in the bidirectional execution step, the motion control driving the collaborative robot adopts an intention-driven variable admittance control strategy: A second-order admittance dynamic model incorporating virtual stiffness and virtual damping is established, and the virtual stiffness is set as a function of "intent confidence" and "human-machine relative distance"; During the process of the robot delivering tools to the operator, the virtual stiffness value is reduced in real time as the relative distance between the robot and the operator decreases. This allows the robot to exhibit compliant zero-gravity characteristics when in contact with the operator, while maintaining high rigidity when moving away from the operator to achieve rapid positioning.

[0020] 3. The method according to claim 1, characterized in that it further includes tacit knowledge extraction and model evolution steps: The difference distance between the actual assembly trajectory of an experienced operator and the standard process trajectory is calculated using the Dynamic Time Warping (DTW) algorithm. When the difference distance exceeds a threshold and the assembly efficiency improves, the difference feature is marked as implicit empirical knowledge and encoded as a fine-tuning sample to update the parameters of the embodied intelligent large model.

[0021] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects: This invention achieves a leap forward compared to existing technologies in terms of perception accuracy, decision-making intelligence, control compliance, and knowledge evolution capability by constructing a deep mathematical model and closed-loop control system. Attached Figure Description

[0022] Figure 1 It is a diagram of the system's logical architecture and data flow.

[0023] Figure 2 It is a schematic diagram of the core algorithm process and the evolution of implicit knowledge. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings.

[0025] Example This invention addresses the technical bottlenecks in existing discrete manufacturing assembly fields, such as insufficient depth of understanding of human-machine collaborative intentions, poor dynamic response capabilities of robots in unstructured environments, and the difficulty in digitally extracting and reusing implicit expert process knowledge. It provides a method for guiding human-machine collaborative assembly and robot-accompanied operation of complex equipment based on spatiotemporal multimodal mapping and embodied intelligence models. The core technical solution of this invention, by constructing a deep mapping between the physical and digital worlds, utilizes a rigorous mathematical model to perform end-to-end perception, decision-making, control, and evolution of the human-machine collaboration process, specifically including the following four levels of technical innovation: I. Constructing a Multimodal Perception Model Based on Spatiotemporal Heterogeneous Graph Neural Networks To address the issue of poor robustness of single-modal data in complex industrial environments, this invention proposes a method for mapping visual, auditory, tactile, and eye-tracking data into dynamic maps.

[0026] 1. Graph Construction: The system models the assembly scene as a dynamic heterogeneous graph Gt=Vt,ℰt.

[0027] The node set Vt contains three types of nodes: worker nodes vhuman (containing pose vectors), object nodes vobj (containing tool / part categories and 6D poses), and robot nodes vrobot (containing end-effector states).

[0028] The edge set ℰt represents the interaction relationship between nodes (such as "hand-tool" contact, "line of sight-object" gaze).

[0029] 2. Multimodal Feature Fusion Algorithm: For the worker node vhuman, this invention introduces a multi-head cross-attention mechanism, which weights and fuses the first-view visual features FV and the eye-tracking gaze features FG to accurately capture the worker's focus of attention. The formula for calculating the fused feature vector Hfused is as follows: Hfused=Concathead1,…,headkWO (1) headi=AttentionQi,Ki,Vi=SoftmaxQiKiTdkVi (2) The query vector Qi = FGWiQ (derived from eye-tracking data, representing "where you want to look"), the key vector Ki = FVWiK (derived from the full visual image, representing "what's in the environment"), and the value vector Vi = FVWiV (derived from visual features) are all learnable weight matrices, and dk is a scaling factor. The beneficial effect of this formula is that by using eye-tracking data as the query vector, the system can automatically suppress feature noise from irrelevant objects in the background (such as cluttered parts), increasing the algorithm's focus weight on the worker's current operating object by more than 300%.

[0030] II. Intent Probability Prediction and Bayesian Filtering Based on Large Model Thinking Chain (CoT) This invention does not directly use an end-to-end black-box model, but instead constructs a probabilistic reasoning engine that combines prior knowledge with real-time perception.

[0031] 1. Mathematical Model for Intent Prediction: Utilizing a vertically fine-tuned large language model as the probability estimator, the posterior probability distribution of the worker's future action intention It+1 is calculated at time step t, given the historical observation sequence X1:t and standard process knowledge K. PIt+1∣X1:t,K=SoftmaxℱLLMEmbX1:t⊕EmbK (1) Here, ⊕ represents the vector concatenation operation, and ℱLLM is the Transformer decoding layer.

[0032] 2. Uncertainty-Based Safety Gating Mechanism: To prevent large models from generating "illusions" that lead to erroneous robot actions, this invention introduces prediction entropy as an uncertainty metric, Ht. Ht=−k=1NPIt+1=klogPIt+1=k (2) The system sets a dynamic threshold δ. When Ht > δ, the model is considered to be in a "hesitant state," the system automatically suspends the robot's actions, and initiates a voice confirmation to the worker via AR glasses ("Please confirm if you need a No. 10 socket?"). Robot control commands are only issued when Ht ≤ δ. The beneficial effect of this mechanism is that, according to actual tests, this gating mechanism reduces the robot's false actuation rate from 5.2% to below 0.1%, ensuring absolute safety in human-robot collaboration.

[0033] III. Intention-Driven Variable Admittance Compliant Control Law To address the contradiction of traditional robots being "either too rigid (unsafe) or too soft (slow positioning)," this invention proposes a parameter-adaptive admittance control method.

[0034] 1. Dynamic Admittance Model: The motion of the robot's end effector in Cartesian space follows the following second-order dynamic equations: Mde + Bdte + Kdte = Fext (1) Where e = X − Xd is the position error, and Fext is the human-computer interaction force.

[0035] 2. Adaptation Law for Stiffness and Damping: The innovation of this invention lies in the fact that the virtual stiffness Kdt and virtual damping Bdt are not constants, but functions of the intention confidence Pconf and the human-machine distance Dhr. Kdt=Kmin+Kmax−Kmin⋅1−e−αDhr⋅Pconf (2) Bdt=2ξMdKdt (3) in: When the robot is far from the worker (Dhr > 0) and its intention is clear (Pconf > 0), the stiffness Kd increases, enabling fast and precise movement. When the robot enters the worker's work area (Dhr ≠ 0) for handover, the exponential term causes Kd to rapidly decay to Kmin, achieving "zero-gravity" compliant interaction. The beneficial effects of this formula are: achieving biomimetic motion characteristics of "fast at long distances and soft at close distances," reducing the average time for tool delivery by 40%, and limiting the maximum impact force in the event of an accidental collision to within 10N.

[0036] IV. Quantitative Extraction of Tacit Knowledge and DTW Evaluation Algorithm For the ineffable feel and skills of "veteran workers", the present invention proposes a method for extracting tacit knowledge based on trajectory similarity measurement.

[0037] 1. Differential modeling of expert trajectories: Let the operation trajectory specified by the standard SOP be Tstd, and the actual operation trajectory of the senior worker be Texp. The system uses the Dynamic Time Warping (DTW) algorithm to calculate the optimal matching path and the difference distance between the two: DistDTWTstd,Texp=minπi,j∈πTstdi−Texpj2 (1) 2. Gain evaluation and knowledge solidification: If DistDTW > ϵ (significant difference) and the process completion time Timeexp < Timestd (efficiency improvement), the system determines that this difference is "positive tacit experience". The system encodes this difference feature as a new Prompt-Tuning vector Δθ and updates the large model parameters: θnew←θold−η∇θℒalignΔθ (2) The beneficial effect of this method is that the system can automatically capture minute actions such as "reverse alignment action before screwing in the bolt" and "specific inclination angle during gluing", shortening the training cycle for novice employees to reach the level of skilled workers by 75% through AR guidance.

[0038] V. Experimental verification and analysis of beneficial effects To verify the effectiveness of the method described in the present invention in a real industrial scenario, the applicant built a verification platform on the general assembly line of an aeroengine fuel pump. The experiment used two Franka Emika 7-degree-of-freedom collaborative robots, the operator wore HoloLens 2, and the edge computing unit used NVIDIA Orin AGX.

[0039] The experiment selected 50 assembly workers with different proficiency levels (20 experts and 30 novices) for a 30-day comparative test. The test was divided into three groups: Control group A: Traditional paper SOP + manual handling tools.

[0040] Control group B: Traditional collaborative robot (fixed program trigger) + tablet electronic SOP.

[0041] Experimental group C (the present invention): Embodied intelligent large model + AR guidance + variable admittance robot accompaniment.

[0042] Key Metrics: Physical Meaning and Calculation Method; Control Group A; Control Group B; Experimental Group C; (This Invention) Year-on-Year Improvement / Optimization Average Intent Recognition Accuracy (mAP) | Model Predicted Action vs. Actual Action | N / A | 65.4% (Rule Only) | 96.8% + 31.4% Average cycle time per operation: Total time to complete a standard assembly sequence: 145 seconds vs. 132 seconds vs. 98 seconds, representing a 32.4% efficiency improvement. Non-value-added time ratio (NVA Ratio): The proportion of time spent searching for tools and consulting documents is 22% - 18% <3%, significantly eliminating waste. Human-computer interaction smoothness (Jerk Metric): The rate of change of robot end-effector acceleration dx / dt N / A 15.4 m / s³ 2.1 m / s³ Smoothness improved by 7 times. First-time pass rate (FTQ) for novice assemblers: 82.5% - 88.0% - 99.5% - near-zero defects. The number of tacit knowledge captured, successfully extracted and solidified, and effective process fine-tuning actions: 0.042, realizing knowledge assetization. 1. Efficiency Improvement Mechanism: The significant efficiency improvement in experimental group C was mainly due to the robot's "accompanying service." Data shows that when the worker's hand leaves the workpiece from the previous process, the robot, based on probability prediction (PIt+1), begins moving the tools needed for the next process 1.5 seconds in advance. This seamless "zero-wait" connection is impossible to achieve with traditional technologies.

[0043] 2. Quality Control Mechanism: The novice pass rate jumped from 82.5% to 99.5%, directly attributable to the "virtual-real alignment guidance" in the AR glasses. Multimodal perception based on ST-GNN ensures that the virtual arrows are accurately anchored to the physical bolts (error <1mm), effectively preventing omissions and mis-installations.

[0044] 3. Safety verification: In the 12 accidental human-machine collision events recorded during the test, the robot based on the variable admittance control law of this invention reduced the contact stiffness Kd to zero within 10ms, without causing any personnel injury or workpiece damage, which proves the reliability of the algorithm under extreme conditions.

[0045] In summary, by constructing a deep mathematical model and a closed-loop control system, this invention has achieved a leapfrog progress over existing technologies in terms of perception accuracy, decision-making intelligence, control compliance, and knowledge evolution capability.

Claims

1. A method for guiding and accompanying robot operations in complex equipment collaborative assembly based on multimodal intent understanding, characterized in that, Includes the following steps: Multi-source perception steps: Simultaneously collect operator eye-tracking data, first-person perspective video data, voice command data, and robot end effector force data in the assembly scenario; Intent prediction step: Construct a dynamic scene graph using a spatiotemporal heterogeneous graph neural network and input it into the embodied intelligence large model. Calculate the posterior probability distribution of the operator's specific assembly intention at the next moment through thought chain reasoning. Safety decision-making steps: Calculate the information entropy of the posterior probability distribution as an uncertainty index. When the index is lower than the safety threshold, generate a definite action instruction. Two-way execution steps: Based on the action instructions, on the one hand, virtual assembly guidance is superimposed on the augmented reality device, and on the other hand, the collaborative robot is driven to perform accompanying tool delivery or auxiliary support actions.

2. The method according to claim 1, characterized in that, In the bidirectional execution step, the motion control driving the collaborative robot adopts an intention-driven variable admittance control strategy: A second-order admittance dynamic model incorporating virtual stiffness and virtual damping is established, and the virtual stiffness is set as a function of "intent confidence" and "human-machine relative distance"; During the process of the robot delivering tools to the operator, the virtual stiffness value is reduced in real time as the relative distance between the robot and the operator decreases. This allows the robot to exhibit compliant zero-gravity characteristics when in contact with the operator, while maintaining high rigidity when moving away from the operator to achieve rapid positioning.

3. The method according to claim 1, characterized in that, It also includes tacit knowledge extraction and model evolution steps: The difference distance between the actual assembly trajectory of an experienced operator and the standard process trajectory is calculated using the Dynamic Time Warping (DTW) algorithm. When the difference distance exceeds a threshold and the assembly efficiency improves, the difference feature is marked as implicit empirical knowledge and encoded as a fine-tuning sample to update the parameters of the embodied intelligent large model.

Images