Aircraft part precision assembly method based on multi-modal fusion perception and action generation

Abstract

The application discloses an aviation part precision assembly method based on multi-modal fusion perception and action generation, takes a visual-language-action model (VLA) as a core, first collects two-dimensional images, three-dimensional point clouds, force / tactile signal multi-modal information and natural language task instructions of aviation part assembly, then respectively encodes the same to generate visual tokens, state tokens and text tokens, jointly models the multi-modal tokens through the VLA model to obtain a fusion representation, generates a continuous assembly action sequence containing grabbing, moving and aligning based on the representation, finally decodes the action sequence into control instructions to drive an executing mechanism to complete precision assembly. The application realizes end-to-end automatic control from environment perception, task understanding to action generation, improves the automation level, execution precision and task adaptability of aviation assembly, reduces the cost of manual demonstration and debugging, and is suitable for aviation part high-precision automatic assembly scenes.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing and industrial robot technology, specifically involving a precision assembly method for aerospace parts based on multimodal fusion perception and motion generation. It is applicable to high-precision operations such as automated gripping, alignment, and assembly of aerospace parts, and is especially suitable for aerospace industry application scenarios with high requirements for assembly accuracy, stability, and reliability. Background Technology

[0002] The aerospace manufacturing industry has stringent requirements for the precision, stability, and reliability of parts assembly. Existing aerospace parts assembly technologies are mostly based on traditional visual recognition and rule control methods. Meanwhile, some patents have proposed methods for assembling precision parts or multimodal detection, such as the precision mechanical parts assembly method of Shanghai Aerospace Research Institute, the intelligent detection method for parts by multimodal self-encoders that integrate 3D point clouds of Huaqiao University, and the assembly robot system and method based on multimodal information fusion and apprentice learning of the Guangdong-Hong Kong-Macao Greater Bay Area Innovation Research Institute of Hunan University.

[0003] Existing patents related to the precision assembly method of aerospace parts based on multimodal fusion perception and motion generation include: "An assembly method for precision mechanical parts" (application number 201910487038.9) applied for by Shanghai Aerospace Research Institute; "Intelligent detection method and device for parts with multimodal autoencoder fusion of 3D point cloud" (application number 202510315537.5) applied for by Huaqiao University; and "Assembly robot system and method based on multimodal information fusion and apprentice learning" (application number 202410906502.4) applied for by Hunan University Guangdong-Hong Kong-Macao Greater Bay Area Innovation Research Institute (Zengcheng, Guangzhou).

[0004] The method disclosed in application number 201910487038.9 is as follows: S1. Pre-treat the inner surface of the first precision mechanical component and the outer surface of the second precision mechanical component by cleaning with anhydrous ethanol and air-drying naturally to improve the cleanliness of the assembly surfaces. S2. Uniformly coat the outer surface of the second precision mechanical component with adhesive, controlling the thickness and uniformity of the adhesive application through needle injection. S3. Assemble the second precision mechanical component with adhesive into the first precision mechanical component, monitoring the assembly torque in real time during the assembly process, and stopping the assembly when the torque reaches a preset threshold. S4. Control the rotation speed of the second precision mechanical component according to its dimensional parameters, causing it to rotate around its axis during the assembly process to ensure uniform distribution of the adhesive. S5. After assembly, test the sealing and bonding strength of the assembled components, and verify the assembly quality by heat curing.

[0005] The method disclosed in application number 202510315537.5 is as follows: S1. Cleaning and drying the inner surface of the first precision mechanical component and the outer surface of the second precision mechanical component to improve the cleanliness and bonding performance of the assembly surfaces. S2. Applying adhesive evenly to the outer surface of the second precision mechanical component using a needle injection method, controlling the dispensing parameters to ensure the thickness and uniformity of the adhesive. S3. Assembling the adhesive-coated second precision mechanical component into the first precision mechanical component, and using a torque sensor to monitor the assembly status in real time during the assembly process. When the torque reaches a preset threshold, the assembly is considered complete. S4. Controlling the rotation speed of the component according to its dimensions during the assembly process to ensure that the adhesive is evenly distributed within the bonding interface, avoiding insufficient or excessive adhesive. S5. After assembly, curing the assembly and testing its sealing and bonding strength to verify the assembly quality.

[0006] The method disclosed in application number 202410906502.4 is as follows: S1. Collect multi-source sensing data of the parts to be inspected, including two-dimensional image data, three-dimensional point cloud data, and related geometric information, and preprocess the collected data to eliminate noise and unify coordinates. S2. Perform voxelization and feature encoding on the preprocessed three-dimensional point cloud data to map irregular point clouds into structured feature representations for subsequent feature learning. S3. Input the two-dimensional image features and three-dimensional point cloud features into a multimodal autoencoder network, and obtain the multimodal depth feature representation of the parts through joint feature learning. S4. Use the trained autoencoder model to reconstruct and compare the features of the parts, and extract potential abnormal features or defect information. S5. Output the detection results of the parts based on the feature reconstruction error or similarity evaluation results, and realize the automatic identification of defects or abnormal states of the parts.

[0007] However, existing technologies still have many shortcomings: First, they lack an assembly decision-making mechanism based on vision-language-action joint modeling, making it difficult to unify the representation and collaborative reasoning of assembly task semantics and visual perception information. They also lack end-to-end learning capabilities to support continuous assembly operations such as grasping, moving, and alignment, making it difficult to adapt to complex assembly processes. Second, they mostly rely on two-dimensional images and shallow features for part recognition, which is poorly adaptable to changes in lighting, partial occlusion, and random placement of parts. When parts have similar appearances, similar sizes, or overlap, the recognition stability and assembly reliability are insufficient. Third, they rely on preset rules or fixed action processes to complete assembly tasks, lacking the ability to learn and generate complex assembly action sequences. They are unable to dynamically adjust assembly strategies according to task semantics and environmental changes, resulting in limited system flexibility and generalization capabilities, and failing to meet the high-precision assembly requirements of aerospace manufacturing scenarios.

[0008] To address the aforementioned issues, this invention constructs a vision-language-action (VLA) joint modeling framework, which integrates multi-source perception information and introduces a learnable action generation mechanism to achieve end-to-end automated control of precision assembly of aerospace parts, effectively solving the shortcomings of existing technologies. Summary of the Invention

[0009] The purpose of this invention is to provide a precision assembly method for aerospace parts based on multimodal fusion perception and motion generation, which solves the problems of poor adaptability, lack of end-to-end learning capability, and insufficient flexibility of existing aerospace assembly technologies, and improves the automation level, execution accuracy and task adaptability of aerospace assembly operations.

[0010] This invention proposes a precision assembly method for aerospace parts based on multimodal fusion perception and motion generation. Using a vision-language-action (VLA) model as its core, it constructs a joint modeling framework integrating vision, task semantics, and assembly actions. This framework enables unified perception and representation of the visual, spatial, and task semantic information of aerospace parts, and generates a continuous sequence of assembly actions end-to-end, achieving overall learning and automatic execution of complex assembly processes. The method specifically includes the following steps: S1. Multimodal assembly information acquisition: Collect multimodal perception information of the aerospace parts assembly environment and natural language task instructions corresponding to the assembly task. The multimodal perception information includes two-dimensional image information, three-dimensional point cloud information and force / tactile signals during the assembly process of the aerospace parts. S2. Multimodal token generation: The multimodal perception information and natural language task instructions obtained in step S1 are respectively encoded to generate visual tokens, state tokens, and text tokens; among them, the two-dimensional image information and three-dimensional point cloud information are encoded to obtain visual tokens that represent the appearance, spatial structure, and relative pose of the part; the natural language task instructions are encoded to obtain text tokens that reflect the assembly target, object, and operation constraints; and the force / tactile signals are encoded to obtain state tokens that describe the contact state, force changes, and assembly stability of the part. S3. Multimodal token fusion modeling: Visual tokens, text tokens, and state tokens are combined into a unified token sequence according to predetermined rules and input into the VLA backbone model. The relationship between different modal tokens is modeled through cross-modal attention mechanism and sequence modeling mechanism to generate a multimodal fusion representation that matches the current assembly task. S4. Assembly action sequence generation: Based on multimodal fusion representation, the action generation module of VLA model predicts and outputs action tokens. Multiple action tokens are arranged in time order to form a continuous assembly action sequence. The assembly action sequence includes at least grasping action, spatial movement action, and assembly alignment action. S5. Assembly Execution Control: Decode the action token sequence into specific assembly control instructions, control the assembly execution mechanism to execute the assembly action sequence in sequence, and complete the precision assembly operation of aerospace parts.

[0011] Compared with the prior art, the technical solution of the present invention has the following advantages: It has achieved end-to-end automated control from environmental perception and task understanding to assembly action generation, and constructed a vision-language-action joint modeling framework. It unifies the semantics of assembly tasks with visual and force-tactile perception information for representation and collaborative reasoning. It has end-to-end learning capability for continuous assembly operations and can adapt to the complex assembly process of aerospace manufacturing. By integrating multi-source sensing information such as two-dimensional images, three-dimensional point clouds, and force / tactile signals, it breaks through the limitations of traditional shallow feature recognition and greatly improves the adaptability to changes in lighting, partial occlusion, and random placement of parts. Even in scenarios where parts have similar appearances and sizes, it can still ensure recognition stability and assembly reliability. By introducing a learnable action generation mechanism, the assembly strategy can be dynamically adjusted according to the task semantics and environmental changes, generating an adaptive sequence of complex assembly actions, which improves the system's flexibility and generalization ability, and meets the precision assembly requirements of aerospace manufacturing for high precision and high reliability. It eliminates the need for manual teaching and fixed assembly rules, reducing the cost of manual teaching and system debugging. It has high engineering practical value and can be widely applied in precision assembly scenarios in the aerospace industry. Attached Figure Description

[0012] Figure 1 This is a technical roadmap of the present invention; Figure 2 This is a diagram illustrating the multimodal fusion process of the present invention; The connection relationship between the modules in the attached figure illustrates the complete technical process of the present invention from multimodal information acquisition to assembly execution. The multimodal assembly information is processed by each coding module to achieve feature fusion, and then the assembly is completed through the action generation and execution control module. Detailed Implementation

[0013] The present invention will be further described in detail below with reference to specific embodiments. These embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0014] like Figure 1 , Figure 2 As shown, the precision assembly method for aerospace parts based on multimodal fusion perception and motion generation proposed in this invention includes the following steps: (1) Acquisition of multimodal assembly information During the assembly of aerospace parts, multimodal information of the assembly environment is collected. This multimodal information includes two-dimensional image information, three-dimensional point cloud information, and force / tactile signals collected during assembly. Natural language task instructions corresponding to the assembly task are also obtained. Two-dimensional image information is used to characterize the appearance and overall position of the parts; three-dimensional point cloud information is used to describe the spatial structure and geometric relationships of the parts; force / tactile signals are used to reflect the assembly contact state and force changes; and task instructions are used to describe the assembly objectives and operational requirements. This multimodal information serves as the raw input to the VLA model, used to subsequently generate corresponding visual tokens, state tokens, and text tokens, providing a complete input foundation for VLA-based joint modeling.

[0015] (2) Visual and spatial feature encoding (visual token generation) The acquired 2D image information and 3D point cloud information are input into the visual encoding submodule of the VLA model. Visual features are extracted from the images, and spatial geometric features are modeled from the point clouds. The original visual and spatial perception information is mapped into a set of visual-spatial feature tokens (visual tokens). These visual tokens are used to characterize the appearance features, spatial structure, and relative pose relationships of aerospace parts, and serve as one of the inputs for unified sequence modeling in the VLA model.

[0016] (3) Task semantic feature encoding (text token generation) The natural language instructions corresponding to the assembly task are input into the language encoding submodule of the VLA model. The task text is segmented and converted into discrete text tokens, which are then encoded by the language model to generate a semantic feature representation of the task. The text tokens are used to reflect the assembly target, assembly object, and operational constraints, guiding the VLA model to understand the "assembly task to be completed" during the action generation process.

[0017] (4) Force / tactile information encoding (state token generation) The collected force / tactile signals are input into the VLA model as state information during the assembly process. The force / tactile signals are numerically encoded and feature-mapped to generate corresponding state tokens. These state tokens are used to describe the contact state of the parts, force changes, and assembly stability, and provide physical constraints and state feedback information for motion generation during the precision alignment and assembly stages.

[0018] (5) Multimodal token fusion modeling based on VLA The visual tokens, text tokens, and state tokens are combined into a unified token sequence according to time sequence or predetermined rules, and then input into the VLA backbone model for joint modeling. The VLA model models the correlation between different modal tokens through cross-modal attention and sequence modeling mechanisms, generating a multimodal fusion representation that is highly relevant to the current assembly task, thereby achieving a unified understanding of the assembly scene state and task intent.

[0019] (6) Assembly action sequence generation (action token output) Based on the multimodal fusion representation, the action generation module in the VLA model is used to progressively predict and output the action tokens corresponding to the assembly actions. The action tokens represent the motion instructions of the assembly actuator at each time step. Multiple action tokens are arranged in chronological order to form a continuous sequence of assembly actions. The action sequence includes at least grasping actions, spatial movement actions, and assembly alignment actions.

[0020] (7) Assembly execution control Based on the generated action token sequence, it is decoded into specific assembly control instructions, which control the assembly actuator to sequentially complete the gripping, moving and precision assembly operations of aerospace parts, thereby realizing the automatic execution of aerospace part assembly tasks.

[0021] The present invention relates to a precision assembly method for aerospace parts based on multimodal fusion perception and motion generation, which is implemented based on a precision assembly system for aerospace parts. The hardware and software configuration of the system is as follows: Computing platform: Composed of 4 NVIDIA RTX 4090 (24GB VRAM) GPUs, meeting the computing power requirements for multimodal data processing and VLA model calculation; Assembly actuator: The Franka robotic arm is used, which has high-precision motion control and force sensing capabilities, and is adapted to the operation requirements of precision assembly of aerospace parts. Sensing devices: Equipped with a visual acquisition device, a three-dimensional sensing device, and a force / tactile sensing device, which are used to acquire two-dimensional image information, three-dimensional point cloud information, and force / tactile signals, respectively; Software framework: A multimodal data processing framework is built based on PyTorch, and the assembly task is modeled and verified using the Isaac Sim simulation environment; Core Model: The OpenVLA vision-language-action model is used to achieve joint processing of multimodal perception information and task instructions.

[0022] The specific implementation steps for the precision assembly of aerospace parts in this embodiment are as follows: Multimodal information and task instruction acquisition: During the assembly of aerospace parts, two-dimensional image information is obtained by capturing images of aerospace parts through a vision acquisition device, and three-dimensional point cloud information is obtained by scanning parts through a three-dimensional perception device. Force / tactile sensing devices collect force / tactile signals such as contact status and force changes during the assembly process in real time. At the same time, the operator or the upper system inputs the natural language task instructions corresponding to the assembly task, such as "grab aerospace shaft parts and complete coaxial alignment assembly with sleeve parts".

[0023] Multimodal token encoding generation: Input 2D images and 3D point cloud information into the visual encoding submodule of OpenVLA to extract visual features and spatial geometric features and map them into visual tokens; input natural language task instructions into the language encoding submodule, and encode them into text tokens after word segmentation and discretization; input force / tactile signals into the model for numerical encoding and feature mapping to generate state tokens.

[0024] Multimodal token fusion: Visual tokens, text tokens, and state tokens are combined into a unified token sequence in chronological order and input into the OpenVLA backbone model. The model uses a cross-modal attention mechanism to mine the correlation between different modal tokens, completes joint modeling, and generates a multimodal fusion representation to accurately understand the assembly task intent and the current assembly scene state.

[0025] Assembly action sequence generation: The OpenVLA action generation module predicts and outputs action tokens step by step based on multimodal fusion representation, and forms a continuous assembly action sequence in chronological order. In this embodiment, the sequence includes: the movement action of the robotic arm moving to the gripping position of the shaft part, the gripping action of the gripper closing to grip the part, the movement action of the robotic arm carrying the part to the assembly position of the sleeve part, the alignment action of the shaft part and the sleeve part being aligned coaxially, and the assembly action of pushing the part in to complete the assembly.

[0026] Assembly execution and control: The action token sequence is decoded into specific control instructions for the Franka robotic arm. The robotic arm executes the above action sequence in sequence according to the instructions. During the assembly process, the force / tactile sensing device collects signals in real time and generates status tokens to feed back to the model. The model can dynamically fine-tune the actions according to the feedback to ensure assembly accuracy and finally complete the precision assembly operation of aerospace parts.

[0027] In this embodiment, the assembly system does not rely on fixed assembly rules and manual instruction procedures. Even when parts are randomly placed and there is slight obstruction, it can still stably complete high-precision assembly operations, verifying the effectiveness and engineering feasibility of the method of the present invention.

[0028] The above description is only a preferred embodiment of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for precision assembly of aerospace parts based on multimodal fusion perception and motion generation, characterized in that, Using a vision-language-motion model as the core, end-to-end precision assembly of aerospace parts is achieved, including the following steps: S1. Multimodal assembly information acquisition: Collect multimodal perception information of the aerospace parts assembly environment and natural language task instructions corresponding to the assembly task. The multimodal perception information includes two-dimensional image information, three-dimensional point cloud information and force / tactile signals during the assembly process of the aerospace parts. S2. Multimodal token generation: The multimodal perception information and natural language task instructions obtained in step S1 are respectively encoded to generate visual tokens, state tokens, and text tokens; among them, the two-dimensional image information and three-dimensional point cloud information are encoded to obtain visual tokens that represent the appearance, spatial structure, and relative pose of the part; the natural language task instructions are encoded to obtain text tokens that reflect the assembly target, object, and operation constraints; and the force / tactile signals are encoded to obtain state tokens that describe the contact state, force changes, and assembly stability of the part. S3. Multimodal token fusion modeling: Visual tokens, text tokens, and state tokens are combined into a unified token sequence according to predetermined rules and input into the VLA backbone model. The relationship between different modal tokens is modeled through cross-modal attention mechanism and sequence modeling mechanism to generate a multimodal fusion representation that matches the current assembly task. S4. Assembly action sequence generation: Based on multimodal fusion representation, the action generation module of VLA model predicts and outputs action tokens. Multiple action tokens are arranged in time order to form a continuous assembly action sequence. The assembly action sequence includes at least grasping action, spatial movement action, and assembly alignment action. S5. Assembly Execution Control: Decode the action token sequence into specific assembly control instructions, control the assembly execution mechanism to execute the assembly action sequence in sequence, and complete the precision assembly operation of aerospace parts.

2. The precision assembly method for aerospace parts based on multimodal fusion perception and motion generation according to claim 1, characterized in that, In step S1, two-dimensional image information is used to characterize the appearance and overall position of the part, three-dimensional point cloud information is used to describe the spatial structure and geometric relationship of the part, and force / tactile signals are used to reflect the assembly contact state and force changes.

3. The precision assembly method for aerospace parts based on multimodal fusion perception and motion generation according to claim 1, characterized in that, In step S2, the visual token is generated by the visual encoding submodule in the VLA model. The visual encoding submodule extracts visual features from the two-dimensional image information and performs spatial geometric feature modeling on the three-dimensional point cloud information before mapping. The text token is generated by the language encoding submodule in the VLA model. The language encoding submodule first performs word segmentation on the task text to convert it into discrete text tokens, and then encodes the task semantic feature representation through the language model. The state token is generated by numerically encoding and feature mapping the force / tactile signal through the VLA model.

4. The precision assembly method for aerospace parts based on multimodal fusion perception and motion generation according to claim 1, characterized in that, In step S3, the predetermined rule is to combine them in chronological order, and the VLA backbone model achieves a unified understanding of the assembly scenario state and task intent through joint modeling.

5. The precision assembly method for aerospace parts based on multimodal fusion perception and motion generation according to claim 1, characterized in that, In step S5, the assembly execution mechanism is a Franka robotic arm. During the execution of the assembly control commands, force / tactile signals are fed back to the VLA model in real time, providing physical constraints and state feedback for the dynamic adjustment of the assembly action.

6. The precision assembly method for aerospace parts based on multimodal fusion perception and motion generation according to any one of claims 1-5, characterized in that, This method runs on a computing platform consisting of multiple NVIDIA RTX 4090 GPUs. Multimodal data is processed in a software framework built on PyTorch, and the assembly task is modeled and verified using the Isaac sim simulation environment.

7. The precision assembly method for aerospace parts based on multimodal fusion perception and motion generation according to claim 1, characterized in that, The VLA model adopts the OpenVLA model to achieve joint processing of image information, point cloud information, force / tactile information and task instructions.

Images