An aviation manufacturing long-term action execution method based on large model task planning

Abstract

The application discloses an aviation manufacturing long-time action execution method based on a large model task planning, taking a large language model as the core, constructing a VLA model of "task planning+multimodal dynamic alignment+configurable component adaptation layer", first analyzing multimodal requirements of multi-component assembly by the large language model and splitting standardized general and exclusive subtasks, then fusing visual, language and action multimodal data to realize real-time perception of assembly state, relying on reinforcement learning to optimize scheduling strategy and importing component exclusive constraints and knowledge base through the configurable adaptation layer, finally collecting execution data through the closed-loop feedback link, linking multimodal information to analyze and process assembly abnormalities, and synchronously adjusting subsequent scheduling plans. The application effectively improves the adaptation flexibility, scheduling robustness and assembly reliability of multi-component assembly in aviation manufacturing, reduces the cost of manual intervention and the loss of downtime, and provides technical support for stable deployment of intelligent equipment in complex assembly scenarios of aviation manufacturing.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to the optimization technology of the Vision-Language-Action Architecture (VLA), which combines multimodal embodied intelligence and reinforcement learning, in long-duration tasks in the aerospace manufacturing field. In particular, it relates to a long-duration action execution method for aerospace manufacturing based on large-model task planning. Background Technology

[0002] The combination of Vision-Language-Motion Architecture (VLA) and reinforcement learning has been researched and applied to some extent in the field of robot task planning and execution. Existing related patents include autonomous robot multi-task operation planning methods driven by embodied cognitive large models, embodied intelligent task planner training methods based on multimodal large models, and embodied intelligent task planning methods based on environmental perception. These existing methods mainly achieve robot task planning and execution through environmental perception, cross-modal fusion, motion trajectory generation, and simulated interaction, achieving good results in short-term tasks.

[0003] Currently, there are several patents related to VLA models and reinforcement learning, including: Tongji University's "A Method and System for Multi-Task Operation Planning of Autonomous Robots Driven by an Embodied Cognitive Large Model" (application number: CN202510687989.6); Shenzhen Ruoyu Technology Co., Ltd.'s "A Training Method and System for Embodied Intelligent Task Planner Based on a Multimodal Large Model" (application number: CN202410250472.6); and South China University of Technology and the Super Robotics Research Institute (Huangpu)'s "An Embodied Intelligent Task Planning Method, Device, Equipment and Medium Based on Environmental Perception" (application number: CN202510260591.4).

[0004] The method disclosed in application number 202510687989.6 is as follows: S1, real-time acquisition of the first RGB image and the first depth map when the robot performs multi-task operations, and encoding based on the RGB image and the depth map to obtain an embodied visual representation; S2, acquisition of the natural language instructions for the multi-task operations, cross-modal fusion of the natural language instructions and the embodied visual representation to obtain fusion features, and multi-task decomposition based on the fusion features to obtain a multi-task decomposition scheme; the multi-task decomposition scheme includes the execution order and priority of a single task; S3, based on the multi-task decomposition scheme, using diffusion... The strategy generates a continuous motion trajectory for the robot's end effector; S4, acquire the second RGB image and the second depth map after the robot executes the continuous motion trajectory, and compare the first RGB image and the first depth map with the second RGB image and the second depth map respectively. If they are the same, jump to S5; otherwise, generate a prompt message based on the differences and return to S2 to update the multi-task decomposition scheme; S5, determine whether the continuous motion trajectory has been successfully executed. If successful, determine whether the multi-task has been completed. If not completed, return to S1; if the execution is unsuccessful, generate a feedback packet and return to S2 to guide the update of the multi-task decomposition strategy and S4 as an attachment condition.

[0005] The method disclosed in application number 202410250472.6 is as follows: S1, reformatting multiple embodied domain original datasets, with the dataset format being image-text-robot motion trajectory pairs. S2, outputting concise embodied planning data to unify the granularity of the embodied planning data. S3, formalizing the embodied task planning problem. S4, searching on a behavior tree for the process of repeatedly using available information for reasoning.

[0006] The method disclosed in application number 202510260591.4 is as follows: S1. By integrating abstract environmental information into a large language model, the large language model is fine-tuned to enhance its environmental perception capability. S2. The task objective and environmental state table are input into the fine-tuned large language model, and a multi-step action sequence with the highest probability is generated through a stepwise bundle search strategy; the environmental state table is obtained through environmental perception. S3. Based on the generated action sequence, the environmental data state table is updated by simulating the interaction between the robot agent and other objects, thereby generating an interaction trajectory corresponding to the steps. S4. For each task, the environmental state table in the trajectory simulation is compared with the expected interaction trajectory, and the trajectory matching score for that task is calculated to evaluate the large language model's ability in embodied task planning.

[0007] However, existing technologies for long-duration tasks are still in their early stages and have significant shortcomings: First, the models have poor generalization ability and cannot adapt to multiple scenarios or tasks; second, after the model decomposes long-duration tasks into sub-tasks, the transition and dependence between sub-tasks can lead to dynamic coupling and error propagation problems. Specifically, in the complex scenario of multi-component assembly in aerospace manufacturing, these shortcomings directly manifest as poor assembly adaptability of multiple components, rigid process scheduling, and delayed anomaly handling, making it difficult to meet the long-duration, high-precision, and multi-scenario assembly requirements of aerospace manufacturing. Summary of the Invention

[0008] To address the shortcomings of existing technologies in the multi-component assembly scenario of aerospace manufacturing, such as poor adaptability, rigid process scheduling, and lagging anomaly handling, this invention provides a long-term action execution method for aerospace manufacturing based on large model task planning. This method improves the adaptability, scheduling robustness, and assembly reliability of the VLA model in long-term multi-component assembly tasks in aerospace manufacturing, while reducing the cost of manual intervention and downtime losses.

[0009] This invention proposes a VLA model method driven by a large language model, consisting of "task planning + multimodal dynamic alignment + configurable component adaptation layer." It employs a full-process architecture of "requirements analysis - subtask decomposition - dynamic scheduling - closed-loop feedback." By integrating core artificial intelligence technologies such as multimodal learning, large language models, and computer vision, the method enables robots to understand and execute complex multimodal input tasks in aerospace manufacturing component assembly, efficiently completing closed-loop management of the entire process from component arrival inspection, modular assembly, system testing to delivery traceability. Specifically, the method includes the following core steps: S1. Multimodal Requirements Analysis: Receives multimodal information input from the assembly of multiple components in aerospace manufacturing, accurately extracts the core elements of the assembly using the semantic understanding capabilities of a large language model, and outputs standardized requirements analysis results; the multimodal information input includes text-based device model requirements and device delivery indicators, as well as visual component drawings; the core elements of the assembly include the precision requirements for the assembly between different components. S2. Standardized Subtask Decomposition: Based on the requirements analysis results of step S1, the large language model is used to automatically decompose and generate a standardized subtask sequence, clarifying the execution order, dependencies and resource requirements of each subtask; the subtasks include general subtasks and component-specific subtasks. General subtasks include parts arrival verification and core module assembly, while component-specific subtasks include engine dynamic balance testing and wing skin fit detection. S3. Multimodal Dynamic Perception and Dynamic Scheduling: Real-time perception of assembly status is achieved by integrating multimodal data information from vision, language, and motion. The scheduling strategy of sub-tasks is dynamically optimized by relying on the feedback iteration mechanism of reinforcement learning. At the same time, the dedicated constraints and knowledge base information of target parts are imported on demand through the adaptation layer of configurable components to adapt to the assembly requirements of different types of parts. The visual data includes assembly process images and accuracy data collected by industrial cameras and laser rangefinders; the language data includes assembly instructions and process documents; the motion data includes robot assembly posture and equipment operating status; the dedicated constraints include accuracy thresholds and process logic; and the knowledge base includes assembly manuals and fault cases. S4. Closed-loop feedback and anomaly handling: Real-time collection of execution results of each subtask and multimodal status data to form a closed-loop feedback link. If an assembly anomaly occurs, multimodal data analysis is used to locate the cause of the anomaly and generate a targeted solution. At the same time, the scheduling plan of subsequent subtasks is adjusted. The assembly anomaly scenarios include substandard assembly accuracy, equipment failure and component quality problems. The targeted solutions include rework processes and component replacement plans.

[0010] This invention proposes a VLA model method driven by a large language model, which includes "task planning + multimodal dynamic alignment + configurable component adaptation layer", comprising the following stages: 1. Multimodal Requirements Analysis Phase: This phase uses a Large Language Model (LLM) as its core, receiving multimodal information input from the assembly of multiple components in aerospace manufacturing. This multimodal information input includes text-based requirements for component models and delivery specifications, as well as visual information such as component drawings. Utilizing the semantic understanding capabilities of the LLM, the core assembly elements are accurately extracted. These core elements include the precision requirements for assembling different components, and standardized requirements analysis results are output, providing a precise basis for subsequent subtask decomposition.

[0011] 2. Standardized Subtask Decomposition Phase: This phase is based on the standardized requirements analysis results obtained in the multimodal requirements analysis phase. Subtasks are automatically decomposed and generated through a large language model. Subtasks are divided into two categories: general subtasks and component-specific subtasks. General subtasks include common aerospace manufacturing processes such as parts arrival verification and core module assembly. Component-specific subtasks include unique assembly and testing processes for different components, such as engine dynamic balance testing and wing skin fit detection.

[0012] Meanwhile, the model standardizes the generated subtasks to form a standardized sequence of subtasks, clarifying the execution order, dependencies, and resource requirements of each subtask, ensuring that there are no logical conflicts between subtasks, and reducing the problems of dynamic coupling and error propagation between subtasks from the source.

[0013] 3. Multimodal Dynamic Perception and Dynamic Scheduling Stage: This stage achieves real-time perception of the status of the aerospace manufacturing assembly process by integrating multimodal data information from vision, language, and motion. Among them, visual data is collected through industrial cameras and laser rangefinders, including assembly process images and precision data; language data includes final assembly instructions and process documents; motion data includes robot assembly posture and equipment operating status.

[0014] By leveraging the feedback and iteration mechanism of reinforcement learning, the scheduling strategy of sub-tasks is dynamically optimized based on the real-time perceived assembly status, enabling flexible adjustment of process scheduling and solving the problem of rigid traditional process scheduling. Simultaneously, through a configurable component adaptation layer, the exclusive constraints and knowledge base information of target parts are imported as needed. The exclusive constraints include accuracy thresholds, process logic, etc., and the knowledge base includes assembly manuals, fault cases, etc., further adapting to the assembly requirements of different types of parts and improving the adaptability of multi-part assembly.

[0015] 4. Closed-Loop Feedback and Anomaly Handling Phase: This phase establishes a closed-loop feedback chain throughout the entire process by collecting the execution results and multimodal status data of each subtask in real time. If abnormal scenarios such as substandard assembly accuracy, equipment failure, or component quality issues occur during the assembly process, the model immediately links with the collected multimodal data for comprehensive analysis, quickly pinpointing the cause of the anomaly and generating targeted solutions, including rework procedures and component replacement plans.

[0016] Meanwhile, the model adjusts the scheduling plan of subsequent sub-tasks in sync with the anomaly handling results, ensuring the smooth progress of the multi-component assembly process in aerospace manufacturing, solving the problem of lagging anomaly handling in existing technologies, and improving the continuity and reliability of the assembly process.

[0017] The long-term action execution method for aerospace manufacturing based on large-model task planning of the present invention has the following advantages compared with the prior art: Improve multi-component compatibility: By importing exclusive constraints and knowledge bases of different components on demand through a configurable component adaptation layer, it is compatible with the final assembly scenarios of six types of core aerospace manufacturing components such as engines, wings, and landing gear, which greatly improves the success rate of multi-component adaptation and solves the problem of poor final assembly compatibility of existing technologies. Enhance scheduling robustness: Integrate multimodal data to achieve real-time perception of assembly status, and rely on reinforcement learning feedback iteration mechanism to dynamically optimize scheduling strategy, realize flexible scheduling of processes, reduce response delay of process scheduling, and solve the problem of rigid process scheduling. Improve anomaly handling efficiency: Build a closed-loop feedback chain throughout the entire process, link multimodal data to quickly locate the cause and generate targeted solutions in abnormal scenarios, and adjust subsequent scheduling plans in a synchronized manner to improve anomaly handling efficiency and assembly accuracy pass rate, and solve the problem of delayed anomaly handling; Reduce manufacturing costs: Reduce the need for manual intervention, reduce downtime losses caused by rigid processes and untimely handling of abnormalities, while improving resource utilization and airworthiness compliance rate, and provide technical support for the large-scale implementation of intelligent assembly in aviation manufacturing. Enhance model generalization ability: By standardizing subtask decomposition and multimodal dynamic alignment, the model can be adapted to long-term assembly tasks of various scenarios and types in aerospace manufacturing, effectively solving the problem of poor generalization of existing VLA models. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall architecture of the method of the present invention, showing the full-process architecture of "requirement analysis - subtask decomposition - dynamic scheduling - closed-loop feedback"; Figure 2 This is a schematic diagram of the specific process of the method of the present invention, which shows in detail the entire process of data interaction and execution logic from multimodal input to exception handling and subsequent scheduling adjustment. Detailed Implementation

[0019] The following section provides a more detailed explanation of the long-term action execution method for aerospace manufacturing based on large-scale model task planning, using specific experiments and implementation processes.

[0020] The technical solution proposed in this invention is a long-term action execution method driven by a large language model, consisting of "task planning + multimodal dynamic alignment + configurable component adaptation layer". It adopts a full-process architecture of "requirements analysis - subtask decomposition - dynamic scheduling - closed-loop feedback", such as... Figure 1 As shown.

[0021] Specifically, the entire method is as follows Figure 2 As shown, it includes the following core steps: 1. Multimodal Requirements Analysis Phase. This phase is based on Large Language Model (LLM). By receiving multimodal information input from the assembly of multiple components in aerospace manufacturing (including text-based requirements for component models and delivery indicators, as well as visual component drawings, etc.), the semantic understanding capabilities of LLM are used to accurately extract the core elements of the assembly (such as the precision requirements for the assembly between different components), and standardized requirements analysis results are output.

[0022] 2. Standardized Subtask Decomposition Phase. This phase, based on the requirements analysis results obtained in the first phase, automatically decomposes and generates subtasks using LLM. Each subtask consists of two parts: 1) General sub-tasks: such as component arrival verification, core module assembly 2) Component-specific sub-tasks: such as engine dynamic balancing tests and wing skin fit inspections. During this stage, the model standardizes the subtask sequence, clarifies the execution order, dependencies, and resource requirements of each subtask, and ensures that there are no logical conflicts in the subtasks.

[0023] 3. Multimodal Dynamic Perception and Dynamic Scheduling Stage. This stage achieves real-time perception of the assembly status by fusing multimodal data information from vision (assembly process images and accuracy data acquired through industrial cameras and laser rangefinders), language (assembly instructions and process documents), and motion (robot assembly posture and equipment operating status). In addition, the model relies on the feedback iteration mechanism of reinforcement learning to dynamically optimize the scheduling strategy of sub-tasks. At the same time, through a configurable component adaptation layer, it imports the specific constraints (such as accuracy thresholds and process logic) and knowledge base information (such as assembly manuals and fault cases) of the target parts as needed to further adapt to the assembly requirements of different types of parts and achieve more accurate assembly.

[0024] 4. Closed-Loop Feedback and Anomaly Handling Phase. This phase establishes a closed-loop feedback chain by collecting the execution results and multimodal status data of each subtask in real time. If abnormal scenarios such as substandard assembly accuracy, equipment failure, or component quality issues occur, the collected multimodal data is analyzed to quickly pinpoint the cause of the anomaly and generate targeted solutions (such as rework processes or component replacement plans). Simultaneously, the model will also adjust the scheduling plan of subsequent subtasks to ensure the smooth progress of the final assembly process.

[0025] The following section provides a more detailed explanation of the long-term action execution method for aerospace manufacturing based on large-scale model task planning, using specific experiments and implementation processes.

[0026] Experimental data and hardware environment: The method of this invention uses assembly data of multiple components in aerospace manufacturing, covering the complete assembly process data of six core components such as engines, wings, and landing gear; the hardware used in the experiment is 2×NVIDIA RTX 4090 24GB GPUs, and the deep learning framework used is PyTorch.

[0027] Model training settings: Data partitioning: The assembly task cases are arranged in ascending order. The training set and test set are partitioned using a 5-fold cross-validation method to ensure that the task cases in the training set and test set do not overlap during training, and that the task cases in the test set do not overlap in each of the 5 training sessions, thereby improving the effectiveness and generalization of model training. Training parameters: The total number of training epochs is set to 600, the batch size is 1024, the initial learning rate is set to 0.0001, and the learning rate dynamically decreases by 5% every 50 epochs; the loss function is joint weighted loss, the optimizer is AdamW, and the dropout is set to 0.2. Reasonable parameter settings improve the convergence and stability of model training.

[0028] Model performance evaluation metrics: The model performance is comprehensively evaluated on the test set by calculating six indicators: task completion success rate, process scheduling response delay, anomaly handling efficiency, assembly accuracy pass rate, resource utilization rate, and airworthiness compliance rate. Each indicator directly points to the actual production needs of multi-component assembly in aerospace manufacturing, ensuring that the model performance matches the industrial application scenario.

[0029] After training and testing under the above experimental settings, the method of this invention demonstrates excellent performance in the multi-component assembly scenario of aerospace manufacturing: compared with the existing VLA model, the success rate of multi-component adaptation is significantly improved, the process scheduling response delay is greatly reduced, the efficiency of anomaly handling and the assembly accuracy qualification rate are effectively improved, and the resource utilization rate and airworthiness compliance rate meet the industrial standards of aerospace manufacturing. It can provide strong technical support for the stable deployment of intelligent equipment such as robotic arms in complex aerospace manufacturing assembly scenarios.

[0030] The long-term action execution method for aerospace manufacturing based on large-scale model task planning of the present invention is mainly applied to long-term multi-component assembly tasks in the aerospace manufacturing field, including the assembly process of core components such as engine assembly, wing assembly, and landing gear assembly. It can also be extended to similar high-precision, multi-component, long-term intelligent manufacturing scenarios such as aerospace and high-end equipment manufacturing.

[0031] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for executing long-term actions in aerospace manufacturing based on large-scale model task planning, characterized in that, A vision-language-action model is constructed around a large language model, consisting of "task planning + multimodal dynamic alignment + configurable component adaptation layer". It adopts a full-process architecture of "requirements analysis - subtask decomposition - dynamic scheduling - closed-loop feedback", specifically including the following steps: S1. Multimodal Requirements Analysis: Receives multimodal information input from the assembly of multiple components in aerospace manufacturing, accurately extracts the core elements of the assembly using the semantic understanding capabilities of a large language model, and outputs standardized requirements analysis results; the multimodal information input includes text-based device model requirements and device delivery indicators, as well as visual component drawings; the core elements of the assembly include the precision requirements for the assembly between different components. S2. Standardized Subtask Decomposition: Based on the requirements analysis results of step S1, the large language model is used to automatically decompose and generate a standardized subtask sequence, clarifying the execution order, dependencies and resource requirements of each subtask; the subtasks include general subtasks and component-specific subtasks. General subtasks include parts arrival verification and core module assembly, while component-specific subtasks include engine dynamic balance testing and wing skin fit detection. S3. Multimodal Dynamic Perception and Dynamic Scheduling: Real-time perception of assembly status is achieved by integrating multimodal data information from vision, language, and motion. The scheduling strategy of sub-tasks is dynamically optimized by relying on the feedback iteration mechanism of reinforcement learning. At the same time, the dedicated constraints and knowledge base information of target parts are imported on demand through the adaptation layer of configurable components to adapt to the assembly requirements of different types of parts. The visual data includes assembly process images and accuracy data collected by industrial cameras and laser rangefinders; the language data includes assembly instructions and process documents; the motion data includes robot assembly posture and equipment operating status; the dedicated constraints include accuracy thresholds and process logic; and the knowledge base includes assembly manuals and fault cases. S4. Closed-loop feedback and anomaly handling: Real-time collection of execution results of each subtask and multimodal status data to form a closed-loop feedback link. If an assembly anomaly occurs, multimodal data analysis is used to locate the cause of the anomaly and generate a targeted solution. At the same time, the scheduling plan of subsequent subtasks is adjusted. The assembly anomaly scenarios include substandard assembly accuracy, equipment failure and component quality problems. The targeted solutions include rework processes and component replacement plans.

2. The method for executing long-term aerospace manufacturing actions based on large-scale model task planning according to claim 1, characterized in that, The subtask sequence generated in step S2 must meet the requirement of logical non-conflict to ensure that there is no dynamic coupling problem in the transition and dependency relationship between subtasks.

3. The method for executing long-duration aerospace manufacturing actions based on large-scale model task planning according to claim 1, characterized in that, In step S3, the fusion perception of multimodal data is a real-time perception, and the feedback iteration mechanism of reinforcement learning dynamically updates the scheduling strategy according to the assembly status, so as to realize the flexible adjustment of process scheduling.

4. The method for executing long-term aerospace manufacturing actions based on large-scale model task planning according to claim 1, characterized in that, In step S4, the closed-loop feedback link is a real-time feedback throughout the entire process, and the analysis and solution generation for anomaly handling are linked processes to ensure the timeliness and pertinence of anomaly handling.

5. The method for executing long-term aerospace manufacturing actions based on large-scale model task planning according to claim 1, characterized in that, The method is applicable to long-duration multi-component assembly missions in the aviation manufacturing field, involving core components such as engines, wings, and landing gear.

6. The method for executing long-duration aerospace manufacturing actions based on large-scale model task planning according to claim 1, characterized in that, The proposed method employs 5-fold cross-validation during model training and testing to ensure that the task cases in the training and testing sets do not overlap. During training, a dynamically decaying learning rate is set, and a joint weighted loss function and the AdamW optimizer are used.

Images