A cross-task industrial robot flexible operation method and system based on a VLA multi-modal large model

Abstract

The application discloses a kind of based on VLA multimodal large model's cross-task industrial robot flexible operation method and system, solve the technical problem that existing industrial robot intelligent operation scheme cannot rely on existing model to realize quick adaptation, leading to the poor cross-task flexible adaptation capability of industrial robot.Method includes obtaining industrial original multi-source data and preprocessing to obtain structured multimodal industrial operation dataset, sample construction is carried out by parameter discretization and multi-task dialogue, and output standardized multi-task dialogue training sample set, offline light fine-tuning is carried out to general vision-language-action base model, and industrial adaptation VLA basic model is obtained.Online acquisition multimodal perception data and natural language operation instruction, after time stamp synchronization alignment, space alignment and cross-modal feature fusion are completed based on preset hand-eye homogeneous transformation matrix, end-to-end autonomous decision and operation control are executed by the model, and the final operation result is output.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent robots and multimodal large model cross-technology, and in particular to a cross-task industrial robot flexible operation method and system based on VLA multimodal large model. Background Technology

[0002] Against the backdrop of the continuous deepening of intelligent manufacturing, industrial robots have become core production equipment in fields such as electronic manufacturing, new energy, and precision machining. Their application scenarios are extending from repetitive operations at a single workstation to flexible operations involving multiple categories and processes.

[0003] At the same time, the pace of downstream product iteration is accelerating, and production lines need to frequently switch between different tasks such as measurement, assembly, testing, and handling. Furthermore, the specifications and process requirements of workpieces are dynamically adjusted with each batch, which places higher demands on the adaptation efficiency and autonomy of industrial robots.

[0004] However, existing intelligent operation solutions for industrial robots generally adopt a single-task customized development approach. For different task types such as precision measurement, component assembly, appearance inspection, and material handling, separate perception algorithm frameworks are built, dedicated functional models are trained, and corresponding control programs are written. Each task type corresponds to an independent perception, decision-making, and execution technology chain. In this mode, the model system, training data, and control logic corresponding to different tasks are isolated and not interchangeable. The task knowledge and decision-making capabilities learned by the model cannot be transferred and reused across different operation scenarios. When the production line needs to switch tasks, it cannot rely on existing models to achieve rapid adaptation. Data acquisition, model training, program writing, and on-site integration must be completed again for the new task. The overall changeover cycle is long, the deployment cost is high, and the adaptation efficiency is low, resulting in poor cross-task flexible adaptation capability of industrial robots. Summary of the Invention

[0005] This invention provides a method and system for flexible operation of industrial robots across tasks based on a VLA multimodal large model, which solves the technical problem that existing intelligent operation solutions for industrial robots cannot achieve rapid adaptation based on existing models, resulting in poor cross-task flexible adaptation capability of industrial robots.

[0006] The first aspect of this invention provides a cross-task flexible operation method for industrial robots based on a VLA multimodal large model, comprising:

[0007] Acquire raw multi-source industrial data and preprocess the raw multi-source industrial data to obtain a structured multimodal industrial operation dataset.

[0008] The structured multimodal industrial operation dataset is subjected to parameter discretization and multi-task dialogue pair sample construction processing to output a standardized multi-task dialogue pair training sample set.

[0009] The standardized multi-task dialogue is used to perform offline lightweight fine-tuning of the general vision-language-action base model on the training sample set, and output an industrial-adaptive VLA base model.

[0010] Acquire multimodal perception data and natural language task instructions input by the user, and perform time-stamp synchronization and alignment of RGB images, depth data and six-dimensional force perception data in the multimodal perception data, and output time-aligned multimodal perception data;

[0011] Based on a preset hand-eye homogeneous transformation matrix, the time-aligned multimodal perception data is spatially aligned and then combined with the natural language task instructions to perform cross-modal feature fusion, outputting cross-modal fusion features and structured task definition vectors.

[0012] The industrial-adaptive VLA basic model is used to perform end-to-end autonomous decision-making and job execution control based on the preset hand-eye homogeneous transformation matrix, the cross-modal fusion features and the structured task definition vector, and outputs the final job results.

[0013] Optionally, the preprocessing of the raw multi-source industrial data to obtain a structured multimodal industrial operation dataset includes:

[0014] The original multi-source industrial data is classified and aggregated according to task type and data dimension to obtain classified and aggregated multi-source industrial data.

[0015] The classified and aggregated industrial multi-source data are sequentially subjected to format validation and dimension alignment to obtain a structured multimodal industrial operation dataset.

[0016] Optionally, the structured multimodal industrial operation dataset includes multiple multimodal labeled samples covering various industrial operation scenarios; the step of performing parameter discretization and multi-task dialogue pair sample construction processing on the structured multimodal industrial operation dataset to output a standardized multi-task dialogue pair training sample set includes:

[0017] Semantic discrete encoding is performed on the continuous physical annotation values ​​in each multimodal annotation sample to obtain the discrete key operation parameter text corresponding to each multimodal annotation sample;

[0018] The scene image and natural language instruction text in each of the multimodal labeled samples are used as the model input data, and the hardware scheduling annotation, perception strategy annotation, execution process annotation, output type annotation and discretization key operation parameter text in the corresponding multimodal labeled samples are used as the model output data.

[0019] The model input data and model output data corresponding to each of the multimodal labeled samples are matched to obtain a standardized multi-task dialogue pair training sample set.

[0020] Optionally, the step of using the standardized multi-task dialogue to perform offline lightweight fine-tuning of the general vision-language-action base model on the training sample set, and outputting an industrially adapted VLA base model, includes:

[0021] Freeze the backbone weights of the general vision-language-action base model, insert trainable low-rank matrix pairs into the attention layer weight matrix of the general vision-language-action base model, and use the standardized multi-task dialogue pair training sample set as the training input of the general vision-language-action base model to carry out supervised training, thereby obtaining an industrial-adaptive VLA base model.

[0022] Optionally, the step of spatially aligning the time-aligned multimodal perception data based on a preset hand-eye homogeneous transformation matrix, and then performing cross-modal feature fusion in conjunction with the natural language task instructions to output cross-modal fusion features and a structured task definition vector includes:

[0023] Based on the preset hand-eye homogeneous transformation matrix, the time-aligned multimodal perception data is spatially aligned to obtain spatiotemporally unified multimodal perception data.

[0024] Semantic embedding calculations are performed on the natural language task instructions to obtain text feature vectors;

[0025] Visual encoding calculations are performed on the RGB images and depth data in the spatiotemporally unified multimodal perception data to obtain visual feature vectors;

[0026] Temporal feature encoding is performed on the six-dimensional force perception data in the spatiotemporally unified multimodal sensing data to obtain a force perception feature vector;

[0027] Multimodal cross-modal attention-weighted fusion calculation is performed on the text feature vector, the visual feature vector, and the force feature vector to obtain cross-modal fusion features;

[0028] Based on the cross-modal fusion features, a structured task definition vector is constructed.

[0029] Optionally, the industrial-adaptive VLA basic model performs end-to-end autonomous decision-making and job execution control based on the preset hand-eye homogeneous transformation matrix, the cross-modal fusion features, and the structured task definition vector, outputting the final job result, including:

[0030] The cross-modal fusion features and the structured task definition vector are input into the industrial-adaptive VLA base model to perform inference and generate a multi-dimensional structured decision vector.

[0031] The multidimensional structured decision vector is decomposed to obtain a discrete motion waypoint sequence;

[0032] The discrete motion waypoint sequence is subjected to trajectory smoothing calculation to obtain a smooth motion trajectory sequence;

[0033] Based on the smooth motion trajectory sequence and real-time multidimensional force data, compliance correction calculation and trajectory superposition are performed to output real-time motion control commands.

[0034] The industrial robot's end effector is driven by the real-time motion control commands to complete the corresponding operation. During the execution, based on the preset hand-eye homogeneous transformation matrix and real-time joint pose data, multi-sensor dynamic calibration is completed by minimizing the reprojection error to generate the final operation result.

[0035] Optionally, it also includes:

[0036] Obtain the entire process data package, extract multi-dimensional operation data from the entire process data package and encapsulate it in a structured manner according to a unified format to obtain multiple structured samples;

[0037] Calculate a comprehensive weight for multiple structured samples and attach labels to them to obtain multiple structured job samples with weighted labels;

[0038] Multiple structured job samples with weighted labels are classified, stored, and indexed in a standardized manner. Based on the sample weight, high-value samples are retained first and low-value duplicate samples are filtered out to obtain a dynamically expandable dynamic sample experience pool.

[0039] When the effective sample size of the dynamic sample experience pool reaches a preset trigger threshold, the industrial-adaptive VLA basic model is trained on the newly added samples in the dynamic sample experience pool. The training process introduces a knowledge retention regularization term to suppress catastrophic forgetting, and the incrementally updated VLA candidate model is obtained.

[0040] The incrementally updated VLA candidate models are validated to obtain a new industrial-adaptive VLA base model.

[0041] The second aspect of this invention provides a flexible cross-task industrial robot operation system based on a VLA multimodal large model, comprising:

[0042] The acquisition module is used to acquire raw multi-source industrial data and preprocess the raw multi-source industrial data to obtain a structured multimodal industrial operation dataset.

[0043] The construction module is used to perform parameter discretization and multi-task dialogue pair sample construction processing on the structured multimodal industrial operation dataset, and output a standardized multi-task dialogue pair training sample set.

[0044] The fine-tuning module is used to perform offline lightweight fine-tuning of the general vision-language-action base model using the standardized multi-task dialogue training sample set, and output an industrial-adaptive VLA base model.

[0045] The alignment module is used to acquire multimodal perception data and natural language task instructions input by the user, and to perform time stamp synchronization alignment on the RGB image, depth data and six-dimensional force perception data in the multimodal perception data, and output time-aligned multimodal perception data.

[0046] The output module is used to perform spatial alignment on the time-aligned multimodal perception data based on a preset hand-eye homogeneous transformation matrix, and then perform cross-modal feature fusion in combination with the natural language task instructions to output cross-modal fusion features and structured task definition vectors.

[0047] The task module is used to perform end-to-end autonomous decision-making and task execution control based on the industrial-adaptive VLA basic model, the preset hand-eye homogeneous transformation matrix, the cross-modal fusion features, and the structured task definition vector, and output the final task result.

[0048] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the cross-task industrial robot flexible operation method based on VLA multimodal large model as described above.

[0049] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, it implements the flexible operation method for cross-task industrial robots based on VLA multimodal large model as described above.

[0050] The fifth aspect of the present invention provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the cross-task industrial robot flexible operation method based on VLA multimodal large model as described above.

[0051] As can be seen from the above technical solutions, the present invention has the following advantages:

[0052] The present invention provides a method for flexible cross-task industrial robot operation based on a VLA multimodal large model. This method acquires raw multi-source industrial data and preprocesses it to obtain a structured multimodal industrial operation dataset. The structured multimodal industrial operation dataset is then processed by parameter discretization and multi-task dialogue pair sample construction, outputting a standardized multi-task dialogue pair training sample set. The standardized multi-task dialogue pair training sample set is used to perform offline lightweight fine-tuning of a general vision-language-action base model, outputting an industrial-adaptive VLA basic model. Multimodal perception data and user-input natural language operation instructions are acquired, and the RGB images, depth data, and six-dimensional force perception data in the multimodal perception data are time-stamped and aligned, outputting time-aligned multimodal perception data. Based on a preset hand-eye homogeneous transformation matrix, the time-aligned multimodal perception data is spatially aligned and then combined with natural language operation instructions for cross-modal feature fusion, outputting cross-modal fusion features and a structured task definition vector. Finally, the industrial-adaptive VLA basic model is used based on the preset hand-eye homogeneous transformation matrix, cross-modal fusion features, and structured task definition vector. The invention defines vectors for end-to-end autonomous decision-making and job execution control, outputting the final job results. Based on the above scheme, this invention injects the task logic, perception rules, and execution strategies of multiple types of industrial operations into the same VLA model during the offline fine-tuning stage. This allows a single model to cover the decision-making needs of different job tasks, breaking the limitations of traditional solutions where different tasks require customized models and knowledge systems are fragmented and cannot be reused. When switching tasks online, there is no need to re-collect data, retrain models, or write control programs for new tasks. Only the corresponding natural language job instructions need to be input, and the model can rely on its learned cross-task decision-making capabilities and combine spatiotemporally aligned multimodal perception data to complete scene understanding and autonomously generate job solutions, directly driving the execution end to complete the job. This achieves efficient transfer and reuse of model capabilities between different job tasks. At the same time, the standardized multimodal spatiotemporal alignment and cross-modal feature fusion process ensures the consistency of perception input and inference links under different task scenarios. This eliminates the need to reconstruct perception and decision-making links during task switching, enabling rapid response to the task change requirements of the production line and significantly improving the cross-task flexible adaptability of industrial robots. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1This is a flowchart illustrating the steps of a cross-task industrial robot flexible operation method based on a VLA multimodal large model, as provided in Embodiment 1 of the present invention.

[0055] Figure 2 This is a flowchart illustrating a cross-task industrial robot flexible operation method based on a VLA multimodal large model, as provided in Embodiment 1 of the present invention.

[0056] Figure 3 This is a functional block diagram of a cross-task industrial robot flexible operation system based on a VLA multimodal large model provided in Embodiment 1 of the present invention;

[0057] Figure 4 This is a structural block diagram of a cross-task industrial robot flexible operation system based on a VLA multimodal large model, provided in Embodiment 2 of the present invention. Detailed Implementation

[0058] This invention provides a flexible operation method and system for cross-task industrial robots based on a VLA multimodal large model. Specifically, it relates to a flexible operation method for cross-task industrial robots based on a Vision-Language-Action (VLA) large model, which can be widely applied to various types of industrial operations such as precision measurement, component assembly, appearance quality inspection, and material handling in flexible manufacturing scenarios. It solves the technical problem that existing intelligent operation solutions for industrial robots cannot achieve rapid adaptation based on existing models, resulting in poor cross-task flexible adaptation capabilities of industrial robots.

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that in the optional embodiments of the present invention, the object information and other related data involved require the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. That is to say, if the embodiments of the present invention involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of relevant departments, and in compliance with relevant laws, regulations, and standards. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0060] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a cross-task industrial robot flexible operation method based on a VLA multimodal large model, as provided in Embodiment 1 of the present invention.

[0061] This invention provides a cross-task flexible operation method for industrial robots based on a VLA multimodal large model, comprising:

[0062] Step 101: Obtain raw multi-source industrial data and preprocess the raw multi-source industrial data to obtain a structured multimodal industrial operation dataset.

[0063] It should be noted that the data collection covers a variety of industrial operation scenarios, including precision measurement, component assembly, appearance inspection, and material handling. Scene images are acquired through industrial area scan cameras deployed at workstations, while force-sensing time-series data is acquired at a fixed frequency through a six-dimensional force sensor at the end of a robotic arm. Natural language command text, hardware scheduling tags, standard execution process text, and raw values ​​of operating parameters are extracted and labeled from standard operating procedures, historical operation records, and process documents for the corresponding scenarios. Subsequently, the raw data is classified and collected according to task type and data dimension, and format verification and dimension alignment are performed sequentially. Data from each dimension of the same operation instance is matched and bound, and after standardization and regularization, a structured multimodal industrial operation dataset is obtained.

[0064] Further, step 101 may include the following sub-steps:

[0065] S11. Classify and aggregate the original multi-source industrial data according to task type and data dimension to obtain the classified and aggregated multi-source industrial data.

[0066] S12. Perform format validation and dimension alignment on the classified and aggregated industrial multi-source data in sequence to obtain a structured multimodal industrial operation dataset.

[0067] It should be noted that lightweight adaptation is applied to the VLA model during the offline training phase. Addressing the issues of insufficient industrial decision-making capabilities and high computational costs associated with full-scale fine-tuning of general multimodal large models, this invention obtains a basic model adapted to industrial scenarios through industrial dataset construction, parameter discretization preprocessing, and low-rank adaptive fine-tuning.

[0068] In this step, the present invention constructs a multimodal industrial operation dataset, covering four types of scenarios: precision measurement, component assembly, appearance inspection, and material handling. The samples include six dimensions: scene images, force timing, natural language instructions, hardware scheduling tags, standard execution processes, and operation parameters.

[0069] Specifically, the raw industrial multi-source data is categorized and aggregated according to task type and data dimension to obtain categorized and aggregated industrial multi-source data. First, based on the functional attributes of the work, all raw data is divided into four task types: precision measurement, component assembly, appearance inspection, and material handling. Then, within each task type, data is further categorized and stored according to six data dimensions: scene image, force sensing time series, natural language commands, hardware scheduling tags, standard execution procedures, and operating parameters. This ensures that work data from the same scene is grouped into the same task category, and data with the same attribute is grouped into the same dimension category, completing the hierarchical and orderly organization of the raw data to obtain categorized and aggregated industrial multi-source data. The categorized and aggregated industrial multi-source data is then subjected to format validation and dimension alignment to obtain a structured multimodal industrial work dataset. The format validation step verifies the data for each task category and each data dimension. The format compliance and content validity are verified. For scene images, resolution consistency, imaging integrity, and file format compliance are checked. For force-sensory time series data, sampling frequency consistency, numerical range rationality, and timestamp continuity are checked. For text and label data, field integrity and expression standardization are checked. Invalid data entries with incompatible formats, values ​​exceeding the reasonable range of the process, or missing key fields are removed. In the dimension alignment process, the unique identifier of a single complete operation instance is used as the association primary key. Scene images, force-sensory time series, natural language instructions, hardware scheduling tags, standard execution procedures, and operation parameters corresponding to the same operation process are matched and bound one by one to ensure that the six dimensions of data of a single sample correspond to the same operation instance. All samples have a unified data structure and field standardization, and finally form a structured multimodal industrial operation dataset in which each sample contains six dimensions of data and fully covers four types of industrial operation scenarios.

[0070] Scene images are raw visual data acquired through industrial vision acquisition equipment during industrial operations. Specifically, they include visual content such as the shape and position of workpieces in the work station, the status of tooling fixtures, the layout of surrounding equipment, and the distribution of environmental obstacles. They cover the texture features, spatial location, and status information of the work scene, and correspond to a visual snapshot of a single operation. They are the raw visual dimension data in multimodal data.

[0071] Force sensing time series is a time-series mechanical data collected by a six-dimensional force sensor at a fixed sampling frequency during the operation. Specifically, it includes the three-dimensional spatial force components, three-dimensional torque components and timestamp information of each sampling point at the end of the robotic arm. It can completely record the contact force change process throughout the entire operation cycle, reflect the contact state and mechanical interaction characteristics of the operation, and is the original data of the force sensing dimension in multimodal data.

[0072] Natural language instructions are descriptions of job requirements carried in the form of natural language text. They specifically include core task elements such as task type, job object, accuracy requirements, process constraints, and target output. They are used to clarify the goals and execution rules of a single job and are the original data of the task semantic dimension in multimodal data.

[0073] Hardware scheduling tags are standardized tag data that annotate the hardware configuration scheme corresponding to a single job. Specifically, they include information such as the type of end effector adapted to the job, the combination of sensors to be enabled, the working mode and parameter configuration of each hardware, etc. They are used to define the combination of hardware resources required to complete the corresponding job and are hardware configuration dimension annotation data in multimodal data.

[0074] The standard execution flow is a standardized sequence of steps for a corresponding job task. Specifically, it includes multiple ordered sub-operations obtained from the overall task decomposition, the connection and triggering conditions of each sub-operation, the intermediate state judgment rules, and the job termination conditions. It fully defines the execution sequence and flow logic of the job and is the execution flow dimension annotation data in multimodal data.

[0075] Operational parameters are a set of core physical parameters used to control the state and accuracy of the operation during the execution of the task. Specifically, they include continuous numerical parameters such as the robot arm's movement speed, contact force threshold, positioning accuracy threshold, sensor sampling frequency, and work feed rate. They are the core indicators for quantitatively controlling the execution of the task and are also the original data of the continuous physical quantity dimension in multimodal data.

[0076] Step 102: Perform parameter discretization and multi-task dialogue pair sample construction processing on the structured multimodal industrial operation dataset to output a standardized multi-task dialogue pair training sample set.

[0077] The structured multimodal industrial operation dataset includes multiple multimodal labeled samples covering various industrial operation scenarios.

[0078] The multimodal labeled samples, covering various industrial operation scenarios, form the basic data unit set constituting the structured multimodal industrial operation dataset. They comprehensively cover four typical industrial operation scenarios: precision measurement, component assembly, appearance inspection, and material handling. All samples adhere to unified data labeling specifications and format standards, and each sample independently corresponds to a complete industrial operation instance. This serves as the fundamental data carrier for parameter discretization processing and multi-task dialogue sample construction. From a scenario classification perspective, this sample set can be divided into four sample subsets, each matching one of the four operation scenarios. Specifically, the precision measurement samples contain instance data for workpiece dimension inspection and geometric tolerance measurement operations; the component assembly samples contain instance data for component insertion, precision bonding, and locking assembly operations; the appearance inspection samples contain instance data for workpiece surface defect identification and appearance integrity verification operations; and the material handling samples contain instance data for workpiece gripping, location transfer, and fixed-point stacking operations. From the perspective of the composition of a single sample, each multimodal labeled sample corresponds to a single complete operation process, specifically including scene images, force sensing time series data, natural language instruction text, hardware scheduling annotations, perception strategy annotations, execution process annotations, output type annotations, as well as continuous physical annotation values ​​such as force, pose, and operation parameters; all dimensional data within the same sample are bound to the same unique identifier of the same operation instance, corresponding one-to-one with each other, and together constitute a complete single operation instance annotation unit.

[0079] It should be noted that, firstly, the continuous physical parameters such as force, pose, and operation parameters contained in each sample of the structured multimodal industrial operation dataset are discretized into text. According to the preset numerical classification rules, the continuous numerical values ​​are converted into corresponding hierarchical semantic text, thus aligning the physical quantities with the text semantic space. Then, the scene image and natural language instructions corresponding to each sample are used as the model input data, and the five-dimensional decision content such as the hardware scheduling label, standard execution process, and discretized operation parameters of the corresponding sample are used as the model output data. The input and output data are matched and bound one-to-one according to the unique identifier of the sample. Finally, all the matched sample units are integrated to obtain a standardized multi-task dialogue pair training sample set with a unified format and standardized structure, which is used to support the subsequent lightweight fine-tuning of the model.

[0080] Furthermore, step 102 may include the following sub-steps:

[0081] S21. Semantic discrete encoding is performed on the continuous physical annotation values ​​in each multimodal annotation sample to obtain the discrete key operation parameter text corresponding to each multimodal annotation sample.

[0082] S22. Use the scene image and natural language instruction text in each multimodal labeled sample as the model input data, and use the hardware scheduling annotation, perception strategy annotation, execution process annotation, output type annotation and discretization key operation parameter text in the corresponding multimodal labeled sample as the model output data.

[0083] S23. Match the model input data and model output data corresponding to each multimodal labeled sample to obtain a standardized multi-task dialogue pair training sample set.

[0084] It should be noted that continuous physical parameters such as force perception and pose are discretized into text, and multi-task dialogue pairs are constructed for training samples with image + instruction input and five-dimensional decision + execution parameter output, so as to align physical quantities with semantic space.

[0085] Specifically, by pre-combining industrial operation process specifications and parameter control granularity, a mapping table is established between the numerical grading ranges of continuous physical parameters and their corresponding semantic labels. This clarifies the multi-level numerical ranges and matching semantic descriptions for various physical parameters such as force threshold, motion speed, positioning accuracy, and sampling frequency. For example, for the force threshold parameter, based on the mechanical safety and precision control requirements of contact operations, it is divided into three numerical ranges: low contact force (0~10N), medium contact force (10~30N), and high contact force (above 30N), with corresponding semantic labels of low contact force threshold, medium contact force threshold, and high contact force threshold, respectively. For the robotic arm motion speed parameter, based on the stability and efficiency requirements of different operations, it is divided into low speed (0~30mm / s), medium speed (30~100mm / s), and high speed (above 100mm / s). The three numerical ranges are labeled with semantic tags: low-speed motion mode, medium-speed motion mode, and high-speed motion mode. For positioning accuracy parameters, based on the accuracy requirements of different operations, they are divided into three numerical ranges: high-precision (within ±0.02mm), medium-precision (±0.02mm~±0.1mm), and conventional (above ±0.1mm), with corresponding semantic tags of high-precision control, medium-precision control, and conventional precision control. For sensor sampling frequency parameters, based on the required data acquisition density, they are divided into three numerical ranges: low-frequency sampling (10~50Hz), medium-frequency sampling (50~200Hz), and high-frequency sampling (above 200Hz), with corresponding semantic tags of low-frequency sampling mode, medium-frequency sampling mode, and high-frequency sampling mode. This forms a mapping table where the numerical ranges of various continuous physical parameters correspond one-to-one with their semantic tags.

[0086] Subsequently, all multimodal labeled samples are traversed sample by sample, and continuous physical label values ​​such as force, pose, and operation parameters are extracted. The actual values ​​of each parameter are matched one by one to the corresponding intervals in the mapping table and converted into corresponding hierarchical semantic text. For example, for a multimodal labeled sample of a component assembly scenario, the continuous physical label values ​​extracted are: contact force threshold of 15N, end effector velocity of 50mm / s, positioning accuracy requirement of ±0.05mm, and force sampling frequency of 100Hz. The values ​​of each parameter are matched one by one to the interval ranges in the mapping table. 15N falls into the medium contact force range of 10~30N, corresponding to the medium contact force threshold; 50mm / s falls into the medium speed range of 30~100mm / s, corresponding to the medium speed motion mode; ±0.05mm falls into the medium precision level of ±0.02mm~±0.1mm, corresponding to the medium precision control; 100Hz falls into the medium frequency sampling range of 50~200Hz, corresponding to the medium frequency sampling mode. In this way, the discrete key operation parameter text corresponding to the sample is generated, and the alignment of continuous physical quantities to text semantic space is completed.

[0087] Based on this, scene images and natural language command texts are extracted from each multimodal labeled sample, and both are used as the model input data for that sample. Simultaneously, hardware scheduling annotations, perception strategy annotations, execution flow annotations, and output type annotations are extracted from the same sample. These, combined with the previously generated discretized key operation parameter texts, correspond to five decision dimensions: hardware configuration, multimodal perception strategy, task execution flow, data output type, and key operation parameters, respectively, constituting the model output data for that sample. Then, using the unique instance identifier of each multimodal labeled sample as the association key, the model input data corresponding to the same sample is linked... Data is bound and encapsulated with model output data to form a single, structurally complete dialogue pair training sample. After encapsulating all multimodal labeled samples, the samples are shuffled and formatted according to task categories. Abnormal samples with missing input or output dimensions, mismatched labels, or structural incompleteness are removed. Finally, a standardized multi-task dialogue pair training sample set with a regular structure, complete dimensions, and coverage of multiple work scenarios is obtained. This provides a unified format of training data support for the cross-task lightweight fine-tuning of VLA models, helps a single model learn the decision logic of multiple industrial operations, and effectively supports the improvement of the cross-task flexible adaptation capability of industrial robots.

[0088] Step 103: Use standardized multi-task dialogue to perform offline lightweight fine-tuning of the general vision-language-action base model on the training sample set, and output an industrial-adaptive VLA base model.

[0089] It should be noted that a general vision-language-action (VLA) base model is used as the training base. A low-rank adaptive algorithm is used to perform lightweight fine-tuning. All backbone network weights of the base model are frozen, and only trainable low-rank matrix pairs are inserted into the weight matrix of the model's attention layer. The standardized multi-task dialogue pair training sample set is used as supervised training input for iterative training. During the training process, only the parameters of the low-rank matrix are updated, so that the model gradually learns the five-dimensional decision logic and cross-task planning capabilities in industrial scenarios. After the model loss converges and the decision accuracy of the validation set reaches a preset threshold, the model weight file with fused low-rank parameters is exported to obtain an industrial-adapted VLA (Vision-Language-Action) base model adapted to industrial operation scenarios.

[0090] Among them, the general vision-language-action base model is a pre-trained multimodal pre-trained model with integrated capabilities of visual perception, language understanding and action decision-making. It can receive joint input of image data and natural language text and output the action decision results corresponding to the task. Its specific network architecture is not particularly limited and any model structure that can realize the above vision-language-action joint modeling function can be adopted. This embodiment of the invention does not impose any restrictions on this. For example, a general vision-language-action base model can adopt a unified architecture of three-branch encoding and fusion decision. The vision branch extracts multi-scale features from the input scene image and depth data based on a pre-trained vision Transformer encoder. The language branch performs semantic encoding of natural language operation instructions based on a pre-trained large language model encoder. The action branch models the temporal features of historical motion sequences based on a lightweight temporal Transformer. The three features are mapped to the same high-dimensional feature space through a cross-modal attention fusion module for weighted interactive fusion. Finally, the action decision head outputs a discrete sequence of action tokens or a continuous robotic arm motion trajectory. The overall model can be lightweightly fine-tuned by freezing the backbone weights and only updating the low-rank adaptation matrix to adapt to various types of industrial operation scenarios.

[0091] Furthermore, step 103 can be achieved through the following steps:

[0092] The backbone weights of the general vision-language-action (VLA) base model are frozen. Trainable low-rank matrix pairs are inserted into the attention layer weight matrix of the general vision-language-action base model. The standardized multi-task dialogue pair training sample set is used as the training input of the general vision-language-action base model to carry out supervised training, thus obtaining the industrial-adapted VLA base model.

[0093] It should be noted that this invention is based on a general VLA base model and uses the Low-Rank Adaptation (LoRA) algorithm to perform offline lightweight fine-tuning, freezes the weights of the base backbone, and only updates the parameters of the low-rank matrix, so that the model has the ability to make five-dimensional autonomous decisions and cross-task planning in industrial scenarios, and outputs a deployable base model.

[0094] Specifically, firstly, all backbone network weights of the general vision-language-action base model are locked, covering all native parameters of the visual encoding branch, language semantic encoding branch, and action decision decoding branch, ensuring they do not participate in gradient updates and remain numerically fixed throughout the fine-tuning process. Then, in each Transformer layer of the model, a set of trainable low-rank matrix pairs is inserted into the side branches of the attention weight matrix, including the query matrix Q, key matrix K, value matrix V, and output projection matrix O. Each low-rank matrix pair consists of a d×r downlink matrix A and an r×d uplink matrix B, where r is a preset low-rank value much smaller than the feature dimension d of the original weight matrix. During initialization, downlink matrix A is assigned values ​​using a Gaussian random distribution, and uplink matrix B is initialized to zero, ensuring that the model output characteristics are completely consistent with the original base model at the start of fine-tuning. At this point, the equivalent weights of the attention layer can be expressed as W' = W0 + BA. W0 represents the frozen native attention weight matrix. After deploying the low-rank matrix, a standardized multi-task dialogue training sample set is input into the model in batches. Scene images and natural language commands in the samples are used as input data, and the corresponding hardware configuration, multimodal perception strategy, task execution process, data output type, and key operation parameters (five dimensions of decision content) are used as supervision labels. The cross-entropy loss function is used to calculate the loss value between the model's predicted output and the true label. The gradient is calculated through the backpropagation algorithm, and the parameters of all low-rank matrix pairs are updated only. Multiple rounds of training are carried out iteratively at a preset learning rate. Training stops when the decision generation accuracy of the validation set reaches a preset threshold and the loss function value tends to stabilize and converge. This allows the model to fully learn the five-dimensional autonomous decision-making logic and cross-task planning capabilities in industrial scenarios. Finally, the trained low-rank matrix parameters are fused with the backbone weights of the base model to obtain an industrial-adapted VLA basic model that can be directly deployed and executed for inference.

[0095] Step 104: Acquire multimodal perception data and natural language operation instructions input by the user, and perform time-stamp synchronization and alignment of RGB images, depth data and six-dimensional force perception data in the multimodal perception data, and output the time-aligned multimodal perception data.

[0096] Multimodal sensing data is a collection of various types of sensing data collected in real time from the industrial work site during the online operation phase. Specifically, it includes three categories: RGB images, depth data, and six-dimensional force data, which are collected by corresponding sensors. It can reflect the visual appearance, spatial structure, and contact mechanical state of the work scene in real time and serves as the real-time sensing input for the model to carry out scene understanding and work decisions.

[0097] User-input natural language job instructions are textual job requirements that users input into the system through interactive terminals. These instructions specifically include core task elements such as job type, job object, accuracy requirements, process constraints, and completion standards. They are used to clarify the goals and execution rules of the job and serve as the semantic input basis for driving the model to generate corresponding job solutions.

[0098] It should be noted that, regarding multimodal perception and natural language task parsing, to address the problems of traditional industrial robots having a single perception dimension, high task input threshold, and low efficiency in changeover adaptation, a multi-source fusion global perception system is constructed. This system relies on a pre-trained multimodal large model to achieve end-to-end task parsing and scene understanding of natural language commands.

[0099] This invention first simultaneously acquires RGB images, depth data, and six-dimensional force data of the work scene, covering three perception dimensions: two-dimensional texture features, three-dimensional shape information, and contact mechanical state. Through hardware synchronous triggering and hand-eye calibration, the time alignment and spatial unification of multimodal data are completed, providing spatiotemporally consistent full-dimensional data support for subsequent scene understanding and work control.

[0100] At the time synchronization level, a hardware trigger signal is used as a unified time reference to assign a synchronization timestamp to each frame of RGB image, each group of depth point cloud, and each force sequence. The time synchronization error... satisfy:

[0101] ;

[0102] In the formula, For RGB image timestamps, For deep data timestamps, This is the preset synchronization error threshold.

[0103] In this embodiment, RGB industrial cameras and depth cameras deployed at the workstation, along with a six-dimensional force sensor installed at the end of the robotic arm, collect three types of sensory data: two-dimensional texture features, three-dimensional shape information, and contact mechanical state of the work scene. These data collectively constitute real-time multimodal sensory data. Simultaneously, natural language work instructions input by the user through an interactive terminal are received to clarify the task requirements for the current operation. At the time synchronization level, a hardware synchronization trigger signal is used as a unified time reference. The synchronization control unit simultaneously sends synchronization acquisition trigger signals to the three types of sensory devices, driving each device to start data acquisition according to a unified timing sequence. Each frame of RGB image, each set of depth data, and each force sampling point is marked with a corresponding unified timestamp. The time synchronization error between the RGB image and the depth data satisfies the constraint formula. The time deviation between the six-dimensional force perception data and the visual data is also controlled within the synchronization error threshold. After the timestamp is marked, the three types of perception data are matched and bound frame by frame according to the timestamp. Abnormal data frames with time deviations exceeding the synchronization error threshold are removed to ensure that the visual and force perception data corresponding to the same moment correspond one-to-one and are consistent in time sequence. Finally, the time-aligned multimodal perception data is output, providing spatiotemporally consistent full-dimensional data support for subsequent spatial alignment and cross-modal scene understanding.

[0104] Step 105: Based on the preset hand-eye homogeneous transformation matrix, spatially align the time-aligned multimodal perception data, and then perform cross-modal feature fusion in conjunction with natural language task instructions to output cross-modal fusion features and structured task definition vectors.

[0105] It should be noted that, using the pre-calibrated homogeneous hand-eye transformation matrix as the spatial mapping benchmark, the pixel coordinates and 3D point cloud coordinates corresponding to the camera coordinate system of the RGB image and depth data are uniformly transformed to the robotic arm base coordinate system. At the same time, the six-dimensional force perception data is synchronously mapped from the sensor coordinate system to the same benchmark coordinate system, eliminating the installation pose deviation of different sensors and completing the spatial dimension alignment of multimodal perception data. Subsequently, multi-scale scene features of the spatially aligned visual data and temporal interaction features of the force perception data are extracted respectively. Semantic parsing of the natural language task instructions is performed to obtain task semantic features. The three types of features are mapped to the same high-dimensional feature space through a shared projection layer and then weighted fusion is performed to obtain cross-modal fusion features that simultaneously carry scene vision, mechanical interaction and task semantic information. Simultaneously, the core elements such as the parsed task type, task object and accuracy constraints are encoded into standardized fixed-length vectors, and a structured task definition vector is output, providing a unified feature input and task constraint basis for subsequent end-to-end autonomous decision-making.

[0106] Furthermore, step 105 may include the following sub-steps:

[0107] S51. Based on the preset hand-eye homogeneous transformation matrix, the time-aligned multimodal perception data is spatially aligned to obtain spatiotemporally unified multimodal perception data.

[0108] S52. Perform semantic embedding calculations on natural language task instructions to obtain text feature vectors;

[0109] S53. Perform visual encoding calculations on the RGB images and depth data in the spatiotemporally unified multimodal perception data to obtain visual feature vectors;

[0110] S54. Perform temporal feature encoding on the six-dimensional force perception data in the spatiotemporally unified multimodal sensing data to obtain the force perception feature vector;

[0111] S55. Perform multimodal cross-modal attention weighted fusion calculation on the text feature vector, visual feature vector and force feature vector to obtain cross-modal fusion features;

[0112] S56. Based on cross-modal fusion features, construct a structured task definition vector.

[0113] It should be noted that, at the spatial synchronization level, for the eye-in-hand mounting configuration, the homogeneous transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system is solved based on the hand-eye calibration equation AX=XB, i.e., the preset hand-eye homogeneous transformation matrix X:

[0114] ;

[0115] In the formula, A is the relative transformation matrix of the robot arm's end effector poses in two positions, and B is the relative transformation matrix of the camera poses in two positions. For rotational transformation components, The translation transformation component is used. After solving the equation using the Tsai two-step method to obtain the hand-eye homogeneous transformation matrix, the depth point cloud and force detection points are uniformly mapped to the robot arm base coordinate system, achieving spatial alignment of multimodal data. The rotational transformation component in the relative transformation matrix A of the two poses of the robotic arm end effector is a 3×3 orthogonal rotation matrix that characterizes the spatial rotational relationship of the robotic arm end effector between two different poses. It is obtained by forward kinematics calculation from the readings of the robotic arm joint encoder. Let A be the translational transformation component in the relative transformation matrix A between two poses of the robotic arm's end effector. It is a 3D column vector representing the spatial translational displacement relationship between the robotic arm's end effector and two different poses. Together they form the complete relative pose transformation matrix A of the robotic arm's end effector.

[0116] Furthermore, based on the spatiotemporally aligned multimodal perception data and the natural language instructions input by the user, the system performs text semantic embedding, visual feature encoding, and force temporal feature encoding respectively, and fuses the three types of features through a cross-modal attention mechanism to generate cross-modal fusion features and structured task definition vectors for subsequent model inference.

[0117] Specifically, the text instructions are first semantically embedded to obtain text feature vectors. Visual feature vectors are obtained by visually encoding RGB images and depth data. .in, For dimension The real vector space is the text feature vector. In the high-dimensional feature space, all text feature vectors are one of the features in this space. A dimensional column vector, the size of which determines the ability to represent semantic information and the information capacity of the text. For dimension The real vector space is the visual feature vector. In the high-dimensional feature space, all visual feature vectors are one of the elements in this space. A dimensional real-valued column vector, the size of which determines the ability to represent visual information in a scene and the information capacity.

[0118] Next, the six-dimensional force perception data from the spatiotemporally unified multimodal sensing data are segmented and aggregated according to the sampling time window. Statistical features within each time window are extracted, and the variable-length force perception time sequence is compressed into a fixed-dimensional force perception feature vector through a temporal coding network. Among them, the sampling frequency of force data is higher than that of visual data. By using sliding window aggregation and statistical feature extraction, high-frequency time-series data is compressed into a fixed-dimensional feature vector that matches the visual frame rate, thereby realizing the temporal alignment and joint representation of multimodal data at the feature layer.

[0119] Finally, feature fusion is achieved through a cross-modal attention mechanism:

[0120] ;

[0121] in This indicates that the visual feature vector and the force feature vector are concatenated along the feature dimension; This represents the cross-modal attention calculation function, a core computational unit for achieving multimodal feature fusion. It uses text feature vectors as query input and a joint feature vector obtained by concatenating visual and force feature vectors as key and value inputs. By calculating the attention weight distribution between text semantics and perceptual features, it dynamically weights and aggregates visual regions and force state information with high relevance to task semantics, outputting cross-modal fused features that simultaneously carry task semantics, scene visuals, and contact mechanics information. Based on the constraints of workpieces, equipment, and environment in the fused feature recognition scenario, it automatically extracts task objectives, accuracy requirements, and process limitations, outputting a structured task definition vector. It supports instruction input for all types of industrial operations, including measurement, assembly, inspection, and handling.

[0122] In this embodiment, for the sensor mounting configuration with the eye on the hand, the pre-defined hand-eye homogeneous transformation matrix is ​​obtained by pre-calibrating and solving the hand-eye calibration equation AX=XB. The homogeneous matrix expression of this equation is as follows: The calibration process employs the Tsai two-step method to solve the equation, first separating and solving the rotation component, then substituting it into the translation component to obtain the homogeneous transformation matrix from the camera coordinate system to the robotic arm's end effector coordinate system. In the spatial alignment stage, based on the aforementioned preset hand-eye homogeneous transformation matrix and combined with the robotic arm's current real-time end effector pose, the 3D point cloud and 6D force detection points corresponding to the depth data are mapped from their respective sensor coordinate systems to the robotic arm's base coordinate system. This eliminates spatial deviations caused by differences in the installation poses of different sensors, completing spatial dimension alignment and obtaining spatiotemporally unified multimodal perception data with consistent temporal and spatial dimensions. Subsequently, the user-input natural language task instructions undergo semantic embedding calculations through a semantic embedding layer, mapping the text content to a high-dimensional semantic space, resulting in a dimension of... Text feature vectors Simultaneously, RGB images and depth data from spatiotemporally unified multimodal perception data are subjected to multi-scale feature extraction and visual encoding calculation using a visual encoder, resulting in a dimension of [missing information]. Visual feature vectors The high-frequency force perception time series is divided into time windows according to the visual sampling period. Multidimensional statistical features are extracted within each window. The statistical feature vectors of each window are arranged in chronological order and then compressed and mapped to a dimension of [dimensional value missing] using a multilayer perceptron network. Force feature vector Based on this, a cross-modal attention mechanism is used to perform weighted fusion calculations on the text feature vector, the visual feature vector, and the force feature vector. The fusion calculation process satisfies... By dynamically matching the associated regions of text semantics and multimodal perception scenes through attention weights, the correspondence between task requirements and scene visual information and mechanical interaction information is strengthened, resulting in cross-modal fusion features that simultaneously carry task semantic information, scene visual information, and mechanical interaction information. Finally, when constructing a structured task definition vector based on the cross-modal fusion features, an industrial operation task element coding dictionary is first pre-built, defining the fixed dimensions and field order of the structured task definition vector. Five core coding fields are set sequentially: task type, work object, accuracy level, process constraint, and target output. For each field, a corresponding enumerated value set and standardized coding rules are preset. The task type field corresponds to four operation categories: precision measurement, component assembly, appearance inspection, and material handling, using a 4-bit unique hot coding. The work object field covers common work objects such as electronic components, mechanical structural parts, and tooling fixtures, using an 8-bit category coding. The accuracy level field is divided into three control levels: high precision, medium precision, and conventional precision, using a 2-bit hierarchical coding. The process constraint field includes typical constraint types such as anti-static operations, constant force contact, and obstacle avoidance operations, using a 4-bit constraint coding. The target output... The fields correspond to numerical measurement results, assembly status signals, defect detection reports, and other output formats, using 3-bit output encoding. All field codes are concatenated in a fixed order to form a fixed-length vector structure with uniform dimensions. In the actual construction process, cross-modal fusion features are input into a pre-trained task element decoding network. This network adopts a multi-label classification structure, which can simultaneously perform joint analysis from the textual semantic information and visual scene information carried by the fusion features. First, based on the semantic feature dimension, it extracts the task objectives, accuracy requirements, and constraints explicitly stated in the natural language instructions. Then, based on the visual feature dimension, it identifies the workpiece type, spatial location, and environmental constraints in the work scene. The semantic extraction results are visually verified and supplemented to avoid element deviations caused by ambiguous instruction expressions. Finally, it completely extracts five core task elements that match the current operation: task type, work object, accuracy level, process constraints, and target output. Subsequently, each extracted task element is matched one by one with the task element encoding dictionary and converted into the standard encoding of the corresponding field. The encodings of the five fields are concatenated and integrated in a preset field order to finally obtain a structured task definition vector with a standardized format and fixed dimensions. Taking a real-world component assembly scenario as an example, when a user issues a natural language instruction to "complete the precise insertion of chips on the PCB circuit board, using a medium-speed operation mode," the task element decoding network generates cross-modal fusion features by combining the collected images of the work scenario, which include the PCB circuit board and the chips to be assembled. The task element decoding network then extracts the following information from the fusion features: the task type is component assembly, the work objects are chips and PCB circuit boards, the accuracy level is medium precision, the process constraint is smooth insertion, and the target output is an assembly completion signal. After converting the matching coding dictionary corresponding to each element into a standard code, the code is concatenated into a fixed-length vector in a preset order, thus obtaining the structured task definition vector corresponding to this operation.

[0123] Step 106: Using the industrial-adaptive VLA basic model, end-to-end autonomous decision-making and job execution control are performed based on the preset hand-eye homogeneous transformation matrix, cross-modal fusion features and structured task definition vectors, and the final job results are output.

[0124] It should be noted that cross-modal fusion features and structured task definition vectors are jointly input into the industrial-adaptive VLA basic model. A preset hand-eye homogeneous transformation matrix is ​​used as the spatial coordinate transformation benchmark to support the coordinate mapping calibration of spatial position parameters in the decision results. The model generates a five-dimensional structured decision result through end-to-end inference, which includes hardware combination configuration, multimodal perception strategy, task execution process, data output type, and key operation parameters. The decision result is then parsed and converted into robotic arm motion control instructions, sensor working mode configuration instructions, and end effector action instructions to drive each hardware device to execute the corresponding industrial operation according to the decision scheme. During the operation execution, real-time perception data is synchronously transmitted back to assist the model in dynamically adjusting the execution parameters. After the complete operation process is completed, the corresponding standardized output content is generated according to the output type determined by the decision, and the final operation result is output.

[0125] Furthermore, step 106 may include the following sub-steps:

[0126] S61. Input the cross-modal fusion features and structured task definition vectors into the industrial-adaptive VLA base model to perform inference and generate multi-dimensional structured decision vectors.

[0127] S62. Decompose the multidimensional structured decision vector to obtain a discrete motion waypoint sequence;

[0128] S63. Perform trajectory smoothing calculation on the discrete motion waypoint sequence to obtain a smooth motion trajectory sequence;

[0129] S64. Perform compliance correction calculation and trajectory superposition based on the smooth motion trajectory sequence and real-time multi-dimensional force sensory data, and output real-time motion control commands.

[0130] S65. Drive the end effector of the industrial robot's robotic arm to complete the corresponding operation through real-time motion control commands. During the execution process, based on the preset hand-eye homogeneous transformation matrix and real-time joint pose data, complete the dynamic calibration of multiple sensors by minimizing the reprojection error and generate the final operation result.

[0131] It should be noted that after obtaining the structured task definition vector, the subsequent steps involve VLA-driven cross-task integrated decision-making and multi-sensor collaborative industrial task execution. Addressing the core pain points of traditional solutions—strong single-task customization, weak cross-task generalization ability, and a disconnect between perception, decision-making, and execution—this approach uses a Vision-Language-Action (VLA) multimodal large-scale model as the decision-making center to achieve integrated implementation of autonomous decision-making and precise execution. The specific implementation steps are as follows:

[0132] ① The VLA large model, based on the fusion of features and task definition, autonomously outputs structured decision results in five dimensions, forming a multidimensional structured decision vector D:

[0133] ;

[0134] In the formula, It can configure hardware combinations and autonomously match the type of end effector, sensor combination, and hardware operating mode required for the current task. For multimodal perception strategies, the number of data acquisition viewpoints, sampling frequency, and sensor triggering timing are set autonomously; To define the task execution process, the system autonomously breaks down the overall task into multiple ordered sub-operations and determines the connection logic and judgment conditions of each step. The data output type allows users to independently determine the output format of the final result. For key operational parameters, the robot can autonomously set core physical quantities such as movement speed, contact force threshold, and sensor parameters. The entire decision-making logic is generated autonomously by the model, eliminating the need to write separate control programs for different tasks, thus achieving universal adaptability of a single model to multiple types of industrial operations.

[0135] ② Based on the five-dimensional decision results, the semantic operation process is transformed into a discrete motion waypoint sequence that can be recognized by the robotic arm controller. To ensure motion stability, a smooth trajectory is generated between adjacent waypoints using fifth-order polynomial interpolation. The position interpolation in Cartesian space satisfies:

[0136] ;

[0137] This function represents the position of the robotic arm's end effector as a function of time in Cartesian space. It describes the mapping relationship between the end effector's position and time along its motion trajectory and is the core computational object for fifth-order polynomial trajectory interpolation. It is used to generate smooth, continuous motion paths without abrupt changes, and the coefficients are solved by considering the position, velocity, and acceleration boundary conditions at the start and end points. This ensures that the velocity and acceleration of the trajectory remain continuous and without sudden changes at the junction, thus avoiding mechanical impact.

[0138] ③ For contact-type operations such as assembly and measurement, an admittance control algorithm is used to achieve force / position hybrid compliant control, converting the external contact force detected by the six-dimensional force sensor into a position correction value. The second-order admittance control equation in the continuous domain is:

[0139] ;

[0140] In the formula, M is the virtual mass matrix. K is the virtual damping matrix, and K is the virtual stiffness matrix. This is the position offset correction amount. The external contact force vector detected by the six-dimensional force sensor. The second derivative of the position offset correction with respect to time, i.e., the acceleration of the position offset correction, is a three-dimensional or six-dimensional vector that characterizes the acceleration change of the position adjustment during compliance correction. Together with the virtual mass matrix, it determines the degree of influence of the contact force change on the trajectory correction acceleration.

[0141] Since the robot controller is a discrete sampling system, the continuous domain equations are discretized using the bilinear transform (Tustin transform), let:

[0142] ;

[0143] Substitution The discrete-domain recursive control formula is obtained by reorganization:

[0144] ;

[0145] In the formula, The position offset correction amount is calculated at the k-th discrete sampling time. This represents the external contact force vector detected by the six-dimensional force sensor at the k-th sampling time. For the complex frequency variable of the Laplace transform, Let T be a complex variable of the z-transform. s Let k be the controller sampling period, and k be the discrete sampling time. , and , , This represents the discretization coefficient. The position correction is calculated in real-time using this recursive formula and superimposed onto the planned trajectory to achieve compliant control, ensuring the stability and safety of contact operations.

[0146] ④ During the operation, multi-view perception data is continuously collected. Based on the hand-eye homogeneous transformation matrix and real-time joint pose, real-time spatial registration and timestamp alignment are performed on the multi-source data. Dynamic calibration is completed by minimizing reprojection error.

[0147] ;

[0148] In the formula, Reprojection error is used to quantify the deviation between the projected pixels obtained after spatial transformation and camera projection of 3D points and the corresponding actual pixels in the image. Its accumulated value is a core indicator for evaluating hand-eye calibration accuracy and spatial registration effect, and is also an optimization target for the multi-sensor dynamic calibration process. The total number of feature points involved in the reprojection error calculation corresponds to the total number of successful pairings of 3D spatial points and 2D pixels in the operational scenario. A higher number of feature points results in more robust and reliable dynamic calibration. For image pixels, For a point in three-dimensional space, For camera projection function, This is the real-time homogeneous transformation matrix from the robot arm's base coordinate system to the camera coordinate system at the current moment, obtained by cascading and multiplying the real-time end-effector pose matrix with the preset hand-eye homogeneous transformation matrix. Real-time calibration ensures the spatial consistency and temporal synchronization of perceived data and motion control, guaranteeing the execution accuracy of various tasks.

[0149] In this embodiment, after receiving cross-modal fusion features and a structured task definition vector, the industrial-adaptive VLA base model outputs a five-dimensional structured decision vector through end-to-end inference. The expression form of this decision vector is as follows: After generating the decision vector, the task execution process and key operation parameters in the decision vector are analyzed. Combined with the three-dimensional spatial coordinate information of the work scene, the semantic work steps are converted into a sequence of discrete motion waypoints executable by the robotic arm. Each waypoint contains the corresponding end-effector spatial pose, motion speed constraints, and action trigger commands. To ensure the smoothness of the robotic arm's motion, trajectory smoothing calculations are performed on the discrete motion waypoint sequence. A continuous and smooth motion trajectory is generated between two adjacent waypoints using fifth-order polynomial interpolation. The position interpolation function in Cartesian space satisfies... The polynomial coefficients are solved by substituting the position, velocity, and acceleration boundary conditions of the starting and ending points. To ensure the continuity and lack of abrupt changes in velocity and acceleration at waypoint junctions, avoiding mechanical impact, a smooth motion trajectory sequence is ultimately obtained. For assembly, measurement, and other work scenarios involving mechanical contact, compliance correction calculations are performed on the planned trajectory using real-time acquired multi-dimensional force sensor data. An admittance control algorithm is employed to achieve force / position hybrid compliance control, converting the external contact force detected by the six-dimensional force sensor into trajectory position correction values. The second-order admittance control equation in the continuous domain is: Since the robot controller is a discrete sampling system, the continuous-domain admittance equation is discretized using the bilinear transformation, i.e., the Tustin transform. The transformation relationship is as follows: Substituting this into the continuous domain equation and rearranging, we obtain the discrete domain recursive control formula. The position correction amount is calculated time by time using the recursive formula, and the correction amount is superimposed on the corresponding position of the smooth motion trajectory to output a real-time motion control command that can be directly issued and executed.

[0150] Finally, real-time motion control commands are sent to the robotic arm motion controller. The controller calculates the target angles of each joint according to the commands and drives the joint motors to move the robotic arm end effector along the compliantly corrected trajectory, sequentially completing the corresponding operations in the work process, such as grasping, alignment, contact, assembly, inspection, or stacking. Multi-sensor dynamic calibration is continuously performed throughout the entire operation. During calibration, the absolute encoders of each joint of the robotic arm first collect the current joint angle data in real time, substitute it into the forward kinematics equations of the robotic arm, and solve to obtain the homogeneous transformation matrix of the robotic arm end-effector coordinate system relative to the robotic arm base coordinate system at the current moment, i.e., the real-time end-effector pose matrix. Simultaneously, a preset hand-eye homogeneous transformation matrix obtained through pre-calibrated hand-eye interaction is called. This matrix represents the fixed relative pose relationship between the camera coordinate system and the robotic arm end-effector coordinate system, including rotational and translational transformation components, and does not change with the movement of the robotic arm. The real-time end-effector pose matrix and the preset hand-eye homogeneous transformation matrix are then multiplied by homogeneous matrix concatenation to obtain the real-time transformation matrix from the robotic arm base coordinate system to the camera coordinate system at the current moment. This allows the real-time pose of the camera to be uniformly mapped to the reference coordinate system of the robotic arm base; based on this, N scene feature points in the current frame's RGB image are extracted as the actual observed pixels. Simultaneously, the three-dimensional spatial points corresponding to each feature point are obtained from the registered depth data. After substituting the three-dimensional spatial points into the real-time transformation matrix to complete the coordinate transformation, the coordinates are then projected through the camera projection function. Mapping to a two-dimensional image plane yields the corresponding theoretically projected pixels. The sum of the Euclidean distances between the actual pixels and the theoretically projected pixels of all feature points is then calculated to obtain the reprojection error. With minimizing reprojection error as the optimization objective, a nonlinear optimization algorithm is used to dynamically fine-tune the pose parameters of the preset hand-eye homogeneous transformation matrix to offset calibration deviations caused by factors such as mechanical vibration, temperature drift, and load deformation during operation. This continuously corrects the spatial mapping relationship between multimodal sensing data and robotic arm motion control, ensuring spatiotemporal alignment accuracy. After the complete operation process is completed, multimodal sensing data at the operation completion time is collected for result verification. The corresponding operation data and state judgment conclusions are organized according to the data output type of the decision output to generate standardized final operation results.

[0151] The final work result is the standardized output data generated by the industrial robot after completing the entire process of the work according to the work decision plan, based on the preset data output type. It is the final output carrier of a single industrial work execution process, carrying the execution results, quantitative indicators and status judgment information of the corresponding work. It can be directly used for status feedback of the production process, work data archiving and product quality traceability. All result data are bound to the unique identifier of the corresponding work instance, and can be connected and synchronized with the production management system.

[0152] Depending on the type of work scenario, the specific composition of the final work result varies. For precision measurement work, the final result includes the measured dimensions of the workpiece, calculated form and position tolerances, coordinates of measurement points, measurement deviations, accuracy qualification conclusions, and corresponding measurement calibration parameters and timestamps. It can be output as a standardized measurement value report or a dimensional inspection report. For component assembly work, the final result includes assembly completion status signals, positional accuracy deviations, torque confirmation data for tightening operations, records of abnormal events during assembly, and visual verification conclusions after assembly. It can be output as an assembly completion status message and an assembly quality confirmation signal. For appearance inspection work, the final result includes the type, location coordinates, and dimensional parameters of surface defects on the workpiece, defect level judgment results, workpiece appearance integrity verification conclusions, and visual index information for defect markings. It can be output as a standardized defect inspection report and a pass / fail report. Non-conformance judgment signals; the final operation results of material handling operations specifically include the workpiece successfully picked up status, target station arrival confirmation signal, stacking position deviation, workpiece posture correction result, and operation completion status indicator, which can be output as arrival trigger signal and operation completion status feedback. From a general structure perspective, the final operation results of all scenarios include three components: basic operation identification information, core result data, and status judgment conclusion, fully covering the operation's identity information, core output, and quality judgment dimensions.

[0153] Optionally, it also includes:

[0154] Obtain the entire process data package, extract multi-dimensional operation data from the entire process data package and encapsulate it in a structured manner according to a unified format to obtain multiple structured samples;

[0155] Calculate the comprehensive weight for multiple structured samples and attach labels to them to obtain multiple structured job samples with weight labels;

[0156] Multiple structured job samples with weighted labels are classified, stored, and indexed in a standardized manner. Based on the sample weight, high-value samples are retained first and low-value duplicate samples are filtered out to obtain a dynamically expandable sample experience pool.

[0157] When the effective sample size of the dynamic sample experience pool reaches the preset trigger threshold, the industrial-adaptive VLA basic model is trained on the newly added samples in the dynamic sample experience pool. The training process introduces a knowledge retention regularization term to suppress catastrophic forgetting, and the incrementally updated VLA candidate model is obtained.

[0158] The incrementally updated VLA candidate models are validated to obtain a new industrial-adaptive VLA base model.

[0159] It should be noted that this invention conducts cross-task data accumulation and model closed-loop self-learning. Addressing the problems of traditional fixed-deployment models being unable to adapt to changing operating conditions, having high iteration costs, and lacking autonomous evolution capabilities, it constructs a full-process data accumulation mechanism and a lightweight iteration system to achieve continuous autonomous optimization of system capabilities. The specific implementation steps are as follows:

[0160] ① Automatically records the entire process data of all types of task execution. Each sample is structured and includes scene images, original task instructions, decision-making schemes, force-time sequences, execution trajectories, job results, and success / failure labels. Samples are categorized and stored according to task type, workpiece specifications, and execution effect, establishing a standardized indexing mechanism. A dynamic weighted selection mechanism is also introduced, assigning higher weights to failed samples and boundary samples.

[0161] ;

[0162] In the formula, Let i be the weight of the i-th sample. This is an indicator function; failed samples are assigned a value of 1, and successful samples are assigned a value of 0. To score the abnormality of the sample, , , where is the weighting coefficient. High-value samples are preferentially retained through weighted selection, constructing a dynamically expandable high-quality sample experience pool.

[0163] ②Preset sample accumulation trigger threshold N th When the number of newly added valid samples in the experience pool reaches a threshold, the model incremental fine-tuning process is automatically triggered. Lightweight training is performed using the Low-Rank Adaptive (LoRA) algorithm, which inserts trainable low-rank matrix pairs only into the weight matrix of the model's attention layer. The weight update form is as follows:

[0164] ;

[0165] In the formula, The original weights of the base model, , For trainable low-rank matrices, It is a low-rank number. This is the scaling factor. During training, all backbone weights of the base model are frozen, and only the low-rank matrix parameters are updated, significantly reducing computational overhead.

[0166] To suppress catastrophic forgetting of old tasks caused by incremental fine-tuning, a knowledge retention regularization term is introduced into the loss function, resulting in the final loss function. for:

[0167] ;

[0168] In the formula, Fit loss to the new task. The KL divergence is used to constrain the difference between the output distribution of the new model and the original model. This is the regularization weight coefficient. This regularization term allows for the retention of existing task capabilities while learning new task knowledge, supporting cross-task knowledge transfer.

[0169] ③ After incremental fine-tuning, the updated model is validated in multiple dimensions, including decision accuracy and execution success rate, using a reserved multi-task validation set. Once the validation is passed, the model is smoothly replaced during idle periods on the production line, achieving a seamless upgrade. Through continuous data accumulation and iterative optimization, the model can gradually adapt to more workpiece types and more complex working conditions, achieving long-term continuous improvement in decision accuracy and execution stability, ensuring the system's operational performance and scenario adaptability throughout its entire lifecycle.

[0170] In this embodiment, the entire process data package is automatically acquired by the data acquisition module deployed in the online operation system. Throughout the entire cycle of each task execution, it synchronously acquires scene images before the task starts, the user-inputted original task instructions, the decision-making scheme output by the industrial-adaptive VLA basic model, the six-dimensional force-sensing time sequence and the actual trajectory of the robotic arm throughout the task execution, the final task result after completion, and the task success / failure labels automatically determined by the quality verification module or manually reviewed and marked. All data is bound to a unique identifier corresponding to the task instance and packaged into a single complete entire process data package. After data acquisition, data for each dimension is extracted from the data package and structured according to a preset unified field specification. This ensures that each sample is standardized to include seven dimensions: scene image, original task instructions, decision-making scheme, force-sensing time sequence, execution trajectory, task result, and success / failure label, resulting in multiple structured samples. Subsequently, a comprehensive weight is calculated for each structured sample and a corresponding label is attached. The formula for calculating the sample weight is as follows: This formula assigns higher weight values ​​to failed samples and boundary condition samples, resulting in multiple structured operation samples with weight labels. After weight calculation, the samples are classified and stored according to three dimensions: task type, workpiece specifications, and execution effect. A standardized indexing mechanism is established to support fast retrieval. Simultaneously, a dynamic weight filtering mechanism is introduced to prioritize high-value samples with high weight values ​​and remove low-value duplicate samples with highly overlapping features. This controls the balance between sample quality and size in the experience pool, constructing a high-quality dynamic sample experience pool that can be dynamically expanded. A preset sample accumulation trigger threshold N is used. th When the number of newly added valid samples in the dynamic sample experience pool reaches the trigger threshold, the model incremental fine-tuning process is automatically triggered. A low-rank adaptive algorithm is used for lightweight training, updating only trainable low-rank matrix pairs in the weight matrix of the model's attention layer. The weight update form satisfies... During training, all backbone weights of the base model are frozen, and only the low-rank matrix parameters are updated to control computational overhead. To suppress the catastrophic forgetting of old tasks caused by incremental fine-tuning, a knowledge retention regularization term is introduced into the training loss function. The final loss function is calculated as follows: This regularization term enables the model to retain its original task decision-making ability while learning new working condition knowledge. After multiple rounds of iterative training, an incrementally updated VLA candidate model is obtained. Finally, a reserved multi-task validation set is used to conduct multi-dimensional verification of the candidate model. The verification indicators cover decision generation accuracy, cross-task execution success rate, and inference time. When all indicators meet the preset standards, the verification is deemed successful. The model is then smoothly replaced and seamlessly upgraded during the idle period of the production line, resulting in a new industrial-adaptive VLA basic model with stronger adaptability.

[0171] This new model will be directly deployed in the online operation phase, replacing the original decision model to handle the input reasoning of multimodal perception data and natural language operation instructions. It will output five-dimensional structured decision results adapted to the current working conditions, driving industrial robots to complete all types of operation tasks such as precision measurement, component assembly, appearance inspection, and material handling. At the same time, new data generated during the operation will be continuously fed back to the dynamic sample experience pool, forming a closed-loop self-learning link of data accumulation, model iteration, and performance upgrade. While maintaining low fine-tuning computing power costs, it can continuously expand the types of workpieces and the range of complex working conditions that the model can adapt to, steadily improve decision accuracy and operation execution stability, effectively alleviate the problems of traditional solidified models being unable to adapt to working condition drift, having high iteration costs, and lacking autonomous evolution capabilities, and ensure the operation performance and scenario generalization capabilities of the system throughout its entire life cycle.

[0172] For comparison of technological effects, existing technologies can be used as a reference. Against the backdrop of the continuous deepening of intelligent manufacturing, industrial robots have become core production equipment in fields such as electronics manufacturing, new energy, and precision machining. Their application scenarios are extending from repetitive tasks at a single workstation to flexible operations involving multiple categories and processes. At the same time, the pace of downstream product iteration is accelerating, requiring production lines to frequently switch between different tasks such as measurement, assembly, inspection, and handling. Furthermore, workpiece specifications and process requirements are dynamically adjusted with each batch, placing higher demands on the adaptability and autonomy of industrial robots.

[0173] Traditional industrial robots employ a fixed "teach-program - stationary execution" work mode. Each time a new task is added or a workpiece is replaced, specialized technicians must use offline programming software to set motion trajectories, configure sensor parameters, and write decision-making logic point by point. This results in long debugging cycles and high labor costs for each task changeover, making it difficult to match the flexible production rhythm of small batches and multiple varieties. Furthermore, the perception, decision-making, and execution modules of traditional control systems are independent, with no data communication, making it impossible to autonomously adjust work strategies based on on-site conditions and exhibiting poor robustness in unstructured scenarios.

[0174] With the development of artificial intelligence technology, multimodal large models are gradually being applied to industrial scenarios, but existing solutions still have obvious technical limitations:

[0175] Most large industrial models are only used at the perception layer, such as for appearance defect classification and workpiece identification and positioning. They can only output classification or coordinate results and cannot directly generate motion control instructions and work processes that can be executed by robots. The intermediate link of "model output - program conversion" needs to be completed manually, which does not truly achieve end-to-end intelligent operation and has limited efficiency improvement.

[0176] Meanwhile, most existing industrial intelligence solutions are customized for single tasks, training detection models for measurement tasks and control models for assembly tasks. Models cannot be reused between different tasks, and the ability to transfer knowledge across tasks is weak. Deploying multiple sets of models on the production line is costly and difficult to maintain.

[0177] In addition, existing models generally adopt the "offline training-fixed deployment" mode. After deployment, the performance cannot be optimized autonomously as production data accumulates. When faced with changes such as production line operating conditions drifting or the addition of new workpiece types, the model accuracy is prone to decay. It requires manual data collection and full retraining, resulting in long iteration cycles and high computing costs, which cannot adapt to the dynamic needs of long-term production line operation.

[0178] To address the aforementioned technical shortcomings, this invention proposes a cross-task flexible operation method for industrial robots based on a VLA (Vision-Language-Motion) multimodal large-scale model. Using this model as the core decision-making hub, it establishes a complete chain from "natural language commands to scene perception, autonomous decision-making, task execution, and closed-loop optimization," achieving end-to-end intelligent adaptation for various industrial tasks. The method completes scene information collection and command parsing through a full-domain multimodal perception system. The large-scale model autonomously outputs comprehensive task decisions and transforms them into executable motion plans. Simultaneously, it constructs a dynamic sample pool and a lightweight incremental fine-tuning mechanism to achieve continuous autonomous iteration of model performance, comprehensively improving the cross-task flexible adaptation capability and the level of intelligence throughout the entire lifecycle of industrial robots.

[0179] Specifically, such as Figure 2 As shown, the engineering implementation logic is divided into three core stages that are interconnected: offline training, online application, and continuous learning. These three stages work together to form a complete technical loop of "lightweight model adaptation - end-to-end job execution - closed-loop self-learning iteration", which enables industrial robots to adapt to various types of jobs such as measurement, assembly, inspection, and handling and continuously evolve their autonomous capabilities.

[0180] The first step is the offline training phase, which involves lightweight adaptation of the VLA model for industrial scenarios. Addressing the issues of insufficient industrial decision-making capabilities and excessively high costs associated with full-scale fine-tuning of general multimodal models, a multimodal industrial dataset covering multiple task types was collected. Data preprocessing was performed through text discretization of force / pose parameters, and standardized multi-task dialogue pairs were constructed. Based on the general vision-language-action (VLA) foundation model, a low-rank adaptive (LoRA) algorithm was used for offline lightweight fine-tuning, enabling the model to possess five-dimensional autonomous decision-making and cross-task planning capabilities for industrial scenarios. This provides a foundational model for adapting to industrial scenarios during the online operation phase.

[0181] The second stage is the online application phase, which enables VLA-driven cross-task integrated operation execution. After hardware system initialization, the system receives natural language task commands input by the user, integrates RGB images, depth data, and six-dimensional force perception data to form a full-domain perception data, and performs end-to-end inference using the VLA model. It autonomously outputs structured decision results in five dimensions: hardware combination configuration, multimodal perception strategy, task execution process, data output type, and key operation parameters. Based on the decision results, it generates structured motion planning to drive the robotic arm to complete the task execution. During the process, it simultaneously achieves force / position hybrid compliant control and multi-sensor spatiotemporal synchronous calibration. After execution, it automatically determines the success or failure of the task and outputs the result.

[0182] Finally, there is the continuous learning phase, which achieves cross-task data accumulation and closed-loop self-evolution of the model. Data from the entire workflow is automatically collected, categorized, and stored to build a dynamic sample experience pool. When the accumulated sample size in the experience pool reaches a preset fine-tuning threshold, LoRA incremental fine-tuning is automatically triggered. A knowledge retention mechanism is introduced during training to suppress catastrophic forgetting. After the model passes validation, it is smoothly replaced and deployed, achieving seamless iteration of the system's operational capabilities and cross-task knowledge transfer, enabling long-term adaptation to production line condition drift and new task requirements.

[0183] In addition, such as Figure 3 As shown, the implementation framework of a cross-task industrial robot flexible operation method based on VLA multimodal large model of the present invention is as follows: Figure 2 As shown, the core comprises three main functional modules:

[0184] (1) Offline model training and management module: responsible for lightweight adaptation of VLA models for industrial scenarios. It includes a multimodal dataset construction module, a parameter discretization preprocessing module, and a LoRA offline fine-tuning module. By collecting multi-task industrial data, constructing standardized dialogue samples, and performing low-parameter directional fine-tuning, it outputs an industrial-adapted VLA model with five-dimensional autonomous decision-making and cross-task planning capabilities, providing a model foundation for online operations.

[0185] (2) Online Sensing Decision and Operation Execution Module: This is the core execution unit of the system. It includes a multimodal sensing and acquisition module, a natural language command parsing module, a VLA five-dimensional decision module, a motion planning and compliant control module, and a multi-sensor synchronous calibration module. After receiving user natural language commands, it completes the entire process of scene perception, autonomous decision-making, motion generation, and compliant execution, supporting end-to-end execution of various types of industrial operations such as measurement, assembly, inspection, and handling.

[0186] (3) Continuous learning and iteration module: responsible for the autonomous evolution of system capabilities. It includes a full-process data acquisition module, a dynamic sample experience pool management module, a LoRA incremental fine-tuning module, and a model verification and replacement module; it automatically accumulates and classifies job data, triggers lightweight incremental fine-tuning after the sample accumulation reaches the target, and smoothly updates the model after verification, so as to realize the continuous improvement of system performance and cross-task knowledge transfer.

[0187] In summary, this invention addresses the problems of low efficiency in changing operational modes, weak cross-task adaptability, and insufficient autonomous iteration capability of current industrial robots. It proposes a cross-task flexible operation method based on a VLA multimodal large model, achieving the following main technical effects:

[0188] First, it significantly improves task changeover efficiency and lowers the operational threshold. By replacing traditional offline programming with natural language interaction, users can issue tasks without needing professional programming skills. The system automatically completes task parsing and solution generation, reducing the debugging time for task changeover from hours to minutes. This significantly improves the task changeover efficiency of flexible production lines and reduces reliance on professional technical personnel.

[0189] Second, it enables a single model to adapt to different tasks, reducing deployment costs. Through the generalization capability of the VLA large model, a single model can cover multiple types of industrial operations such as measurement, assembly, inspection, and handling, eliminating the need to develop and deploy independent control systems and algorithm models for different tasks, thus significantly reducing the deployment costs and maintenance complexity of intelligent transformation of production lines.

[0190] Third, it ensures operational stability and accuracy under complex working conditions. By combining multimodal perception fusion and force / position compliant control, it takes into account both the global perception capability of vision and the fine contact feedback of force perception, which can effectively adapt to unstructured scenarios and contact-based operation requirements, avoid the risk of rigid collisions, and ensure operational accuracy and equipment safety.

[0191] Fourth, it endows the system with autonomous evolution capabilities throughout its entire lifecycle. Through a closed-loop mechanism of dynamic sample pool and LoRA lightweight incremental fine-tuning, the system can continuously and autonomously optimize based on on-site operating data, completing model iterations without full retraining. It features low computing power costs, short iteration cycles, and can effectively adapt to the long-term needs of production line condition drift, new workpieces, and new tasks, ensuring the long-term performance stability of the system.

[0192] In summary, this invention constructs a complete technical system of "offline lightweight adaptation - online end-to-end execution - closed-loop self-learning iteration", which truly realizes the upgrade of industrial robots from "fixed execution" to "intelligent autonomy", and provides efficient, low-cost and sustainable intelligent solutions for various types of industrial operations in flexible manufacturing scenarios.

[0193] Therefore, the key point of this invention is:

[0194] Key Point 1: A three-stage cross-task industrial robot flexible operation framework based on VLA multimodal large model: "offline training - online execution - continuous learning".

[0195] This invention innovatively proposes a three-stage end-to-end flexible operation architecture, which includes three core components: 1) Offline training stage: Based on industrial multimodal datasets and LoRA lightweight fine-tuning, an industrial-adaptive VLA model with five-dimensional autonomous decision-making capabilities is obtained; 2) Online application stage: Integrating RGB, depth, and six-dimensional force perception, the VLA model parses natural language commands, autonomously outputs full-dimensional decisions, and completes multi-sensor collaborative operation without requiring separate programming for each task; 3) Continuous learning stage: Through a dynamic sample experience pool and incremental fine-tuning, the model is updated online and cross-task knowledge is transferred, supporting continuous performance iteration.

[0196] Key Point Two: VLA-driven five-dimensional autonomous decision-making and multi-sensor collaborative operation execution mechanism:

[0197] This invention innovatively proposes an end-to-end decision-making and execution mechanism applicable to various types of industrial operations. The core implementation logic is as follows: Based on a pre-trained VLA multimodal large model, it autonomously outputs structured decision results across five dimensions—hardware combination configuration, multimodal perception strategy, task execution process, data output type, and key operating parameters—based on input natural language commands and real-time scene perception data. Based on these decision results, it automatically generates a structured motion plan executable by the robotic arm, achieves trajectory smoothing through fifth-order polynomial interpolation, and combines force / position hybrid compliant control with multi-sensor spatiotemporal synchronous calibration to complete precise and stable execution of various types of industrial operations. This mechanism breaks through the limitations of traditional task-specific customized programming, realizing an end-to-end closed loop of "perception-decision-execution," significantly improving cross-task adaptation efficiency and operational autonomy.

[0198] Key Point 3: A cross-task closed-loop self-learning method based on dynamic sample experience pooling and low-rank adaptive incremental fine-tuning:

[0199] This invention innovatively proposes a model continuous evolution mechanism suitable for industrial scenarios. Through phased data accumulation and lightweight iteration, it adapts to the long-term operational needs of industrial sites. Specifically, it is implemented as follows: During the operation phase, the entire process data of all types of tasks is automatically recorded. A sample weighting screening mechanism is introduced to prioritize the retention of high-value samples, constructing a dynamic sample experience pool for categorized management. During the iteration update phase, a sample accumulation trigger threshold is set. When the sample size reaches the threshold, a low-rank adaptive algorithm is used to perform lightweight incremental fine-tuning based on newly added samples. Combined with a knowledge retention regularization term to suppress catastrophic forgetting, the model is smoothly replaced and deployed after verification. This mechanism achieves online model iteration and cross-task knowledge transfer at extremely low computational cost. It can effectively adapt to the long-term needs of production line condition drift, new workpieces, and new tasks, ensuring the performance stability and continuous evolution capability of the system throughout its entire lifecycle.

[0200] In this embodiment of the invention, a method for flexible cross-task industrial robot operation based on a VLA multimodal large model is provided. The method involves acquiring raw multi-source industrial data and preprocessing it to obtain a structured multimodal industrial operation dataset. The structured multimodal industrial operation dataset is then processed by parameter discretization and multi-task dialogue pair sample construction, outputting a standardized multi-task dialogue pair training sample set. This standardized multi-task dialogue pair training sample set is used to perform offline lightweight fine-tuning of a general vision-language-action base model, outputting an industrial-adaptive VLA basic model. Multimodal perception data and user-input natural language operation instructions are acquired, and the RGB images, depth data, and six-dimensional force perception data in the multimodal perception data are time-stamped and aligned, outputting time-aligned multimodal perception data. Based on a preset hand-eye homogeneous transformation matrix, the time-aligned multimodal perception data is spatially aligned and then combined with natural language operation instructions for cross-modal feature fusion, outputting cross-modal fusion features and a structured task definition vector. Finally, the industrial-adaptive VLA basic model is used based on the preset hand-eye homogeneous transformation matrix, cross-modal fusion features, and structured task definition vector. This invention utilizes task definition vectors for end-to-end autonomous decision-making and job execution control, outputting the final job results. Based on this approach, the invention injects the task logic, perception rules, and execution strategies of multiple industrial jobs into a single VLA model during the offline fine-tuning stage. This allows a single model to cover the decision-making needs of different job tasks, breaking the limitations of traditional solutions where different tasks require customized models and knowledge systems are fragmented and unusable. When switching tasks online, there is no need to re-collect data, retrain models, or write control programs for new tasks. Only the corresponding natural language job instructions need to be input, and the model can rely on its learned cross-task decision-making capabilities, combined with spatiotemporally aligned multimodal perception data, to complete scene understanding and autonomously generate job solutions, directly driving the execution end to complete the job. This achieves efficient transfer and reuse of model capabilities across different job tasks. At the same time, the standardized multimodal spatiotemporal alignment and cross-modal feature fusion process ensures the consistency of perception input and inference links under different task scenarios, eliminating the need to reconstruct perception and decision-making links during task switching. This enables rapid response to the task change requirements of the production line and significantly improves the cross-task flexible adaptability of industrial robots.

[0201] Please see Figure 4 , Figure 4 This is a structural block diagram of a cross-task industrial robot flexible operation system based on a VLA multimodal large model, provided in Embodiment 2 of the present invention.

[0202] This invention provides a cross-task industrial robot flexible operation system based on a VLA multimodal large model, comprising:

[0203] The acquisition module 401 is used to acquire raw multi-source industrial data and preprocess the raw multi-source industrial data to obtain a structured multimodal industrial operation dataset.

[0204] Module 402 is used to perform parameter discretization and multi-task dialogue pair sample construction processing on the structured multimodal industrial operation dataset, and output a standardized multi-task dialogue pair training sample set.

[0205] The fine-tuning module 403 is used to perform offline lightweight fine-tuning of the general vision-language-action base model using a standardized multi-task dialogue training sample set, and output an industrial-adaptive VLA base model.

[0206] Alignment module 404 is used to acquire multimodal perception data and natural language operation instructions input by the user, and to perform time stamp synchronization alignment on the RGB image, depth data and six-dimensional force perception data in the multimodal perception data, and output time-aligned multimodal perception data;

[0207] The output module 405 is used to perform spatial alignment on the time-aligned multimodal perception data based on the preset hand-eye homogeneous transformation matrix, and then perform cross-modal feature fusion in combination with natural language task instructions to output cross-modal fusion features and structured task definition vectors.

[0208] The task module 406 is used to perform end-to-end autonomous decision-making and task execution control based on the industrial-adapted VLA basic model, the preset hand-eye homogeneous transformation matrix, cross-modal fusion features and structured task definition vectors, and output the final task results.

[0209] Optionally, it also includes:

[0210] The first module is used to acquire the entire process data package of the operation, extract multi-dimensional operation data from the entire process data package, and encapsulate it in a structured manner according to a unified format to obtain multiple structured samples.

[0211] The second module is used to calculate the comprehensive weight for multiple structured samples and attach labels to them, so as to obtain multiple structured job samples with weight labels.

[0212] The third module is used to classify, store, and standardize the index of multiple structured operation samples with weight labels, and to prioritize the retention of high-value samples and filter low-value duplicate samples based on sample weight, so as to obtain a dynamic sample experience pool that can be dynamically expanded.

[0213] The fourth module is used to perform lightweight training on the industrial-adapted VLA basic model based on the newly added samples in the dynamic sample experience pool when the effective sample size in the dynamic sample experience pool reaches the preset trigger threshold. The training process introduces a knowledge retention regularization term to suppress catastrophic forgetting and obtains the incrementally updated VLA candidate model.

[0214] The fifth module is used to verify the incrementally updated VLA candidate models and obtain a new industrial-adaptive VLA base model.

[0215] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0216] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the cross-task industrial robot flexible operation method based on VLA multimodal large model as described in the above embodiments.

[0217] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the cross-task industrial robot flexible operation method based on VLA multimodal large model as described in the above embodiments.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0219] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0220] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0221] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0222] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-task flexible operation method for industrial robots based on a VLA multimodal large model, characterized in that, include: Acquire raw multi-source industrial data and preprocess the raw multi-source industrial data to obtain a structured multimodal industrial operation dataset. The structured multimodal industrial operation dataset is subjected to parameter discretization and multi-task dialogue pair sample construction processing to output a standardized multi-task dialogue pair training sample set. The standardized multi-task dialogue is used to perform offline lightweight fine-tuning of the general vision-language-action base model on the training sample set, and output an industrial-adaptive VLA base model. Acquire multimodal perception data and natural language task instructions input by the user, and perform time-stamp synchronization and alignment of RGB images, depth data and six-dimensional force perception data in the multimodal perception data, and output time-aligned multimodal perception data; Based on a preset hand-eye homogeneous transformation matrix, the time-aligned multimodal perception data is spatially aligned and then combined with the natural language task instructions to perform cross-modal feature fusion, outputting cross-modal fusion features and structured task definition vectors. The industrial-adaptive VLA basic model is used to perform end-to-end autonomous decision-making and job execution control based on the preset hand-eye homogeneous transformation matrix, the cross-modal fusion features and the structured task definition vector, and outputs the final job results.

2. The method for flexible cross-task industrial robot operation based on VLA multimodal large model according to claim 1, characterized in that, The preprocessing of the raw, multi-source industrial data to obtain a structured, multimodal industrial operation dataset includes: The original multi-source industrial data is classified and aggregated according to task type and data dimension to obtain classified and aggregated multi-source industrial data. The classified and aggregated industrial multi-source data are sequentially subjected to format validation and dimension alignment to obtain a structured multimodal industrial operation dataset.

3. The method for flexible cross-task industrial robot operation based on VLA multimodal large model according to claim 1, characterized in that, The structured multimodal industrial operation dataset includes multiple multimodal labeled samples covering various industrial operation scenarios; the parameter discretization and multi-task dialogue pair sample construction processing of the structured multimodal industrial operation dataset outputs a standardized multi-task dialogue pair training sample set, including: Semantic discrete encoding is performed on the continuous physical annotation values ​​in each multimodal annotation sample to obtain the discrete key operation parameter text corresponding to each multimodal annotation sample; The scene image and natural language instruction text in each of the multimodal labeled samples are used as the model input data, and the hardware scheduling annotation, perception strategy annotation, execution process annotation, output type annotation and discretization key operation parameter text in the corresponding multimodal labeled samples are used as the model output data. The model input data and model output data corresponding to each of the multimodal labeled samples are matched to obtain a standardized multi-task dialogue pair training sample set.

4. The method for flexible cross-task industrial robot operation based on VLA multimodal large model according to claim 1, characterized in that, The standardized multi-task dialogue training sample set is used to perform offline lightweight fine-tuning of the general vision-language-action base model, outputting an industrially adapted VLA base model, including: Freeze the backbone weights of the general vision-language-action base model, insert trainable low-rank matrix pairs into the attention layer weight matrix of the general vision-language-action base model, and use the standardized multi-task dialogue pair training sample set as the training input of the general vision-language-action base model to carry out supervised training, thereby obtaining an industrial-adaptive VLA base model.

5. The method for flexible operation of cross-task industrial robots based on a VLA multimodal large model according to claim 1, characterized in that, The process involves spatially aligning the time-aligned multimodal perception data based on a preset hand-eye homogeneous transformation matrix, then combining this data with the natural language task instructions to perform cross-modal feature fusion, outputting cross-modal fusion features and a structured task definition vector, including: Based on the preset hand-eye homogeneous transformation matrix, the time-aligned multimodal perception data is spatially aligned to obtain spatiotemporally unified multimodal perception data. Semantic embedding calculations are performed on the natural language task instructions to obtain text feature vectors; Visual encoding calculations are performed on the RGB images and depth data in the spatiotemporally unified multimodal perception data to obtain visual feature vectors; Temporal feature encoding is performed on the six-dimensional force perception data in the spatiotemporally unified multimodal sensing data to obtain a force perception feature vector; Multimodal cross-modal attention-weighted fusion calculation is performed on the text feature vector, the visual feature vector, and the force feature vector to obtain cross-modal fusion features; Based on the cross-modal fusion features, a structured task definition vector is constructed.

6. The method for flexible operation of cross-task industrial robots based on a VLA multimodal large model according to claim 1, characterized in that, The industrial-adaptive VLA basic model performs end-to-end autonomous decision-making and job execution control based on the preset hand-eye homogeneous transformation matrix, the cross-modal fusion features, and the structured task definition vector, outputting the final job results, including: The cross-modal fusion features and the structured task definition vector are input into the industrial-adaptive VLA base model to perform inference and generate a multi-dimensional structured decision vector. The multidimensional structured decision vector is decomposed to obtain a discrete motion waypoint sequence; The discrete motion waypoint sequence is subjected to trajectory smoothing calculation to obtain a smooth motion trajectory sequence; Based on the smooth motion trajectory sequence and real-time multidimensional force data, compliance correction calculation and trajectory superposition are performed to output real-time motion control commands. The industrial robot's end effector is driven by the real-time motion control commands to complete the corresponding operation. During the execution, based on the preset hand-eye homogeneous transformation matrix and real-time joint pose data, multi-sensor dynamic calibration is completed by minimizing the reprojection error to generate the final operation result.

7. The method for flexible cross-task industrial robot operation based on VLA multimodal large model according to claim 1, characterized in that, Also includes: Obtain the entire process data package, extract multi-dimensional operation data from the entire process data package and encapsulate it in a structured manner according to a unified format to obtain multiple structured samples; Calculate a comprehensive weight for multiple structured samples and attach labels to them to obtain multiple structured job samples with weighted labels; Multiple structured job samples with weighted labels are classified, stored, and indexed in a standardized manner. Based on the sample weight, high-value samples are retained first and low-value duplicate samples are filtered out to obtain a dynamically expandable dynamic sample experience pool. When the effective sample size of the dynamic sample experience pool reaches a preset trigger threshold, the industrial-adaptive VLA basic model is trained on the newly added samples in the dynamic sample experience pool. The training process introduces a knowledge retention regularization term to suppress catastrophic forgetting, and the incrementally updated VLA candidate model is obtained. The incrementally updated VLA candidate model is validated to obtain a new industrial-adaptive VLA base model.

8. A flexible cross-task industrial robot operation system based on a VLA multimodal large model, characterized in that, include: The acquisition module is used to acquire raw multi-source industrial data and preprocess the raw multi-source industrial data to obtain a structured multimodal industrial operation dataset. The construction module is used to perform parameter discretization and multi-task dialogue pair sample construction processing on the structured multimodal industrial operation dataset, and output a standardized multi-task dialogue pair training sample set. The fine-tuning module is used to perform offline lightweight fine-tuning of the general vision-language-action base model using the standardized multi-task dialogue training sample set, and output an industrial-adaptive VLA base model. The alignment module is used to acquire multimodal perception data and natural language task instructions input by the user, and to perform time stamp synchronization alignment on the RGB image, depth data and six-dimensional force perception data in the multimodal perception data, and output time-aligned multimodal perception data. The output module is used to perform spatial alignment on the time-aligned multimodal perception data based on a preset hand-eye homogeneous transformation matrix, and then perform cross-modal feature fusion in combination with the natural language task instructions to output cross-modal fusion features and structured task definition vectors. The task module is used to perform end-to-end autonomous decision-making and task execution control based on the industrial-adaptive VLA basic model, the preset hand-eye homogeneous transformation matrix, the cross-modal fusion features, and the structured task definition vector, and output the final task result.

9. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the cross-task industrial robot flexible operation method based on VLA multimodal large model as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the cross-task industrial robot flexible operation method based on VLA multimodal large model as described in any one of claims 1-7.

Images