Method and system for autonomous operation of an archival robot driven by visual language action model

Abstract

The application belongs to the technical field of archive management robots, and provides an archive robot autonomous operation method and system driven by a visual language action model, wherein, according to different specifications of archive features and voice instructions, an action track, a grabbing point and force that are adapted to targets are generated in the trained visual language action model, the archive box, bagged archives and scattered documents and other different specifications of archives and voice instructions can be used to generate adapted fine action tracks, grabbing points and forces, the problem of unstable clamping and action failure caused by the diversity of archive specifications and forms in the traditional scheme is overcome, the maximum opening angle limit of the clamping jaw and the physical constraint of the mechanical arm movement range are added in the action planning, the force and speed parameters within the safe range are output while the action sequence is ensured, the archive material damage caused by the robot misoperation is avoided, the physical risk to the archive body in the operation is significantly reduced, and the archive grabbing reliability is further improved.

Description

Technical Field

[0001] This invention belongs to the field of archival management robot technology, and in particular relates to an autonomous operation method and system for archival robots driven by a visual language action model. Background Technology

[0002] With the rapid advancement of artificial intelligence and robotics, archive storage robots are being widely used, making the demand for dexterous robot control and motion intelligence increasingly urgent.

[0003] Current archival management robots often rely on fixed trajectory planning or pre-calibrated grasping schemes, and mostly use single vision guidance. They lack multimodal fusion processing and cross-modal precise alignment mechanisms for voice commands, and cannot combine language commands such as gentle handling and specific location instructions for fine-grained control. In addition, the size, weight, and hardness of archival media in actual warehouses vary, and the packaging is not standardized. Traditional control programs are difficult to adapt to grasping archives of different sizes and shapes, which can easily lead to unstable grasping, damage to archival materials, or operation failure. Archival equipment is mostly standardized mobile shelving or rotary cabinets, which often have narrow working surfaces, uneven lighting, and visual obstruction problems. Especially for archives stored deep in the storage space, the robot's positioning and grasping are constrained by physical space, making path planning and robotic arm posture adjustment difficult. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a method and system for autonomous operation of an archival robot driven by a visual language action model. This invention overcomes the problems of unstable gripping and action failure caused by the diversity of archival specifications and shapes in traditional solutions, avoids damage to archival materials due to robot misoperation, significantly reduces the physical risk to the archival body during operation, and improves the reliability of archival grasping.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the present invention provides a method for autonomous operation of an archival robot driven by a visual language action model, comprising: Based on the visual, linguistic, and motion data related to the document management robot, annotated multimodal triplet data is generated; Based on multimodal triplet data, in the trained visual-language-action model, the action trajectory, grasping point, and force of the target are generated according to the file features of different specifications and voice commands. Specifically, the visual encoder extracts grasping-related visual features and navigation-related visual features respectively, and the language encoder converts text and voice commands into language feature vectors. Based on the language feature vectors, a high-level semantic inferencer generates a reference action sequence, and a low-level action inferencer directly acts on the key-value cache to obtain their respective action features. The action features are fused into action prediction to obtain the action sequence. The maximum opening angle limit of the gripper and the physical constraints of the robotic arm's range of motion are added to the action planning. The file management robot is controlled based on the obtained action sequence.

[0006] Furthermore, the visual data includes capture-related visual data and navigation-related visual data.

[0007] Furthermore, the visual data capture-related annotations include the archive QR code, archive storage location coordinates, archive size, archive size specifications, and archive capture point; the navigation-related visual data navigation-related annotations include the archive equipment number, archive equipment layer, warehouse obstacle location, and ground navigation path nodes.

[0008] Furthermore, the visual encoder uses the MobileNetV3 network as its basic backbone and introduces a residual connection mechanism to obtain shallow features and deep features, which are used as inputs to the grasping visual branch network and the deep features as inputs to the navigation branch network, respectively; both the grasping visual branch network and the navigation branch network include convolutional layers and fully connected layers.

[0009] Furthermore, the language encoder uses BERT-base as its backbone network.

[0010] Furthermore, based on image feature vectors and language feature vectors, a unified semantic representation space for vision and language is established, historical key vectors and value vectors are stored, and key vectors and value vectors are mapped to a lower-dimensional space.

[0011] Furthermore, based on the key vector K' and value vector V' mapped to a lower-dimensional space, and the noisy state sequence, cross-attention calculation is performed on the vector and value vectors. Multimodal contextual prior information is injected using the cross-attention mechanism. Then, the intermediate representations are processed in parallel with residuals through depthwise separable convolution to update the output features of the high-level semantic inference engine. After training with the stream matching algorithm, the denoised action sequence is obtained.

[0012] Furthermore, the step of fusing the action features obtained from the high-level semantic inferencer and the low-level action inferencer into action prediction includes: encoding the noisy action sequence into an action query vector using projection; and generating action features Z using the high-level semantic inferencer. HL The low-level action inferencer generates action features Z. LL Cross-attention calculations are performed with the query vector to obtain high-level semantic guidance features and low-level action guidance features; the high-level semantic guidance features and low-level action guidance features are then concatenated to obtain the fused features.

[0013] Furthermore, a stream matching objective optimization strategy is adopted for the specific training of the framework, with a weighted joint optimization loss function, including a high-level semantic action inferencer. Flow matching mean square error and action head The mean squared error of stream matching is used to balance the training intensity of the two modules through weight coefficients, allowing the high-level semantic inferencer and the action head model to be co-optimized; the loss function is: ; Physical constraint regularization term Defined as: ; in, These are preset parameters. and Preset weights; d For the action vector dimension; , For the first i Hard threshold for each action.

[0014] Secondly, the present invention also provides an autonomous operating system for an archival robot driven by a visual language action model, comprising: The data acquisition module is configured to generate labeled multimodal triplet data based on visual data, language data, and motion data related to the file management robot. Based on multimodal triplet data, in the trained visual-language-action model, the action trajectory, grasping point, and force of the target are generated according to the file features of different specifications and voice commands. Specifically, the visual encoder extracts grasping-related visual features and navigation-related visual features respectively, and the language encoder converts text and voice commands into language feature vectors. Based on the language feature vectors, a high-level semantic inferencer generates a reference action sequence, and a low-level action inferencer directly acts on the key-value cache to obtain their respective action features. The action features are fused into action prediction to obtain the action sequence. The maximum opening angle limit of the gripper and the physical constraints of the robotic arm's range of motion are added to the action planning. Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual language action model-driven autonomous operation method for an archival robot described in the first aspect.

[0015] Fourthly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the steps of the visual language action model-driven autonomous operation method for an archival robot described in the first aspect.

[0016] Fifthly, the present invention also provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the steps of the visual language action model-driven autonomous operation method for archival robots described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention innovatively proposes a visual language motion model-driven autonomous operation method for archival robots. By generating adaptive motion trajectories, grasping points, and forces based on the characteristics of archives of different specifications and voice commands in the trained visual language motion model, it can generate suitable fine motion trajectories, grasping points, and forces according to different specifications of archives such as archive boxes, bagged archives, and scattered documents, and voice commands. This overcomes the problems of unstable gripping and motion failure caused by the diversity of archive specifications and shapes in traditional solutions, and improves the reliability of archive grasping in unstructured environments. By adding the maximum opening angle limit of the gripper and the physical constraints of the robotic arm's range of motion to the motion planning, while ensuring the completion of the motion sequence, it outputs force and speed parameters within a safe range, avoiding damage to archive materials due to robot misoperation, significantly reducing the physical risk to the archive body during operation, and further improving the reliability of archive grasping.

[0018] 2. This invention innovatively proposes a visual language action model-driven autonomous operation method for archival robots. It designs a dual-module differentiated weighted joint loss function, combining physical constraint regularization terms with flow matching loss. By setting relaxed constraint weights for the high-level semantic inferencer (ensuring the flexibility of high-level action planning) and strict constraint weights for the low-level action inferencer (ensuring the hardware compliance of low-level action execution), the model is forced to output a continuous action sequence that conforms to the hardware limits of the archival robot from the perspective of loss optimization. This completely solves the problems of motion distortion, mechanical collision, and jamming in the migration from simulation to real machine, and significantly improves the effectiveness and safety of action execution.

[0019] 3. This invention innovatively proposes a hybrid reasoning VLA model architecture for archive storage scenarios, breaking through the limitations of the single reasoning mode of traditional VLA models. The high-level semantic inferencer realizes high-level action sequence planning and logical interpretation, while the low-level action inferencer realizes low-level action real-time fine-tuning, taking into account both the logic of action planning and the real-time performance of action execution. It breaks the limitations of the fragmented workflow of traditional archive robot operations, realizing end-to-end mapping from visual perception and language understanding to action generation, and can complete the entire process of archive storage, retrieval, inventory, and transportation autonomously without manual intervention.

[0020] 4. This invention innovatively proposes an autonomous operation method for archival robots driven by a visual language action model. It designs a visual encoder and adopts MobileNetV3 network as the basic backbone. By introducing a residual connection mechanism, it achieves the goal of effectively extracting basic features of visual images at multiple scales and solves the gradient vanishing problem of deep networks.

[0021] 5. This invention innovatively proposes a method for autonomous operation of an archive robot driven by a visual language action model. It designs a language encoder and uses BERT-base as the backbone network. Through improvements and lightweight pruning of the backbone network, it achieves the goal of outputting fixed high-dimensional language feature vectors, thus solving the problems of high model parameter quantity and computational complexity while ensuring semantic understanding ability. Attached Figure Description

[0022] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0023] Figure 1 This is a VLA model framework diagram of Embodiment 1 of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0026] The VLA model, or Vision-Language-Action model, is a multimodal model that integrates visual perception, language understanding, and action generation capabilities, providing integrated technical support for perception, decision-making, and execution for embodied intelligent robots.

[0027] Embodied intelligence: It gives robots the ability to interact autonomously and collaborate adaptively with the physical world, realizing the integration of perception, decision-making and execution, and breaking through the limitations of traditional robots' passive execution.

[0028] Flow matching algorithm: an optimization algorithm for modeling continuous motion sequences. It achieves accurate generation and optimization of continuous motion by learning the continuous flow field from noise distribution to real motion distribution.

[0029] Key-value caching technology: Stores the key vector K and value vector V of historical tokens during model inference to avoid redundant calculations and improve model inference speed.

[0030] Canonical correlation analysis (CCA): A multivariate statistical analysis method used to map high-dimensional features to a low-dimensional space, reducing computational complexity while maintaining feature correlation.

[0031] Example 1: With the rapid advancement of artificial intelligence and robotics, robots have been widely applied in various fields such as manufacturing, medical services, and home life, making the demand for dexterous manipulation and intelligent movement of robots increasingly urgent. Traditional archival robots are mostly limited to single functions such as storage, retrieval, transportation, or inventory. Their work processes are fragmented, and they often use fixed programs to execute tasks, requiring manual coordination of each step. They are also susceptible to operational deviations due to environmental interference. In recent years, embodied intelligence technology has endowed robots with integrated capabilities of perception, decision-making, and execution, enabling autonomous interaction and adaptive collaboration between robots and the physical world. Compared to traditional archival robots, embodied intelligent archival management robots break through the limitations of traditional equipment's "single function, passive execution, and disconnected operation," significantly improving operational efficiency and intelligent adaptability.

[0032] As described in the background section, existing document management robots often rely on fixed trajectory planning or pre-defined grasping schemes, and mostly employ single visual guidance. They lack multimodal fusion processing and cross-modal precise alignment mechanisms for voice commands, making it impossible to perform fine-grained manipulation by combining language instructions (such as "handle with care" or "take the third box on the left of folder A"). Furthermore, existing methods do not fully utilize the procedural and structured operational characteristics of document circulation processes such as "grasping-placement-transportation," but only generate discrete outputs for actions. This limits the stability and accuracy of document robots in complex operations (such as collaborative operations of multiple types of documents, dynamic target positioning, and long-sequence task execution).

[0033] Robotic operations in archive storage often require the handling of archives. However, the actual archive media (such as boxed documents, files, drawings, and photographs) vary in size, weight, and hardness, and their packaging is often non-standard (archive boxes may be of inconsistent sizes, archive materials may be loose, and archive bags may not be properly sealed), introducing significant variables to the robot's dexterity. Traditional control programs struggle to adapt to the handling of archives of different sizes and shapes, easily leading to unstable gripping, damage to archive materials, or operational failures. Archive storage is mostly standardized mobile shelving / racks, which often presents challenges such as narrow working surfaces, uneven lighting, and visual obstruction. This is especially true for archives stored deep within the archive, where the robot's positioning and handling are constrained by physical space, making path planning and robotic arm posture adjustment difficult. During operation, it is crucial to strictly control the gripping force and movement trajectory to avoid physical damage to archive materials (such as paper tearing or archive bag delamination) due to mechanical impact or excessive gripping force. Furthermore, the archives themselves may be fragile (such as ancient books or faded paper), requiring a high success rate and zero damage rate during operations.

[0034] To address the aforementioned issues, this embodiment provides a method for autonomous operation of an archival robot driven by a visual language action model, comprising: S1. Data Acquisition and Processing: S1.1 Data Acquisition: S1.1.1 Simulation data: Based on the simulated environment of the archive storage, the system automatically generates triplet data corresponding to different types of archive equipment, archives of different sizes, and different instructions, generating 100,000 simulated sample data, thus solving the problem of high cost of real data collection.

[0035] S1.1.2, Manual collection: In a real archive storage environment, operators remotely control the archive robot to perform operations such as picking up, removing, putting on and transporting archives, while simultaneously collecting images / videos, voice / text commands and robot motion parameters.

[0036] For example, the collected data sample is as follows: Vision: File A on the second shelf of file cabinet No. 3 + Language: Take out file A from the second shelf of file cabinet No. 3 + Action: The robot chassis moves to file cabinet No. 3 → The robotic arm lifts to the second shelf → The end vision sensor aligns with file A → The dexterous hand picks it up → It is placed in the temporary storage box → The robot chassis moves to the file retrieval point.

[0037] S1.2, Data Labeling: The annotations for the archive storage scenario include two categories of visual data: capture-related visual data and navigation-related visual data. Capture-related annotations include archive QR codes, archive storage location coordinates, archive dimensions / specifications, and key points for archive retrieval. Navigation-related annotations include archive equipment numbers / layers, storage obstacle locations, and ground navigation path nodes. Language data annotations include instruction types (retrieve, place, transport, inventory, etc.), operation objects, target locations, and other auxiliary instructions (such as handle with care, prioritize transport, etc.). Motion data includes movement coordinates, chassis motor speed, robotic arm joint angles, dexterous hand gripping opening and closing degrees, motion execution sequence, and operation success rate. Key nodes and threshold parameters for motion execution are also annotated (such as robotic arm lifting height threshold and dexterous hand gripping force threshold).

[0038] S1.3, Data Processing: The raw data is filtered to remove invalid samples, improving dataset quality. Samples with blurred visual data, out-of-focus cameras, or completely occluded targets are removed, as are samples with abnormal motion parameters (such as joint angles exceeding the robot's range of motion, negative gripper opening / closing angles), or interrupted data recording. Failed samples are retained with added error explanations, allowing the model to learn and avoid grasping errors. Motion parameters are normalized by mapping robot joint angles, chassis movement coordinates, motor speeds, and other parameters to a uniform numerical range, eliminating the impact of dimensional differences on model training.

[0039] S2, such as Figure 1 As shown, the improved VLA model and training method are as follows: The development of Vision-Language-Action (VLA) models has become an important direction in motion modeling research for embodied intelligent robots. VLA models are typically built upon pre-trained multimodal large language models, with the addition of an action output head or a dedicated action module to generate actions. Multimodal large language models excel in perception and decision-making, enabling VLA models to demonstrate good generalization ability across various robotic tasks.

[0040] However, this model has significant limitations: First, it often lacks a deep understanding of the actions themselves. In existing methods, actions are treated merely as outputs, not as inputs for analysis and modeling. This inherently limits its ability to convey the complete, fine-grained information required for precise action execution, making it unable to transmit the complete, fine-grained motion information needed for the precise action execution of the archival robot. Consequently, it struggles to adapt to the refined operational needs of tasks such as file retrieval, precise shelving / removal, and narrow-space navigation. Second, model inference requires complex multimodal processing and reliance on large-scale resources. Inference latency leads to significant response delays, impacting operational efficiency. Third, existing VLA models are general-purpose models that do not incorporate the specific characteristics of the archival storage environment (such as standardized file equipment, fixed storage locations, and streamlined operational procedures) for feature extraction and model architecture design. This results in significant deficiencies in the operational adaptability and action accuracy of the archival robot.

[0041] S2.1, Visual Encoder: The core function is to extract visual features related to grasping (target location, size) and navigation (environmental layout, obstacle location, path nodes) respectively, and transform them into high-dimensional visual feature vectors to provide a basis for subsequent grasping pose decisions and navigation path planning. A dual-branch parallel design is adopted, sharing some backbone network parameters to effectively reduce the number of model parameters, while also performing differentiated optimizations for the characteristics of the two types of tasks. The visual encoder consists of a backbone network and branch networks, and outputs image feature vectors.

[0042] The lightweight MobileNetV3 network is used as the basic backbone. The residual connection mechanism is introduced to solve the gradient vanishing problem of deep networks, effectively extracting multi-scale basic features of visual images and outputting multi-scale feature map C3 (shallow features, containing fine-grained target information) and feature map C5 (deep features, containing global scene semantic information).

[0043] Feature map C3 serves as the input to the visual branch network for object capture. This branch network consists of four convolutional layers and two fully connected layers, focusing on fine-grained target features. The specific network structure is a 1×1 convolutional layer, a 3×3 convolutional layer, a spatial attention module, a 1×1 convolutional layer, a fully connected layer 1, and a fully connected layer 2. It detects document targets, including bounding boxes and keypoints, ultimately achieving bounding box detection and keypoint detection for document targets.

[0044] Feature map C5 serves as the input to the navigation branch network, which consists of two convolutional layers and two fully connected layers, focusing on global scene semantics. The specific network structure comprises a 3×3 convolutional layer, a 1×1 convolutional layer, a fully connected layer 1, and a fully connected layer 2. Ultimately, it extracts global semantic features such as warehouse environment layout, obstacle locations, and path nodes.

[0045] S2.2, Language Encoder: The core function is to convert text and voice commands into high-dimensional language feature vectors, capturing the task semantics (target file, task requirements, destination, additional requirements, etc.) in the commands, allowing the model to simultaneously understand "where to go, what task to perform, and what requirements to achieve". The encoder uses BERT-base as the backbone network and performs lightweight pruning to reduce the number of model parameters and computational complexity while ensuring semantic understanding capabilities, ultimately outputting fixed-dimensional high-dimensional language feature vectors.

[0046] S2.3, Visual-Language Model: Image-language features output from the visual encoder and speech encoder are used as input to a multimodal large-scale model to unify the visual-language semantic representation space. Simultaneously, optimization techniques are introduced to improve model inference speed and address the inference latency issue of traditional VLA models. Key-value caching is employed to store the key vectors of historical tokens. K Sum value vector V To avoid redundant calculations of the same features, and to further improve inference speed, canonical correlation analysis (CCA) is used to map the key vector K and value vector V to a lower-dimensional space, resulting in a feature dimension of 256, calculated using the following formula.

[0047] K ' =a T K ; V ' =b T V ; in, K 'and V ' represents the mapped key vector and value vector; a , b This is a vector of typical coefficients.

[0048] S2.4, Action Decoding Module: The action decoding module adopts a hybrid reasoning architecture of high-level semantics and low-level actions, which takes into account both the logic of high-level task planning and the real-time performance of low-level action fine-tuning.

[0049] S2.4.1 The high-level semantic inferencer is used to generate reference action sequences, with the input being the key vector of the visual language model. K 'Sum value vector V ', and the state sequence with added noise Input the transformer model together. H The reference action sequence length is set to 10, and the model simultaneously performs vector calculations during the computation. K 'Sum value vector V Cross-attention calculation, in the firsti Each transformer layer outputs a reference representation. for: ; in, for i Output representation of the -1 sequence; For the first i Transformer layer key vectors; No. i A vector of transformer layer values.

[0050] Here, a cross-attention mechanism is used to inject multimodal contextual prior information from the vision-language model. Then, the intermediate representations are processed in residual parallelism through depthwise separable convolution to update the output features of the i-th layer high-level semantic inference: ; in, This serves as an intermediate representation.

[0051] After further training using a flow matching algorithm, It can learn the distribution pattern of motion trajectories and obtain the denoised motion sequence.

[0052] ; in, This is a high-level semantic action reasoner model; , For full-layer context key-value (KV) caching of pre-trained vision-language models; N This represents the number of layers in the visual-language model, with a value of 10.

[0053] For the one-dimensional temporal structure of action sequences, a single 1D convolutional layer (kernel size set to 3, stride of 1) is used, employing batch normalization and ReLU activation function. Finally, global average pooling (GAP) is applied to output fixed-dimensional embedding features. Z HL .

[0054] S2.4.2, The low-level action inferencer directly operates on the key-value cache output by the visual-language model. It queries the ontology state matrix. Q Key vector K 'Sum value vector V 'The cross-attention mechanism extracts action-related information from the key values. The resulting features are then integrated using average pooling to obtain 256-dimensional action features. These features are then transformed using a single 1D convolutional layer to obtain representational features that capture the semantic meaning of low-level actions.'Z LL .

[0055] S2.5, Action Prediction Module: The above steps generate action features based on a high-level semantic inference engine. Z HL Generate action features with low-level action inferencer Z LL This integrates two types of action guidance information into action prediction.

[0056] S2.5.1 First, use MLP projection to transform the noisy action sequence. Encoded as an action query vector .

[0057] S2.5.2, then use Z HL and Z LL respectively with query vector Perform cross-attention calculation to obtain cross-attention output features. and The calculation formula is: ; ; S2.5.3 In order to achieve two complementary guiding features, high-level semantic guiding features will be used. and low-level action guidance features By concatenating and stitching features together, we obtain the fused features: ; S2.5.4. A two-layer MLP is used to transform the concatenated features to obtain the final output features. S final Input action head It predicts the denoised action sequence. .

[0058] S2.6 Training process: The flow matching algorithm is adopted as the basic optimization strategy. By learning the continuous flow field from the distribution of noisy action sequences to the distribution of real archival operation action sequences, the model can master the trajectory rules and fine-grained features of archival operation actions, realize the accurate generation and denoising of continuous action sequences, avoid the accuracy loss problem of traditional discrete action modeling, and adapt to the needs of refined archival robot operations.

[0059] S2.6.1 Weighted Joint Optimization Loss Function: The framework is trained using a stream matching objective optimization strategy, with a weighted joint optimization loss function, including a high-level semantic action inferencer. Flow matching mean square error and action head The mean squared error of flow matching is used to balance the training intensity of the two modules through weight coefficients, allowing the high-level semantic inferencer and the action head model to be co-optimized. Due to the physical constraints of the archive robot, the loss function needs to consider action compliance. A physical constraint regularization term is added to the loss function to force the model output to conform to the hardware limits. The loss function is defined as follows: ; in, These are preset parameters. and Preset weights; optional. =0.4, =0.2, =0.5.

[0060] Physical constraint regularization term Defined as: ； in, d For the action vector dimension, , For the first i Hard thresholds for each action, such as the upper and lower limits of the closing angle of a robotic arm, and the upper and lower limits of the opening and closing angle of a robot gripper; For the first i Action parameters.

[0061] S2.6.2 Training Process: The preprocessed "visual-language-action" triplet training dataset is divided into training set, validation set, and test set in a ratio of 8:1:1.

[0062] The improved VLA model is initialized by setting hyperparameters such as network layer learning rate, batch size, and number of training epochs.

[0063] The training data is input into the model, and forward propagation is performed according to the above model architecture to obtain the output results of the high-level semantic inference engine and the action head model, respectively.

[0064] Calculate the numerical value of the weighted joint optimization loss function, and use the backpropagation algorithm to propagate the loss gradient back to each layer of the model to update the model parameters; thus obtaining the prediction model.

[0065] In each training round, the performance of the prediction model is validated using a validation set. If the validation set loss does not decrease, hyperparameters such as the learning rate are adjusted to avoid model overfitting.

[0066] After training, the prediction model is tested using a test set to verify its accuracy, real-time performance, and robustness in generating file operation actions.

[0067] This embodiment uses visual-language-action fusion modeling. The model can generate a fine motion trajectory, grasping point and force adapted to the target based on the characteristics of different types of files (such as file boxes, bagged files, and scattered documents) and voice commands (such as "take out file A from file shelf No. 3"). This overcomes the problems of unstable grasping and action failure caused by the diversity of file sizes and shapes in traditional solutions, and improves the reliability of file grasping in unstructured environments.

[0068] This embodiment utilizes a motion reasoning architecture based on flow matching and explicit / low-level motion fusion. Physical constraints (such as the maximum opening and closing angle limit of the gripper and the range of motion of the robotic arm) are added to the motion planning. While ensuring the completion of the motion sequence, the model is forced to output force and speed parameters within a safe range, avoiding damage to the archival materials due to robot misoperation and significantly reducing the physical risk to the archival body during operation.

[0069] This embodiment uses key-value caching and canonical correlation analysis to accelerate multimodal processing, reduce redundant computation and inference latency during cross-modal fusion, and achieve rapid parsing of voice commands and real-time action generation. It supports the dynamic adjustment and continuous operation of the archive robot in narrow equipment environments, and demonstrates high efficiency, especially in the turnover of large amounts of archives.

[0070] This embodiment utilizes a visual encoder branch optimized for archival storage scenarios (such as standardized archival equipment, fixed storage locations, and streamlined operational processes) and labeled data. This enables the system to better extract the semantics of the storage layout, path nodes, and detailed archival attributes, thereby improving the accuracy of positioning and path planning while reducing the cost of human intervention during multi-task switching.

[0071] In this embodiment, the high-level semantic inferencer implements task-level long-order planning, such as sequentially transferring multiple files and generating a continuous and smooth sequence of actions, while the low-level action inferencer responds to dynamic changes in the environment to achieve real-time fine-grained adjustment of actions (such as slight avoidance when encountering sudden obstacles). The combination of the two ensures the logical correctness of the process and improves the robustness and anti-interference of action execution, making it suitable for possible dynamic environmental changes during file retrieval.

[0072] This embodiment combines physical constraint regularization with flow matching loss to design a dual-module differentiated weighted joint loss function. By setting relaxed constraint weights for the high-level semantic inference engine (ensuring flexibility in high-level action planning) and strict constraint weights for the action head (ensuring hardware compliance of low-level action execution), the model is forced to output a continuous action sequence that conforms to the hardware limits of the archival robot from the perspective of loss optimization. This completely solves problems such as action distortion, mechanical collisions, and jamming during the transfer from simulation to the real machine, significantly improving the effectiveness and safety of action execution. At the same time, it retains the advantage of the flow matching algorithm in fine-grained modeling of continuous actions, achieving dual optimization of action flow field fitting accuracy and hardware physical compliance.

[0073] This embodiment proposes a hybrid reasoning VLA model architecture for archive storage scenarios. This architecture overcomes the limitations of traditional VLA models' single reasoning mode. A high-level semantic inferencer handles high-level action sequence planning and logical interpretation, while a low-level action inferencer performs real-time fine-tuning of low-level actions. This balances the logicality of action planning with the real-time performance of action execution, resolving the reasoning latency issue of traditional models. By integrating this improved VLA model into an archive management robot, the fragmented workflow of traditional archive robots is broken, achieving end-to-end mapping from visual perception and language understanding to action generation. This enables fully autonomous operations, including archive storage, retrieval, inventory, and transportation, without the need for manual intervention.

[0074] Example 2: This embodiment provides an autonomous operation system for an archival robot driven by a visual language action model, including: The data acquisition module is configured to generate labeled multimodal triplet data based on visual data, language data, and motion data related to the file management robot. Based on multimodal triplet data, in the trained visual-language-action model, the action trajectory, grasping point, and force of the target are generated according to the file features of different specifications and voice commands. Specifically, the visual encoder extracts grasping-related visual features and navigation-related visual features respectively, and the language encoder converts text and voice commands into language feature vectors. Based on the language feature vectors, a high-level semantic inferencer generates a reference action sequence, and a low-level action inferencer directly acts on the key-value cache to obtain their respective action features. The action features are fused into action prediction to obtain the action sequence. The maximum opening angle limit of the gripper and the physical constraints of the robotic arm's range of motion are added to the action planning. The control module is configured to control the file management robot based on the obtained action sequence.

[0075] The system operates in the same way as the visual language action model-driven autonomous operation method for the archival robot in Embodiment 1, including: S1. Data Acquisition and Processing: S1.1 Data Acquisition: S1.1.1 Simulation data: Based on the simulated environment of the archive storage, the system automatically generates triplet data corresponding to different types of archive equipment, archives of different sizes, and different instructions, generating 100,000 simulated sample data, thus solving the problem of high cost of real data collection.

[0076] S1.1.2, Manual collection: In a real archive storage environment, operators remotely control the archive robot to perform operations such as picking up, removing, putting on and transporting archives, while simultaneously collecting images / videos, voice / text commands and robot motion parameters.

[0077] For example, the collected data sample is as follows: Vision: File A on the second shelf of file cabinet No. 3 + Language: Take out file A from the second shelf of file cabinet No. 3 + Action: The robot chassis moves to file cabinet No. 3 → The robotic arm lifts to the second shelf → The end vision sensor aligns with file A → The dexterous hand picks it up → It is placed in the temporary storage box → The robot chassis moves to the file retrieval point.

[0078] S1.2, Data Labeling: The annotations for the archive storage scenario include two categories of visual data: capture-related visual data and navigation-related visual data. Capture-related annotations include archive QR codes, archive storage location coordinates, archive dimensions / specifications, and key points for archive retrieval. Navigation-related annotations include archive equipment numbers / layers, storage obstacle locations, and ground navigation path nodes. Language data annotations include instruction types (retrieve, place, transport, inventory, etc.), operation objects, target locations, and other auxiliary instructions (such as handle with care, prioritize transport, etc.). Motion data includes movement coordinates, chassis motor speed, robotic arm joint angles, dexterous hand gripping opening and closing degrees, motion execution sequence, and operation success rate. Key nodes and threshold parameters for motion execution are also annotated (such as robotic arm lifting height threshold and dexterous hand gripping force threshold).

[0079] S1.3, Data Processing: The raw data is filtered to remove invalid samples, improving dataset quality. Samples with blurred visual data, out-of-focus cameras, or completely occluded targets are removed, as are samples with abnormal motion parameters (such as joint angles exceeding the robot's range of motion, negative gripper opening / closing angles), or interrupted data recording. Failed samples are retained with added error explanations, allowing the model to learn and avoid grasping errors. Motion parameters are normalized by mapping robot joint angles, chassis movement coordinates, motor speeds, and other parameters to a uniform numerical range, eliminating the impact of dimensional differences on model training.

[0080] S2, such as Figure 1As shown, the improved VLA model and training method are as follows: The development of Vision-Language-Action (VLA) models has become an important direction in motion modeling research for embodied intelligent robots. VLA models are typically built upon pre-trained multimodal large language models, with the addition of an action output head or a dedicated action module to generate actions. Multimodal large language models excel in perception and decision-making, enabling VLA models to demonstrate good generalization ability across various robotic tasks.

[0081] However, this model has significant limitations: First, it often lacks a deep understanding of the actions themselves. In existing methods, actions are treated merely as outputs, not as inputs for analysis and modeling. This inherently limits its ability to convey the complete, fine-grained information required for precise action execution, making it unable to transmit the complete, fine-grained motion information needed for the precise action execution of the archival robot. Consequently, it struggles to adapt to the refined operational needs of tasks such as file retrieval, precise shelving / removal, and narrow-space navigation. Second, model inference requires complex multimodal processing and reliance on large-scale resources. Inference latency leads to significant response delays, impacting operational efficiency. Third, existing VLA models are general-purpose models that do not incorporate the specific characteristics of the archival storage environment (such as standardized file equipment, fixed storage locations, and streamlined operational procedures) for feature extraction and model architecture design. This results in significant deficiencies in the operational adaptability and action accuracy of the archival robot.

[0082] S2.1, Visual Encoder: The core function is to extract visual features related to grasping (target location, size) and navigation (environmental layout, obstacle location, path nodes) respectively, and transform them into high-dimensional visual feature vectors to provide a basis for subsequent grasping pose decisions and navigation path planning. A dual-branch parallel design is adopted, sharing some backbone network parameters to effectively reduce the number of model parameters, while also performing differentiated optimizations for the characteristics of the two types of tasks. The visual encoder consists of a backbone network and branch networks, and outputs image feature vectors.

[0083] The lightweight MobileNetV3 network is used as the basic backbone. The residual connection mechanism is introduced to solve the gradient vanishing problem of deep networks, effectively extracting multi-scale basic features of visual images and outputting multi-scale feature map C3 (shallow features, containing fine-grained target information) and feature map C5 (deep features, containing global scene semantic information).

[0084] Feature map C3 serves as the input to the visual branch network for object capture. This branch network consists of four convolutional layers and two fully connected layers, focusing on fine-grained target features. The specific network structure is a 1×1 convolutional layer, a 3×3 convolutional layer, a spatial attention module, a 1×1 convolutional layer, a fully connected layer 1, and a fully connected layer 2. It detects document targets, including bounding boxes and keypoints, ultimately achieving bounding box detection and keypoint detection for document targets.

[0085] Feature map C5 serves as the input to the navigation branch network, which consists of two convolutional layers and two fully connected layers, focusing on global scene semantics. The specific network structure comprises a 3×3 convolutional layer, a 1×1 convolutional layer, a fully connected layer 1, and a fully connected layer 2. Ultimately, it extracts global semantic features such as warehouse environment layout, obstacle locations, and path nodes.

[0086] S2.2, Language Encoder: The core function is to convert text and voice commands into high-dimensional language feature vectors, capturing the task semantics (target file, task requirements, destination, additional requirements, etc.) in the commands, allowing the model to simultaneously understand "where to go, what task to perform, and what requirements to achieve". The encoder uses BERT-base as the backbone network and performs lightweight pruning to reduce the number of model parameters and computational complexity while ensuring semantic understanding capabilities, ultimately outputting fixed-dimensional high-dimensional language feature vectors.

[0087] S2.3, Visual-Language Model: Image-language features output from the visual encoder and speech encoder are used as input to a multimodal large-scale model to unify the visual-language semantic representation space. Simultaneously, optimization techniques are introduced to improve model inference speed and address the inference latency issue of traditional VLA models. Key-value caching is employed to store the key vectors of historical tokens. K Sum value vector V To avoid redundant calculations of the same features, and to further improve inference speed, canonical correlation analysis (CCA) is used to map the key vector K and value vector V to a lower-dimensional space, resulting in a feature dimension of 256, calculated using the following formula.

[0088] K ' =a T K ; V ' =b T V ; in, K 'and V ' represents the mapped key vector and value vector; a , b This is a vector of typical coefficients.

[0089] S2.4, Action Decoding Module: The action decoding module adopts a hybrid reasoning architecture of high-level semantics and low-level actions, which takes into account both the logic of high-level task planning and the real-time performance of low-level action fine-tuning.

[0090] S2.4.1 The high-level semantic inferencer is used to generate reference action sequences, with the input being the key vector of the visual language model. K 'Sum value vector V ', and the state sequence with added noise Input the transformer model together. H The reference action sequence length is set to 10, and the model simultaneously performs vector calculations during the computation. K 'Sum value vector V Cross-attention calculation, in the first i A transformer layer, outputting a representation. for: ; Here, a cross-attention mechanism is used to inject multimodal contextual prior information from the vision-language model. Then, the intermediate representations are processed in residual parallelism through depthwise separable convolution to update the output features of the i-th layer high-level semantic inference: ; in, This serves as an intermediate representation.

[0091] After further training using a flow matching algorithm, It can learn the distribution pattern of motion trajectories and obtain the denoised motion sequence.

[0092] ; in, N This represents the number of layers in the visual-language model, with a value of 10.

[0093] For the one-dimensional temporal structure of action sequences, a single 1D convolutional layer (kernel size set to 3, stride of 1) is used, employing batch normalization and ReLU activation function. Finally, global average pooling (GAP) is applied to output fixed-dimensional embedding features. Z HL .

[0094] S2.4.2, The low-level action inferencer directly operates on the key-value cache output by the visual-language model. It queries the ontology state matrix. Q Key vector K 'Sum value vector V'The cross-attention mechanism extracts action-related information from the key values. The resulting features are then integrated using average pooling to obtain 256-dimensional action features. These features are then transformed using a single 1D convolutional layer to obtain representational features that capture the semantic meaning of low-level actions.' Z LL .

[0095] S2.5, Action Prediction Module: The above steps generate action features based on a high-level semantic inference engine. Z HL Generate action features with low-level action inferencer Z LL This integrates two types of action guidance information into action prediction.

[0096] S2.5.1 First, use MLP projection to transform the noisy action sequence. Encoded as an action query vector .

[0097] S2.5.2, then use Z HL and Z LL respectively with query vector Perform cross-attention calculation to obtain cross-attention output features. and The calculation formula is: ; ; S2.5.3 In order to achieve two complementary guiding features, high-level semantic guiding features will be used. and low-level action guidance features By concatenating and stitching features together, we obtain the fused features: ; S2.5.4. A two-layer MLP is used to transform the splicing features to obtain the final result. S final Input action head It predicts the denoised action sequence. .

[0098] S2.6 Training process: The flow matching algorithm is adopted as the basic optimization strategy. By learning the continuous flow field from the distribution of noisy action sequences to the distribution of real archival operation action sequences, the model can master the trajectory rules and fine-grained features of archival operation actions, realize the accurate generation and denoising of continuous action sequences, avoid the accuracy loss problem of traditional discrete action modeling, and adapt to the needs of refined archival robot operations.

[0099] S2.6.1 Weighted Joint Optimization Loss Function: The framework is trained using a stream matching objective optimization strategy, with a weighted joint optimization loss function, including a high-level semantic action inferencer. Flow matching mean square error and action head The mean squared error of flow matching is used to balance the training intensity of the two modules through weight coefficients, allowing the high-level semantic inferencer and the action head model to be co-optimized. Due to the physical constraints of the archive robot, the loss function needs to consider action compliance. A physical constraint regularization term is added to the loss function to force the model output to conform to the hardware limits. The loss function is defined as follows: ; in, These are preset parameters. and Preset weights; optional. =0.4, =0.2, =0.5.

[0100] Physical constraint regularization term Defined as: ； in, d For the action vector dimension, , For the first i Hard thresholds for each action, such as the upper and lower limits of the closing angle of a robotic arm, and the upper and lower limits of the opening and closing angle of a robot gripper.

[0101] S2.6.2 Training Process: The preprocessed "visual-language-action" triplet training dataset is divided into training set, validation set, and test set in a ratio of 8:1:1.

[0102] The improved VLA model is initialized by setting hyperparameters such as network layer learning rate, batch size, and number of training epochs.

[0103] The training data is input into the model, and forward propagation is performed according to the above model architecture to obtain the output results of the high-level semantic inference engine and the action head model, respectively.

[0104] Calculate the numerical value of the weighted joint optimization loss function, and use the backpropagation algorithm to propagate the loss gradient back to each layer of the model to update the model parameters; thus obtaining the prediction model.

[0105] In each training round, the performance of the prediction model is validated using a validation set. If the validation set loss does not decrease, hyperparameters such as the learning rate are adjusted to avoid model overfitting.

[0106] After training, the prediction model is tested using a test set to verify its accuracy, real-time performance, and robustness in generating file operation actions.

[0107] This embodiment uses visual-language-action fusion modeling. The model can generate a fine motion trajectory, grasping point and force adapted to the target based on the characteristics of different types of files (such as file boxes, bagged files, and scattered documents) and voice commands (such as "take out file A from file shelf No. 3"). This overcomes the problems of unstable grasping and action failure caused by the diversity of file sizes and shapes in traditional solutions, and improves the reliability of file grasping in unstructured environments.

[0108] This embodiment utilizes a motion reasoning architecture based on flow matching and explicit / low-level motion fusion. Physical constraints (such as the maximum opening and closing angle limit of the gripper and the range of motion of the robotic arm) are added to the motion planning. While ensuring the completion of the motion sequence, the model is forced to output force and speed parameters within a safe range, avoiding damage to the archival materials due to robot misoperation and significantly reducing the physical risk to the archival body during operation.

[0109] This embodiment uses key-value caching and canonical correlation analysis to accelerate multimodal processing, reduce redundant computation and inference latency during cross-modal fusion, and achieve rapid parsing of voice commands and real-time action generation. It supports the dynamic adjustment and continuous operation of the archive robot in narrow equipment environments, and demonstrates high efficiency, especially in the turnover of large amounts of archives.

[0110] This embodiment utilizes a visual encoder branch optimized for archival storage scenarios (such as standardized archival equipment, fixed storage locations, and streamlined operational processes) and labeled data. This enables the system to better extract the semantics of the storage layout, path nodes, and detailed archival attributes, thereby improving the accuracy of positioning and path planning while reducing the cost of human intervention during multi-task switching.

[0111] In this embodiment, the high-level semantic inferencer implements task-level long-order planning, such as sequentially transferring multiple files and generating a continuous and smooth sequence of actions, while the low-level action inferencer responds to dynamic changes in the environment to achieve real-time fine-grained adjustments (such as slight avoidance when encountering sudden obstacles). The combination of the two ensures the logical correctness of the process and improves the robustness and anti-interference of action execution, making it suitable for possible dynamic environmental changes during file retrieval.

[0112] This embodiment combines physical constraint regularization with flow matching loss to design a dual-module differentiated weighted joint loss function. By setting relaxed constraint weights for the high-level semantic inference engine (ensuring flexibility in high-level action planning) and strict constraint weights for the action head (ensuring hardware compliance of low-level action execution), the model is forced to output a continuous action sequence that conforms to the hardware limits of the archival robot from the perspective of loss optimization. This completely solves problems such as action distortion, mechanical collisions, and jamming during the transfer from simulation to the real machine, significantly improving the effectiveness and safety of action execution. At the same time, it retains the advantage of the flow matching algorithm in fine-grained modeling of continuous actions, achieving dual optimization of action flow field fitting accuracy and hardware physical compliance.

[0113] This embodiment proposes a hybrid reasoning VLA model architecture for archive storage scenarios. This architecture overcomes the limitations of traditional VLA models' single reasoning mode. A high-level semantic inferencer handles high-level action sequence planning and logical interpretation, while a low-level action inferencer performs real-time fine-tuning of low-level actions. This balances the logicality of action planning with the real-time performance of action execution, resolving the reasoning latency issue of traditional models. By integrating this improved VLA model into an archive management robot, the fragmented workflow of traditional archive robots is broken, achieving end-to-end mapping from visual perception and language understanding to action generation. This enables fully autonomous operations, including archive storage, retrieval, inventory, and transportation, without the need for manual intervention.

[0114] Example 3: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the visual language action model-driven autonomous operation method for an archival robot described in Embodiment 1.

[0115] This embodiment generates adaptive motion trajectories, gripping points, and forces based on the characteristics of different types of archives and voice commands within a trained visual-language motion model. It can generate precise motion trajectories, gripping points, and forces according to different archive sizes, such as archive boxes, bagged archives, and loose documents, and voice commands. This overcomes the problems of unstable gripping and motion failure caused by the diversity of archive sizes and shapes in traditional solutions, improving the reliability of archive gripping in unstructured environments. By incorporating the maximum opening angle limit of the gripper and physical constraints on the robotic arm's range of motion into the motion planning, it ensures the completion of the motion sequence while outputting force and speed parameters within a safe range. This avoids damage to archive materials due to robot misoperation, significantly reducing the physical risks to the archive itself during operation and further improving the reliability of archive gripping.

[0116] This embodiment combines physical constraint regularization with flow matching loss. By setting lenient constraint weights for the high-level semantic inference (ensuring the flexibility of high-level action planning) and strict constraint weights for the low-level action inference (ensuring hardware compliance of low-level action execution), it forces the model to output a continuous action sequence that conforms to the hardware limits of the archive robot from the perspective of loss optimization. This completely solves the problems of motion distortion, mechanical collision, and jamming in the migration from simulation to real machine, and greatly improves the effectiveness and safety of action execution.

[0117] This embodiment breaks through the limitations of the traditional VLA model's single reasoning mode. The high-level semantic inferencer realizes high-level action sequence planning and logical interpretation, while the low-level action inferencer realizes low-level action real-time fine-tuning. It takes into account both the logic of action planning and the real-time performance of action execution, breaking the limitations of the fragmented workflow of traditional archival robots. It realizes end-to-end mapping from visual perception and language understanding to action generation, and can complete the entire process of autonomous operation such as archival storage, retrieval, inventory, and transportation without manual intervention.

[0118] Example 4: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the steps of the visual language action model-driven autonomous operation method for an archival robot described in Embodiment 1.

[0119] This embodiment generates adaptive motion trajectories, gripping points, and forces based on the characteristics of different types of archives and voice commands within a trained visual-language motion model. It can generate precise motion trajectories, gripping points, and forces according to different archive sizes, such as archive boxes, bagged archives, and loose documents, and voice commands. This overcomes the problems of unstable gripping and motion failure caused by the diversity of archive sizes and shapes in traditional solutions, improving the reliability of archive gripping in unstructured environments. By incorporating the maximum opening angle limit of the gripper and physical constraints on the robotic arm's range of motion into the motion planning, it ensures the completion of the motion sequence while outputting force and speed parameters within a safe range. This avoids damage to archive materials due to robot misoperation, significantly reducing the physical risks to the archive itself during operation and further improving the reliability of archive gripping.

[0120] This embodiment combines physical constraint regularization with flow matching loss. By setting lenient constraint weights for the high-level semantic inference (ensuring the flexibility of high-level action planning) and strict constraint weights for the low-level action inference (ensuring hardware compliance of low-level action execution), it forces the model to output a continuous action sequence that conforms to the hardware limits of the archive robot from the perspective of loss optimization. This completely solves the problems of motion distortion, mechanical collision, and jamming in the migration from simulation to real machine, and greatly improves the effectiveness and safety of action execution.

[0121] This embodiment breaks through the limitations of the traditional VLA model's single reasoning mode. The high-level semantic inferencer realizes high-level action sequence planning and logical interpretation, while the low-level action inferencer realizes low-level action real-time fine-tuning. It takes into account both the logic of action planning and the real-time performance of action execution, breaking the limitations of the fragmented workflow of traditional archival robots. It realizes end-to-end mapping from visual perception and language understanding to action generation, and can complete the entire process of autonomous operation such as archival storage, retrieval, inventory, and transportation without manual intervention.

[0122] Example 5: This embodiment provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the visual language action model-driven autonomous operation method for archival robots described in Embodiment 1.

[0123] This embodiment generates adaptive motion trajectories, gripping points, and forces based on the characteristics of different types of archives and voice commands within a trained visual-language motion model. It can generate precise motion trajectories, gripping points, and forces according to different archive sizes, such as archive boxes, bagged archives, and loose documents, and voice commands. This overcomes the problems of unstable gripping and motion failure caused by the diversity of archive sizes and shapes in traditional solutions, improving the reliability of archive gripping in unstructured environments. By incorporating the maximum opening angle limit of the gripper and physical constraints on the robotic arm's range of motion into the motion planning, it ensures the completion of the motion sequence while outputting force and speed parameters within a safe range. This avoids damage to archive materials due to robot misoperation, significantly reducing the physical risks to the archive itself during operation and further improving the reliability of archive gripping.

[0124] This embodiment combines physical constraint regularization with flow matching loss. By setting lenient constraint weights for the high-level semantic inference (ensuring the flexibility of high-level action planning) and strict constraint weights for the low-level action inference (ensuring hardware compliance of low-level action execution), it forces the model to output a continuous action sequence that conforms to the hardware limits of the archive robot from the perspective of loss optimization. This completely solves the problems of motion distortion, mechanical collision, and jamming in the migration from simulation to real machine, and greatly improves the effectiveness and safety of action execution.

[0125] This embodiment breaks through the limitations of the traditional VLA model's single reasoning mode. The high-level semantic inferencer realizes high-level action sequence planning and logical interpretation, while the low-level action inferencer realizes low-level action real-time fine-tuning. It takes into account both the logic of action planning and the real-time performance of action execution, breaking the limitations of the fragmented workflow of traditional archival robots. It realizes end-to-end mapping from visual perception and language understanding to action generation, and can complete the entire process of autonomous operation such as archival storage, retrieval, inventory, and transportation without manual intervention.

[0126] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A method for autonomous operation of an archival robot driven by a visual language action model, characterized in that, include: Based on the visual, linguistic, and motion data related to the document management robot, annotated multimodal triplet data is generated; Based on multimodal triplet data, in the trained visual-language-action model, the action trajectory, grasping point, and force of the target are generated according to the file features of different specifications and voice commands. Specifically, the visual encoder extracts grasping-related visual features and navigation-related visual features respectively, and the language encoder converts text and voice commands into language feature vectors. Based on the language feature vectors, a high-level semantic inferencer generates a reference action sequence, and a low-level action inferencer directly acts on the key-value cache to obtain their respective action features. The action features are fused into action prediction to obtain the action sequence. The maximum opening angle limit of the gripper and the physical constraints of the robotic arm's range of motion are added to the action planning. The file management robot is controlled based on the obtained action sequence.

2. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 1, characterized in that, The visual data includes capture-related visual data and navigation-related visual data.

3. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 2, characterized in that, The visual data capture-related annotations include the file QR code, file storage location coordinates, file size, file size specifications, and file grabbing point; the navigation-related visual data navigation-related annotations include the file equipment number, file equipment layer, warehouse obstacle location, and ground navigation path nodes; the language data annotation instruction types include instruction type, operation object, target location, and other auxiliary instructions, the instruction type includes take out, place, transport, and inventory, and the other auxiliary instructions include handle with care; the motion data includes movement coordinates, chassis motor speed, robotic arm joint angle, gripping opening and closing degree, motion execution sequence, and operation success rate, while also annotating the robotic arm lifting height threshold and gripping force threshold for motion execution.

4. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 1, characterized in that, The visual encoder uses the MobileNetV3 network as its basic backbone and introduces a residual connection mechanism to obtain shallow features and deep features, which are used as inputs to the grasping visual branch network and the deep features as inputs to the navigation branch network, respectively. Both the grasping visual branch network and the navigation branch network include convolutional layers and fully connected layers.

5. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 1, characterized in that, The language encoder uses BERT-base as its backbone network.

6. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 1, characterized in that, Image feature vectors are obtained using a visual encoder. Based on the image feature vectors and language feature vectors, the semantic representation spaces of vision and language are unified. Historical key vectors and value vectors are stored, and the key vectors and value vectors are mapped to a lower-dimensional space.

7. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 6, characterized in that, Based on the key vector and value vector mapped to a lower-dimensional space, and the noisy state sequence, cross-attention calculation is performed on the vector and value vectors. Multimodal contextual prior information is injected using the cross-attention mechanism. Then, the intermediate representation is processed in residual parallel through depthwise separable convolution to update the output features of the high-level semantic inferencer. The denoised action sequence is obtained after training using a stream matching algorithm.

8. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 7, characterized in that, The step of fusing the action features obtained from the high-level semantic inferencer and the low-level action inferencer into action prediction includes: encoding the noisy action sequence into an action query vector using projection; generating action features using the high-level semantic inferencer and the low-level action inferencer, and performing cross-attention calculation with the query vector to obtain high-level semantic guidance features and low-level action guidance features; and concatenating the high-level semantic guidance features and the low-level action guidance features to obtain the fused features.

9. The method for autonomous operation of an archival robot driven by a visual language action model as described in claim 1, characterized in that, The framework is trained using a stream matching objective optimization strategy, with a weighted joint optimization loss function, including a high-level semantic action inferencer. Flow matching mean square error and action head The mean squared error of stream matching is used to balance the training intensity of the two modules through weight coefficients, allowing the high-level semantic inferencer and the action head model to be co-optimized; the loss function is: ； Physical constraint regularization term Defined as: in, These are preset parameters. and Preset weights; d For the action vector dimension; , For the first i Hard threshold for each action.

10. A visual language action model-driven autonomous operating system for archival robots, characterized in that, include: The data acquisition module is configured to generate labeled multimodal triplet data based on visual data, language data, and motion data related to the file management robot. Based on multimodal triplet data, in the trained visual-language-action model, the action trajectory, grasping point, and force of the target are generated according to the file features of different specifications and voice commands. Specifically, the visual encoder extracts grasping-related visual features and navigation-related visual features respectively, and the language encoder converts text and voice commands into language feature vectors. Based on the language feature vectors, a high-level semantic inferencer generates a reference action sequence, and a low-level action inferencer directly acts on the key-value cache to obtain their respective action features. The action features are fused into action prediction to obtain the action sequence. The maximum opening angle limit of the gripper and the physical constraints of the robotic arm's range of motion are added to the action planning. The control module is configured to control the file management robot based on the obtained action sequence.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the visual language action model-driven autonomous operation method for archival robots as described in any one of claims 1-9.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the steps of the visual language action model-driven autonomous operation method for an archive robot as described in any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the visual language action model-driven autonomous operation method for an archival robot as described in any one of claims 1-9.

Images