A dual-arm robot control method based on a VLA model and a thinking chain cognitive alignment and electronic equipment

Abstract

This invention relates to the field of medical robotics and embodied intelligence, specifically to a dual-arm robot control method and electronic device based on VLA model and cognitive alignment of thought chain. The method includes the following steps: S1, multimodal task scene perception; S2, full-modal space alignment based on multidimensional state lexicalization and low-rank adaptive alignment; S3, dynamic intervention mechanism based on cognitive alignment assessment; S4, end-to-end joint space motion generation based on reparameterization; S5, compliant execution of cognitive-motor dual-loop closed-loop. This invention constructs a multimodal state vector by fusing robotic arm motion data, a six-dimensional force sensor at the end effector, and a third-view camera. Based on a deep learning fusion model, it performs feature extraction and correlation analysis to objectively and quantitatively assess the user's dual-arm motor ability. This allows for the capture of the subject's motion details and neuromuscular coordination characteristics, quantitative characterization of the subject's dual-arm motor ability, dynamic monitoring of the user's dual-arm motor ability progress, and scientific quantification.

Description

Technical Field

[0001] This invention relates to the field of medical robotics and embodied intelligence, specifically to a dual-arm robot control method and electronic device based on VLA model and cognitive alignment of thought chain. Background Technology

[0002] With the rising incidence of stroke, spinal cord injury, and neuromuscular diseases, the need for exercise among patients with upper limb motor dysfunction is becoming increasingly urgent. Clinical studies have shown that task-oriented therapy can effectively activate the cerebral cortex and promote the plasticity reorganization of the central nervous system. Against this backdrop, dual-arm robots are widely used in clinical practice due to their high-intensity and repetitive training capabilities. However, existing dual-arm robots and related intelligent control systems still face multiple intertwined clinical and technical bottlenecks in practical deployment.

[0003] First, traditional task-oriented training generally falls into the trap of "emphasizing movement while neglecting cognition." Existing devices mostly focus on purely mechanical physical following and passive trajectory control, with patients often passively receiving drags from robotic arms. Because the system cannot effectively awaken the patient's motor intention before the limbs produce physical movement, "the machine moves but the brain is not fully involved," significantly reducing the efficiency of central nervous system circuit remodeling. To compensate for this perceptual deficiency, some systems attempt to introduce multimodal fusion, but often rely excessively on cumbersome wearable physiological sensors such as surface electromyography (sEMG). These devices are not only time-consuming to wear clinically, but are also highly susceptible to interference from sweat and electrode misalignment, making it difficult to naturally integrate the visual environment, medical instructions, and the patient's immediate physical force perception. More seriously, existing end-to-end deep learning control algorithms often exhibit "black box" characteristics, with the system directly outputting abrupt action commands without explaining the action logic to the patient beforehand. This can easily trigger the patient's fear of the unknown and resistance, thereby inducing dangerous antagonistic spasticity.

[0004] Furthermore, existing technologies struggle to balance low latency and extremely high security at the network architecture and underlying motion control levels. On one hand, traditional multimodal fusion, if employing external adapter modules for feature alignment, inevitably introduces additional inference overhead. The resulting significant computational latency is simply unacceptable for force control's stringent kilohertz-level real-time requirements. On the other hand, existing control frameworks heavily rely on complex Cartesian inverse kinematics (IK) solutions. During complex multi-DOF human-machine collaborative training, IK solutions are highly susceptible to spatial singularities, leading to sudden changes in joint angular velocities or even complete system failure—completely unacceptable in medical training scenarios where absolute safety is paramount.

[0005] Therefore, how to introduce embodied intelligence models to achieve "semantic closure" and "cognitive alignment" between machine intentions and patient brain motor intentions under the premise of getting rid of cumbersome wearable devices in clinical practice, while completely abandoning high-latency external network structures and error-prone IK solution architectures, and directly outputting smooth low-level joint control commands with extremely low latency has become a key technical bottleneck that urgently needs to be overcome in the current field of robotics. Summary of the Invention

[0006] The purpose of this invention is to provide an efficient and safe dual-arm robot control method and electronic device based on VLA model and cognitive alignment of thought chain, from "multimodal scene understanding", "cognitive semantic closed loop" to "end-to-end joint space control".

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A dual-arm robot control method based on VLA model and cognitive alignment with thought chain includes the following steps:

[0009] S1. Multimodal task scene perception: Acquire visual images of the work area environment and the user through a third-person depth camera. Receive natural language task instructions through a microphone array The robot acquires its real-time physical state through a six-dimensional force / torque sensor at the end effector and sensors on the robot itself. ;

[0010] S2. Full-modal space alignment based on multidimensional state lexicalization and low-rank adaptation: Align the visual image obtained in step S1. With natural language task instructions The encoded visual and linguistic lexical units are then used to project the continuous physical states obtained in step S1 into a projection network. Mapping to high-dimensional physical state lexical units; injecting a trainable low-rank adaptive LoRA matrix into the attention layer of the Visual-Language-Action (VLA) big model; concatenating the high-dimensional physical state lexical units and inputting them into the VLA model for cross-modal feature alignment; generating a thought chain sequence based on autoregressive decoding; and converting the current subtask into a perceptual cue signal and outputting it to the user.

[0011] S3. Dynamic intervention mechanism based on cognitive alignment assessment: Within the cognitive response time window after the user receives the perceptual cue signal in step S2, the cognitive participation score is calculated by monitoring the user's subtle force exertion trend. If the score is lower than the preset safety threshold, the VLA model is triggered to regenerate the downgraded thought chain strategy and re-output the downgraded perception prompt signal to the user. Step S3 is repeated to recalculate the cognitive engagement score. If the recalculated cognitive engagement score reaches the preset safety threshold, the subsequent steps are executed.

[0012] S4. End-to-end joint space motion generation based on reparameterization: when cognitive engagement rating When the preset safety threshold is reached, the task-state fine-tuning matrix weights trained by low-rank adaptive LoRA in step S2 are obtained, and they are fused with the basic static weights of the VLA model through linear reparameterization technology to construct a branch-free integrated control model; the multimodal feature-aligned full-modal lexical units and the currently activated thought chain steps are input into the parameter-reconstructed VLA model action head ActionHead; the action head bypasses the inverse kinematics IK solution of Cartesian space and directly outputs joint space control commands, including the target joint position increment or target joint torque of the robot's underlying motors;

[0013] S5. Cognitive-Motor Dual Closed-Loop Compliant Execution: The joint space control commands output by the VLA model in step S4 are combined with the underlying admittance or impedance control model to control the upper limb robot to compliantly assist the user in performing corresponding actions.

[0014] Preferably, the real-time physical state in the multimodal task scene perception step S1 Represented as:

[0015]

[0016] in, For robot joint angles, The joint angular velocity, The human-computer interaction force is measured by a six-dimensional force / torque sensor at the end.

[0017] Preferably, the thought chain sequence in step S2, based on multidimensional state word tuple transformation and low-rank adaptive full-modal space alignment, is represented as follows:

[0018]

[0019] in, Indicates the first Each sub-task reasoning step Represents a thought chain sequence. This represents the reasoning step of the i-th subtask. This represents the total number of sub-task reasoning steps contained in the thought chain sequence; its joint conditional probability distribution formula is:

[0020] in, This represents a sequence of visual lexical units obtained from the encoding of a visual image. This represents the sequence of language lexical units encoded by the instructions of a natural language task.

[0021] Preferably, step S3 is based on the cognitive engagement score in the dynamic intervention mechanism of cognitive alignment assessment. Represented as:

[0022]

[0023] in, The response delay generates effective motion trends for users. The magnitude of the vector representing the deviation between the user's actual force direction and the intended trajectory direction. The force threshold, , , , These are the weights and adjustment coefficients.

[0024] Preferably, step S4 is based on the output time in the reparameterized end-to-end joint space motion generation. Target joint position increment Or target joint torque mapping function Represented as:

[0025]

[0026] in, This is a neural network mapping function that has undergone efficient parameter fine-tuning and weight merging.

[0027] Preferably, in step S4, the end-to-end joint space motion generation based on reparameterization employs a network architecture based on a discretized motion space or diffusion strategy to predict the motion sequence for multiple future time steps, and its joint space increment prediction... Represented as:

[0028]

[0029] in, To predict the step size.

[0030] Preferably, in step S5, during the cognition-motion dual closed-loop compliant execution, after combining the underlying admittance or impedance control model, the joint torque ultimately executed by the robotic arm is... Represented as:

[0031]

[0032] in, , These are the stiffness and damping matrices, respectively. This is the transpose of the Jacobian matrix for the robot.

[0033] An upper limb robot control system based on VLA model and cognitive alignment with thought chain includes:

[0034] The multimodal perception module is used to acquire environmental, natural language commands, and physical state information through a third-view depth camera and an end-effector six-dimensional force sensor.

[0035] The lexicalization and semantic loop closure module is used to map multidimensional data into lexical sequences, generate thought chains using a VLA model injected with LoRA, and output perceptual cue signals.

[0036] The dynamic intervention assessment module is used to calculate the cognitive engagement score within the cognitive response time window, determine whether to trigger strategy downgrade, and re-enter the cognitive response assessment after the downgrade strategy is output.

[0037] The motion reparameterization generation module is used to directly output control commands in the joint space after merging weights;

[0038] The dual closed-loop execution module is used to combine VLA feedforward commands with underlying impedance feedback to drive the upper limb robot.

[0039] An electronic device includes a memory and a processor, characterized in that the memory stores a computer program, and the processor executes the computer program to implement any step of the dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described above.

[0040] The beneficial effects of this invention are:

[0041] This invention reconstructs the upper limb training paradigm, breaking the "black box" barrier and achieving a deep alignment between cognition and movement. Traditional robots passively drag patients' limbs, while this invention innovatively applies the multimodal large-scale model of thought chain (CoT) to the training scenario. Complex daily training tasks (such as "pouring water") are deconstructed into logically rigorous and interpretable sub-steps, and feedback is provided to the patient in voice / visual form before the action occurs. This "cognition first, force second, assistance third" model fully pre-activates the patient's motor cortex before limb movement, completely resolving the patient's resistance to black-box AI and significantly improving the efficiency of central nervous system plasticity remodeling.

[0042] This invention eliminates cumbersome electromyography (sEMG) devices and high-latency adapters, improving the simplicity of system deployment and reducing the computational overhead and response latency caused by additional inference structures. Addressing the clinical pain points of time-consuming sEMG device wear and susceptibility to external interference, this system accurately captures patient intentions using only a third-view camera and a high-precision six-dimensional force sensor at the end. At the algorithm layer, this invention replaces the traditional cross-modal adapter module with an architecture combining physical state lexicalization and LoRA fine-tuning. During deployment, reparameterization technology is used to incorporate LoRA weights into the backbone network, achieving low-additional structural overhead alignment of "force perception-vision-language," significantly reducing the computational power required for model training and ensuring the extremely high real-time performance (high-frequency operation) required for underlying force control.

[0043] This invention achieves end-to-end direct output of joint control commands, reducing the risk of spatial singularities caused by IK calculations in multi-DOF systems. Traditional training control frameworks heavily rely on inverting the inverse kinematics (IK) matrix from Cartesian space to joint space, which is prone to deadlock or sudden velocity changes under complex postures. This invention utilizes an embodied intelligent large model to directly learn and output control commands (position increments) in joint space. or feedforward torque By avoiding the vulnerable aspects of IK calculation at the underlying algorithmic logic level, the robotic arm's motion trajectory exhibits a high degree of smoothness and naturalness, much like that of a human arm.

[0044] This invention employs a dynamic error correction mechanism that constructs a semantic-physical dual closed loop, which helps improve the safety and compliance of the system during operation. The system not only possesses a low-level physical impedance closed loop operating at the kilohertz level (instantly yielding when a strong opposing torque is detected), but also innovatively incorporates a top-level semantic-level cognitive closed loop. When the calculated cognitive engagement score... When the intensity is too low, the system can "read the room" like a professional therapist and proactively pause the planned high-intensity movements. The VLA model dynamically reverts to a safe soothing strategy or downgraded task, achieving truly personalized, adaptive, advanced intelligent flexible training. Attached Figure Description

[0045] Figure 1 This is an overall flowchart of a dual-arm robot control method based on VLA model and cognitive alignment of thought chain in Embodiment 1 of the present invention;

[0046] Figure 2 This is a schematic diagram of the hardware platform architecture of the dual-arm robot system in Embodiment 1 of the present invention. Detailed Implementation

[0047] Example 1

[0048] The following is a further explanation of the present invention in conjunction with specific embodiments, such as... Figure 1 As shown, this embodiment is a dual-arm robot control method based on VLA model and cognitive alignment of thought chain, and the hardware platform on which it relies is as follows: Figure 2 As shown, it includes a third-view depth camera, an interactive display and voice broadcast terminal, an integrated industrial control computer host, a seven-degree-of-freedom collaborative robotic arm, and a high-precision six-dimensional force / torque sensor installed at the end handle of the robotic arm.

[0049] Specifically, it includes the following steps:

[0050] S1. Multimodal task scene perception: Training begins. The user sits at the workbench and holds the handle at the end of the robotic arm. The depth camera... The system scans the desktop workspace in real time at a high frequency to acquire RGB-D visual tensors containing the target object (such as a red wooden block or a water glass) and the user's upper body skeletal posture. Doctors give natural language commands via microphone. For example: "Guide the patient to slowly grasp the red wooden block in front of them with their right hand and move it to the target box on the left." Meanwhile, the underlying control bus... High-frequency reading of robotic arm joint angles angular velocity And the human-computer interaction force measured by the end-effector six-dimensional force sensor Construct real-time physical state Real-time physical state Represented as:

[0051]

[0052] in, For robot joint angles, The joint angular velocity, The human-computer interaction force is measured by a six-dimensional force / torque sensor at the end.

[0053] S2. Full-modal space alignment based on multidimensional state lexicalization and low-rank adaptation: Align the visual image obtained in step S1. With natural language task instructions Encoding is done as visual and linguistic lexical units; targeting continuous physical states. The continuous physical states obtained in step S1 are represented by a lightweight projection network. Mapped to high-dimensional physical state tokens, abandoning the traditional external feature adapter; a trainable low-rank adaptive (LoRA) matrix is ​​injected into the Transformer attention layer of the vision-language-action (VLA) big model. The above-mentioned full-modality high-dimensional physical state tokens are concatenated and input into the VLA model for cross-modal feature alignment. The VLA model generates a thought chain sequence based on autoregressive decoding, and transforms the current subtask into a perceptual prompt signal and outputs it to the user.

[0054] The thought chain sequence generated by the VLA model is represented as follows:

[0055]

[0056] in, Indicates the first Each sub-task reasoning step Represents a thought chain sequence. This represents the reasoning step of the i-th subtask. This represents the total number of sub-task reasoning steps contained in the thought chain sequence; its joint conditional probability distribution formula is:

[0057]

[0058] in, This represents a sequence of visual lexical units obtained from the encoding of a visual image. This represents the sequence of language lexical units encoded by the instructions of a natural language task.

[0059] In the initialization phase of step S2, the continuous physical state data generated in step S1 is mapped into physical state tokens (State Tokens) of the same dimension as visual / language tokens through a three-layer MLP projection network.

[0060] To achieve efficient cross-modal alignment, this embodiment injects a low-rank adaptive (LoRA) matrix in parallel into the Transformer layer of the VLA model. The model receives the concatenated full-modal word sequence and generates a thought chain sequence based on autoregressive decoding. For example, generating:

[0061] [Thinking] The target water cup is 20cm in front of you. You need to first drive your shoulder and elbow joints to make the handle move steadily closer to the water cup.

[0062] [Thinking] Once above the target, guide the patient to apply grip strength to complete the grasping.

[0063] [Thinking] Maintain the grip and move to the left along the collision avoidance trajectory. .

[0064] The system then proceeds with the thought chain steps. The semantics are communicated to the user via the voice module: the terminal announces, "Please note that we are now preparing to extend our right arm forward towards the red wooden block." This announcement serves as a perceptual cue signal, constructing a cognitive semantic closed loop.

[0065] S3. Dynamic intervention mechanism based on cognitive alignment assessment: After the user receives the perceptual cue signal in step S2, the system sets a cognitive response time window. For example, in broadcasting Within the subsequent 1.5-second cognitive response window, the system monitors the user's subtle active force exertion trends using a six-dimensional force sensor. A cognitive engagement score is then calculated. :

[0066]

[0067] in, The response delay generates effective motion trends for users. The magnitude of the vector representing the deviation between the user's actual force direction and the intended trajectory direction. The force threshold, , , , These are the weights and adjustment coefficients.

[0068] If cognitive engagement score If the value falls below a preset safety threshold (indicating patient fatigue or cognitive dissonance), the VLA model is triggered to regenerate a downgraded soothing thought chain strategy; for example, if it detects a deviation in force exertion vector due to user distraction or resistance. Too large, rating If the level falls below a safety threshold, the system will determine that there is a cognitive disconnect. The VLA model will immediately interrupt the current task and output a soothing command. After the soothing command is output, the system does not directly enter the physical execution stage. Instead, it feeds back the downgraded perceptual cue signal to the user and returns to the cognitive response time window of step S3 to collect the weak force trend and response delay again, and recalculates the cognitive participation score. When the recalculated score reaches the preset safety threshold, the system will then enter step S4 to execute the corresponding joint space action generation. Otherwise, it will continue to maintain the pause, relaxation, or low-intensity assistance strategy.

[0069] S4. End-to-end Joint Space Action Generation Based on Reparameterization: During the inference deployment phase, the task-state fine-tuning matrix weights trained by low-rank adaptive LoRA in step S2 are seamlessly merged into the basic static weights of the VLA model using linear reparameterization technology. This eliminates the structural inference overhead of cross-modal feature fusion, constructs a branch-free integrated control model, and ensures no additional computational branches during inference. Then, the feature-fused full-modal lexical units and the currently activated thought chain steps are input into the parameter-reconstructed VLA model action head (ActionHead), completely bypassing Cartesian inverse kinematics (IK) calculation. A network architecture based on discretized action space or diffusion strategy is adopted to directly predict the future. Joint spatial position increment at each time step Or directly output the target joint torque. This approach avoids the singularity collapse problem that robotic arms may encounter when working in multi-degree-of-freedom collaboration, and helps to improve the smoothness and continuity of motion trajectories.

[0070] Increment of joint spatial position Represented as:

[0071]

[0072] in, To predict the step size.

[0073] time Target joint position increment Or target joint torque mapping function Represented as:

[0074]

[0075] in, This is a neural network mapping function that has undergone efficient parameter fine-tuning and weight merging.

[0076] S5. Cognitive-Motor Dual Closed-Loop Compliant Execution: Combine the joint space control commands output by the VLA model in step S4 with the underlying admittance or impedance control model to control the dual-arm upper limb robot to compliantly assist the user in performing corresponding actions.

[0077] The final fusion joint torque executed by the robotic arm Represented as:

[0078]

[0079] in, , These are the stiffness and damping matrices dynamically adjusted based on different motion primitives. Let be the transpose of the Jacobian matrix from the joint space of the robotic arm to the Cartesian operation space.

[0080] Example 2

[0081] This embodiment is based on the method proposed in Embodiment 1. When a user experiences "cognitive disconnect" due to mental fatigue during training, or sudden uncontrollable "muscle spasms", how to use a semantic-physical dual closed-loop mechanism to perform dynamic strategy degradation and extreme security fault tolerance protection.

[0082] Scenario 1: Dynamic Degradation Intervention for Cognitive Disconnection Based on Top-Level Semantic Loop

[0083] During the execution of a high-intensity training task (such as "resisted arm raising"), the system broadcasts the corresponding task sub-steps in step S2. Then proceed to the evaluation stage in step S3.

[0084] Within the set cognitive response time window Inside, the system monitors the user's intention to exert force through a high-precision six-dimensional force sensor at the end. If the user, due to mental fatigue or distraction, fails to exert force in accordance with the command, or even generates resistance contrary to the target trajectory, it will cause a deviation in the force exertion vector. Significantly increased, and response delay Approaching the upper limit of the window.

[0085] For example, calculated based on the cognitive engagement rating formula. This value is far below the preset safety threshold (e.g., 0.4). At this point, the top-level semantic closed loop triggers a degradation intervention mechanism:

[0086] Upon receiving a physical state term representing "low cognitive engagement," the VLA model immediately blocks the generation of the originally planned high-intensity action trajectory in subsequent time steps and re-plans a safer, degraded thought chain sequence based on an autoregressive mechanism.

[0087] [Status Assessment] Interactive mechanical characteristics and response delay indicate that the user is currently in a state of high fatigue or cognitive dissociation. [Decision] Abort the resistance lifting task and switch to active assisted relaxation mode.

[0088] Immediately, the system outputs a reassuring prompt to the patient via voice terminal: "We've detected some fatigue in your arm. Let's stop lifting and relax your muscles for a bit." The robotic arm then smoothly transitions to a zero-gravity hover state. After this de-escalation prompt, the system re-enters the cognitive response time window of step S3. Only when the recalculated cognitive engagement score reaches a preset safety threshold will it proceed to step S4 to generate subsequent actions; otherwise, it continues to maintain zero-gravity hover or low-intensity active assistance mode. This mechanism avoids the secondary damage caused by the forced dragging of traditional dual-arm robots when the patient is fatigued.

[0089] Scenario 2: Spasmodic Emergency Fault Tolerance Based on Underlying Physical Closed Loop and Cross-Modal Fusion

[0090] Suppose that during task execution or relaxation hovering, the user experiences a sudden, intense muscle spasm in the affected limb. At this moment, a six-dimensional force sensor mounted on the robotic arm's end effector captures a transient surge of force far exceeding the normal interaction threshold within an extremely short time (based on a high-frequency sampling rate of 1000Hz). .

[0091] First, the underlying physical closed loop triggers an emergency compliant yield. The impedance controller, based on the dynamic equations, responds to a surge in [data / signals]. Upon feedback, the virtual stiffness matrix components in the direction of force application are automatically and instantly reduced. And simultaneously increase the damping matrix components Under this control law, the robotic arm exhibits extremely high physical compliance, smoothly yielding in the direction of the user's spasmodic force, absorbing the destructive energy generated by the spasm like a sponge, and reducing the risk of secondary injury caused by rigid mechanical confrontation from a physical perspective. Secondly, cross-modal fusion triggers a system-wide silent protection mechanism. This abnormally surge in force data is synchronously mapped into high-dimensional physical state terms and reported in real time to the input of the VLA large model. After the VLA model senses the "spasm / extremely high-risk collision" feature term, its Action Head immediately cuts off all active task planning and resets the feedforward target joint torque to zero (i.e., sets...). Until the six-dimensional force sensor detects it within multiple consecutive time windows. Once the system returns to a stable and safe range, the VLA model will attempt to re-establish cognitive alignment using prompts (such as "Are you feeling better now?") and slowly resume the training task.

[0092] Example 3

[0093] This embodiment is a dual-arm robot control system used to implement the dual-arm robot control methods proposed in Embodiments 1 and 2; the system is logically divided into multiple functional modules and deployed in the training host of the dual-arm robot system, including:

[0094] Multimodal perception module: responsible for driving the third-view depth camera, microphone array and six-dimensional force sensor to achieve high-frequency synchronous acquisition and alignment buffering of environment, voice and physical interaction forces.

[0095] The lexicalization and semantic loop closure module encapsulates a large VLA model finely tuned by Hybrid LoRA. It is responsible for converting multimodal data into visual, linguistic, and physical state lexical units, generating subtasks based on thought chain (CoT) reasoning, and outputting cognitive prompts to patients through a speech synthesis terminal.

[0096] Dynamic intervention assessment module: As the core of safety monitoring, it calculates cognitive engagement scores in real time within the cognitive response window. And when the score is not up to standard or an abnormal torque is detected, an interruption and replanning signal is sent to the lexicalization and semantic loop module.

[0097] Action reparameterization generation module: It carries the action execution head of the VLA model. After completing the linear merging of LoRA weights and static backbone, it directly outputs multi-step joint position increments or feedforward torques based on an end-to-end network (such as Diffusion Policy).

[0098] Dual closed-loop execution module: integrates high-speed, low-level admittance / impedance control algorithms, combines large-scale model feedforward commands with high-frequency torque feedback, and drives the motors of each joint of the robotic arm through an industrial bus.

[0099] The above description is merely a further explanation of the present invention in conjunction with specific embodiments. All descriptions made do not imply any limitation on the scope of protection of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A control method for a dual-arm robot based on VLA model and cognitive alignment with thought chain, characterized in that, Includes the following steps: S1. Multimodal task scene perception: Acquire visual images of the work area environment and the user through a third-person depth camera. Receive natural language task instructions through a microphone array The robot acquires its real-time physical state through a six-dimensional force / torque sensor at the end effector and sensors on the robot itself. ; S2. Full-modal space alignment based on multidimensional state lexicalization and low-rank adaptation: Align the visual image obtained in step S1. With natural language task instructions The encoded visual and linguistic lexical units are then used to project the continuous physical states obtained in step S1 into a projection network. The high-dimensional physical state lexical units are mapped to high-dimensional physical state lexical units. A trainable low-rank adaptive LoRA matrix is ​​injected into the attention layer of the visual-language-action VLA large model. The high-dimensional physical state lexical units are concatenated and input into the VLA model for cross-modal feature alignment. Based on autoregressive decoding, a thought chain sequence is generated, and the current subtask is transformed into a perception prompt signal and output to the user. S3. Dynamic intervention mechanism based on cognitive alignment assessment: Within the cognitive response time window after the user receives the perceptual cue signal in step S2, the cognitive participation score is calculated by monitoring the user's subtle force exertion trend. If the score is lower than the preset safety threshold, the VLA model is triggered to regenerate the downgraded thought chain strategy and output the downgraded perception prompt signal back to the user, repeating step S3 to recalculate the cognitive engagement score. If the recalculated cognitive engagement score reaches the preset safety threshold, then proceed with the next steps. S4. End-to-end joint space motion generation based on reparameterization: When the cognitive engagement score reaches a preset safety threshold, the task-state fine-tuning matrix weights trained by low-rank adaptive LoRA in step S2 are obtained, and they are fused with the basic static weights of the VLA model through linear reparameterization technology to construct a branch-free integrated control model; the multimodal feature-aligned full-modal lexical units and the currently activated thought chain sub-steps are input into the parameter-reconstructed VLA model action execution head ActionHead; The motion execution head bypasses the inverse kinematics (IK) calculation process in Cartesian space and directly outputs joint space control commands, including the target joint position increment or target joint torque of the robot's underlying motors. S5. Cognitive-Motor Dual Closed-Loop Compliant Execution: The joint space control commands output by the VLA model in step S4 are combined with the underlying admittance or impedance control model to control the upper limb robot to compliantly assist the user in performing corresponding actions.

2. The dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described in claim 1, characterized in that: The real-time physical state in the multimodal task scene perception step S1 Represented as: in, For robot joint angles, The joint angular velocity, The human-computer interaction force is measured by a six-dimensional force / torque sensor at the end.

3. The dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described in claim 2, characterized in that: The thought chain sequence in step S2, based on multidimensional state word tuples and low-rank adaptive full-modal space alignment, is represented as follows: in, Indicates the first Each sub-task reasoning step Represents a thought chain sequence. This represents the reasoning step of the i-th subtask. This represents the total number of sub-task reasoning steps contained in the thought chain sequence; its joint conditional probability distribution formula is: in, This represents a sequence of visual lexical units obtained from the encoding of a visual image. This represents the sequence of language lexical units encoded by the instructions of a natural language task.

4. The dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described in claim 3, characterized in that: Step S3 is based on the cognitive engagement score in the dynamic intervention mechanism of cognitive alignment assessment. Represented as: in, The response delay generates effective motion trends for users. The magnitude of the vector representing the deviation between the user's actual force direction and the intended trajectory direction. The force threshold, , , , These are the weights and adjustment coefficients.

5. The dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described in claim 4, characterized in that: Step S4 is based on the output time in the reparameterized end-to-end joint space motion generation. Target joint position increment Or target joint torque mapping function Represented as: in, This is a neural network mapping function that has undergone efficient parameter fine-tuning and weight merging.

6. The dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described in claim 5, characterized in that: Step S4, based on the reparameterized end-to-end joint space motion generation, employs a network architecture based on discretized motion space or a diffusion strategy to predict motion sequences at multiple future time steps, with its joint space increment prediction... Represented as: in, To predict the step size.

7. The dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described in claim 6, characterized in that: In step S5, the cognition-motion dual closed-loop compliant execution, after combining the underlying admittance or impedance control model, the joint torque ultimately executed by the robotic arm... Represented as: in, , These are the stiffness and damping matrices, respectively. This is the transpose of the Jacobian matrix for the robot.

8. A control system for an upper limb robot based on VLA model and cognitive alignment with thought chain, characterized in that, include: The multimodal perception module is used to acquire environmental, natural language commands, and physical state information through a third-view depth camera and an end-effector six-dimensional force sensor. The lexicalization and semantic loop closure module is used to map multidimensional data into lexical sequences, generate thought chains using a VLA model injected with LoRA, and output perceptual cue signals. The dynamic intervention assessment module is used to calculate the cognitive engagement score within the cognitive response time window, determine whether to trigger strategy downgrade, and re-enter the cognitive response assessment after the downgrade strategy is output. The motion reparameterization generation module is used to directly output control commands in the joint space after merging weights; The dual closed-loop execution module is used to combine VLA feedforward commands with underlying impedance feedback to drive the upper limb robot.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements any step in the dual-arm robot control method based on VLA model and cognitive alignment of thought chain as described in any one of claims 1 to 7.

Images