Mechanical arm control system and method based on multi-sensor fusion and large language model

Abstract

The invention relates to the technical field of intelligent robot and multi-sensor fusion crossing, in particular to a mechanical arm control system and method based on multi-sensor fusion and a large language model. The system comprises an input module, a sensing layer, a depth camera, a visual language large model, a virtual simulation layer and an execution layer. The system can adapt to instruction requirements of various different operation scenes, the limitation that a traditional system depends on a preset program and instruction analysis is disjointed with the scenes is effectively solved, and the system adopts a dual-guarantee mechanism combining multi-sensor real-time feedback with virtual simulation verification; in cooperation with collaborative optimization of a residual convolutional network, a physical information neural network and a deep reinforcement learning algorithm, the operation precision and the trajectory optimization precision are remarkably superior to those of a traditional scheme; the collision risk can be effectively reduced through a dual-protection system constructed by intelligent barrier segmentation and the limit switch, meanwhile, each module adopts a modular design, later maintenance and function upgrading are facilitated, and the system has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot and multi-sensor fusion technology, and in particular to a robotic arm control system and method based on multi-sensor fusion and large language model. Background Technology

[0002] Automated operations are a core requirement in modern industry and service sectors. In scenarios such as industrial assembly, logistics sorting, and precision manufacturing, robotic arms serve as core execution devices, and their operational accuracy, response flexibility, and environmental adaptability directly determine production efficiency and service quality.

[0003] In such scenarios, operators typically require robotic arms to complete diverse tasks. However, traditional robotic arms often rely on preset programs or simple command control. When faced with complex natural language commands such as "grab the red workpiece on the left and place it on the tray on the right," they struggle to accurately interpret the semantic logic. Furthermore, during robotic arm operation, the state of its joints (e.g., angle, speed) and environmental information (e.g., workpiece position, obstacle distribution) are dynamically changing. In existing technologies, data from state perception sensors and environmental perception sensors are often processed independently and not deeply coupled with the command parsing module. This prevents the robotic arm from adjusting its operational strategy in real time based on its own state and environmental changes, leading to operational deviations or collision risks. Additionally, existing systems often rely on single limit switches for safety protection, lacking a predictive mechanism that combines speed, torque, and other multi-dimensional data. In scenarios involving workpiece position deviation or sudden obstacles, the dynamic correction and safety protection capabilities are insufficient. Therefore, we propose a robotic arm control system and method based on multi-sensor fusion and a large language model. Summary of the Invention

[0004] The purpose of this invention is to provide a robotic arm control system and method based on multi-sensor fusion and large language model to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] The robotic arm control system based on multi-sensor fusion and large language model includes an input module, a perception layer, a depth camera, a visual language large model, a virtual simulation layer, and an execution layer. The input module is a speech recognition microphone array, which is used to receive natural language commands and convert them into digital signals.

[0007] The perception layer includes a state perception unit and an environment perception unit. The environment perception unit integrates a depth camera. The state perception unit is associated with the sensors of the execution layer.

[0008] The large visual language model and the virtual simulation layer together constitute the decision-making layer.

[0009] Preferably, the execution layer includes a robotic arm body and a control module. Each joint of the robotic arm body is equipped with a speed sensor and an angle sensor. A force sensor is installed on the end effector of the robotic arm body. The robotic arm body and the control module are connected through a drive circuit.

[0010] Preferably, the virtual simulation layer integrates the SAC reinforcement learning algorithm, residual convolutional network, and physical information neural network module, and the physical information neural network module embeds the kinematic equation constraints of the robotic arm body; the control module is an MCU microcontroller, which is connected to the virtual simulation layer via a CAN bus.

[0011] Preferably, the visual language big model is connected to the input module and the perception layer via ROS communication to receive language commands and multi-sensor data in real time. The virtual simulation layer communicates with the visual language big model via Ethernet, receives the decision information output by the visual language big model, completes optimization, and then transmits the optimized control parameters to the execution layer.

[0012] Preferably, the data from the depth camera, the speed sensor of the robotic arm, the angle sensor of the robotic arm, and the force sensor of the end effector are converted from analog to digital and then aggregated to the data acquisition card. The data is then transmitted to the visual language large model via ROS communication for real-time interaction.

[0013] A robotic arm control method based on multi-sensor fusion and large language model is applicable to robotic arm control systems based on multi-sensor fusion and large language model, and includes the following steps:

[0014] S1: Receives text or voice commands through the input module and processes the voice commands into standardized text through the voice processing sub-model;

[0015] S2: The perception layer collects real-time data on the robot arm's own status and the working environment, and adds a unified timestamp after preprocessing.

[0016] S3: The decision-making layer's visual language big model fuses multimodal data and parses instructions to output the planned global path;

[0017] S4: The robotic arm executes actions based on the optimized control parameters and feeds back data to the perception layer in real time, forming a closed-loop control.

[0018] Preferably, the specific steps in S4 are as follows:

[0019] Step 1: The optimized control parameters are transmitted to the control module;

[0020] Step 2: The control module converts the parameters into drive signals to control the robotic arm to perform grasping, transporting, and placing actions;

[0021] Step 3: During execution, the perception layer collects the joint status of the robotic arm body, the force on the end effector, and the scene data from the depth camera in real time, and feeds it back to the visual language big model and virtual simulation layer to dynamically fine-tune the key nodes of the global path and the local trajectory parameters until the task is completed.

[0022] Preferably, the virtual simulation layer verifies the feasibility of the path using the Mujoco engine, and combines the SAC algorithm, residual convolutional network, and physical information neural network module to optimize the local trajectory. The residual convolutional network includes depthwise separable convolution and inverted residual blocks, which are used to extract environmental obstacles and the state features of the robotic arm itself.

[0023] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0024] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0025] 1. This invention leverages the fusion advantages of a large visual language model and a speech processing sub-model to accurately parse complex natural language instructions. By combining real-time scene perception data, it achieves deep coupling between semantics and scene, adapting to the instruction requirements of various different work scenarios and effectively solving the limitations of traditional systems that rely on preset programs and whose instruction parsing is disconnected from the scene.

[0026] 2. This invention adopts a dual guarantee mechanism of real-time feedback from multiple sensors combined with virtual simulation verification. With the collaborative optimization of residual convolutional networks, physical information neural networks and SAC algorithms, trajectory parameters can be dynamically adjusted to correct deviations and avoid risks, making the operation accuracy and trajectory optimization accuracy significantly better than traditional solutions. Through the dual protection system constructed by intelligent obstacle segmentation and limit switches, the risk of collision can be effectively reduced, and the dynamic correction response efficiency is significantly better than traditional single sensor control solutions.

[0027] 3. This invention utilizes a lightweight architecture combining residual convolutional networks and physical information neural networks to meet real-time control requirements. The residual structure ensures efficient feature extraction, while the physical information neural network, by embedding robotic arm dynamics constraints, ensures the physical rationality of the optimization results. Coupled with an online fine-tuning mechanism, it can quickly adapt to different workpiece characteristics and scene changes, significantly improving scene adaptability compared to traditional optimization schemes. Furthermore, the system adopts a modular design, facilitating later maintenance and functional upgrades. Its applicable scenarios cover multiple fields such as industry, logistics, and services, demonstrating broad application prospects. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of the intelligent robotic arm control method of the present invention;

[0030] Figure 2 This is a schematic diagram of the layout of the sensing module of the present invention;

[0031] Figure 3 This is a schematic diagram of the embodied intelligent multimodal design of the present invention;

[0032] Figure 4 This is a diagram of the residual convolutional network of the present invention;

[0033] Figure 5 This is a schematic diagram of the robotic arm execution layer of the present invention;

[0034] Figure 6 This is a flowchart illustrating the task execution process of the robotic arm of the present invention.

[0035] The attached diagram lists the components represented by each number as follows:

[0036] In the diagram: 1. Input module; 2. Perception layer; 3. Depth camera; 4. Visual language large model; 5. Virtual simulation layer; 6. Execution layer. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0038] Example 1

[0039] Reference Figure 1-3 and Figure 5 The robotic arm control system based on multi-sensor fusion and large language model includes an input module 1, a perception layer 2, a depth camera 3, a visual language large model 4, a virtual simulation layer 5, and an execution layer 6. The input module 1 is a voice recognition microphone array, which is used to receive natural language commands and convert them into digital signals.

[0040] The perception layer 2 includes a state perception unit and an environment perception unit. The environment perception unit integrates a depth camera 3, and the state perception unit is associated with the sensors of the execution layer 6.

[0041] The visual language big model 4 and the virtual simulation layer 5 together constitute the decision layer; in other embodiments, the decision layer can be replaced by a BERT-based visual-language fusion model to reduce computing power requirements.

[0042] The execution layer 6 includes a robotic arm body and a control module. Speed ​​sensors and angle sensors are installed at each joint of the robotic arm body, and force sensors are installed on the end effector of the robotic arm body. The robotic arm body and the control module are connected through drive lines. In other embodiments, the control module can be replaced by a PLC controller to improve the anti-electromagnetic interference capability and adapt to the strong interference environment in the industry.

[0043] The virtual simulation layer 5 integrates the SAC reinforcement learning algorithm, residual convolutional network, and physical information neural network module. The physical information neural network module embeds the kinematic equation constraints of the robotic arm body. The control module is an MCU microcontroller, which is connected to the virtual simulation layer 5 via a CAN bus. In other embodiments, the SAC algorithm in the virtual simulation layer 5 can be replaced with the PPO algorithm, i.e., the near-end policy optimization algorithm; the residual convolutional network can be replaced with DenseNet, i.e., a densely connected network; and PINNs, i.e., physical information neural network, can be replaced with a combination of "traditional numerical simulation + physical constraint regularization". The overall mode of residual convolutional network combined with PINNs can be replaced with a lightweight Transformer combined with a physical constraint module to adapt to the optimization needs of different computing power scenarios.

[0044] The visual language big model 4 is connected to the input module 1 and the perception layer 2 via ROS communication, receiving language commands and multi-sensor data in real time. The virtual simulation layer 5 communicates with the visual language big model 4 via Ethernet, receiving the decision information output by the visual language big model 4 and performing optimization, and then transmitting the optimized control parameters to the execution layer 6. In other embodiments, alternatives to the perception layer 2 are: the visual sensor in the environment perception unit can be replaced with a lidar, which realizes workpiece recognition and positioning through point cloud data and is suitable for working scenarios with poor lighting conditions; the angle sensor in the state perception unit can be replaced with a Hall sensor, which reduces costs while meeting the requirements of medium-precision operation.

[0045] Data from depth camera 3, speed sensor of robotic arm body, angle sensor of robotic arm body, and force sensor of end effector are converted from analog to digital and then aggregated to data acquisition card. They are then transmitted to visual language large model 4 via ROS communication for real-time interaction. The end effector is a mechanical gripper with a force sensor installed on it.

[0046] Example 2

[0047] Reference Figure 6A robotic arm control method based on multi-sensor fusion and large language model is applicable to robotic arm control systems based on multi-sensor fusion and large language model, and includes the following steps:

[0048] S1: Receives text or voice commands through input module 1, and processes the voice commands into standardized text through the voice processing sub-model;

[0049] S2: Perception layer 2 collects real-time data on the robot arm's own status and the working environment, and adds a unified timestamp after preprocessing.

[0050] S3: The visual language big model 4 of the decision layer fuses multimodal data and parses instructions to output the planned global path; in other embodiments, the visual language big model 4 can be replaced by a BERT-based visual-language fusion model to reduce hardware computing power requirements while ensuring instruction parsing accuracy.

[0051] S4: The robotic arm executes actions based on the optimized control parameters and feeds back data to the perception layer 2 in real time, forming a closed-loop control.

[0052] The specific steps in S4 are as follows:

[0053] Step 1: The optimized control parameters are transmitted to the control module;

[0054] Step 2: The control module converts the parameters into drive signals to control the robotic arm to perform grasping, transporting, and placing actions;

[0055] Step 3: During execution, the perception layer 2 collects the joint status of the robotic arm body, the force on the end effector, and the scene data from the depth camera 3 in real time. Speed ​​sensors and angle sensors are installed at each joint of the robotic arm body, and force sensors are installed on the end effector. The data is fed back to the visual language large model 4 and the virtual simulation layer 5, and the global path key nodes and local trajectory parameters are dynamically fine-tuned until the task is completed.

[0056] The virtual simulation layer 5 verifies path feasibility using the Mujoco engine and combines the SAC algorithm, residual convolutional network, and physical information neural network module for local trajectory optimization. The residual convolutional network includes depthwise separable convolutions and inverted residual blocks to extract environmental obstacles and robotic arm body state features. The residual convolutional network adopts a "downsampling-feature extraction-upsampling" architecture. The physical information neural network embeds physical constraints such as the robotic arm body kinematic equations. After the two work together to generate accurate correction features, the SAC algorithm iteratively optimizes the local trajectory parameters, forming a triple optimization mechanism of "feature extraction-physical constraints-reinforcement learning".

[0057] Example 3

[0058] like Figure 4 As shown, the selection and configuration of each module in this example are as follows: Input module 1 uses, for example, the iFlytek AI microphone array; the environment perception unit of perception layer 2 integrates depth camera 3 and is associated with the sensor of execution layer 6 through state perception unit; the visual language large model 4 uses GPT-4V; the virtual simulation layer 5 is based on the Ubuntu system with the Mujoco engine, integrating SAC reinforcement learning algorithm (reward function includes position error and energy consumption weights), residual convolutional network and physical information neural network module (embedded with constraints of the kinematic equations of the robotic arm body); the control module uses, for example, the STM32F103 MCU; the robotic arm body and control module of execution layer 6 have absolute encoders (angle sensors) and photoelectric encoders (speed sensors) installed at their joints, and multi-dimensional force sensors installed at the end effector, resulting in excellent repeatability of the robotic arm body.

[0059] During operation, input module 1 receives the instruction "grab the red workpiece on the left side of the assembly line and place it on pallet 3 on the right side." After being processed into standardized text by the Whisper-large-v3 sub-model, it is transmitted to the visual language large model 4. Simultaneously, perception layer 2 starts, and its environment perception unit RGB-D depth camera 3 captures scene images at a preset frequency and simultaneously generates raw point cloud data. The state perception unit collects the initial joint angles, joint speeds, end-effector unloaded force, and limit states of the robotic arm. All data is simultaneously transmitted to the visual language large model 4. The visual language large model 4 fuses the instruction and perception data to complete semantic parsing, based on the built-in RRT-connect path planning module and mechanical... Arm joint constraints generate a global path (output as a global joint angle sequence) in a complex scene: "initial position → above the workpiece → gripping point → above the tray → placement point." This global path provides a macroscopic framework and boundary constraints for subsequent local planning. Virtual simulation layer 5 first uses the Mujoco engine to reproduce the scene and the robot arm's dynamic characteristics, verifying the kinematic reachability (e.g., whether joint angles exceed limits) and collision-free nature of the global path. After successful verification, the local path planning process is initiated. At this point, the global path output by the visual language large model 4 and the local path planned by the SAC algorithm form a hierarchical relationship of "global guidance - local optimization" and an interactive mechanism of "dynamic feedback - bidirectional adaptation," specifically manifested as follows:

[0060] 1. Direct Relationship: The global path is the foundation and constraint boundary of the local path. The SAC algorithm needs to plan fine-grained local trajectories between key nodes of the global path (such as grab points and placement points) and must not exceed the task objectives and macro-level obstacle avoidance range set by the global path. The local path is the refinement and implementation of the global path, transforming the discrete joint angle sequence of the global path into continuous and smooth joint motion curves (including parameters such as acceleration and velocity thresholds) to ensure the accurate execution of the global task objectives.

[0061] 2. Interactions: ① Global guidance and input to local: The coordinates of key nodes and joint constraints of the global path output by the Visual Language Model 4 are directly used as the state initialization conditions and core weights of the reward function of the SAC algorithm (e.g., the distance from the global node is set as a penalty). At the same time, the obstacle bounding box information associated with the global path provides prior information for the SAC algorithm's local obstacle avoidance. ② Local feedback and correction to the global: During the local optimization process, the SAC algorithm calculates the robot arm's body dynamics response in real time through the Mujoco engine (e.g., whether the joint torque exceeds the limit) and combines the dynamic scene data fed back by the perception layer 2 (e.g., small workpiece offset). If a sudden occurrence is found in a local area that is not covered by the global path, the algorithm will correct the error. Obstacles or dynamic conflicts are detected, and the conflict information is fed back to the visual language big model 4, triggering fine-tuning of key nodes in the global path (such as fine-tuning the hovering position above the gripping point), forming a closed loop of "global planning - local verification - conflict feedback - global correction"; ③ Collaborative adaptation to complex scenarios: The visual language big model 4 processes macroscopic scenarios (such as multiple workpieces and large-scale obstacles) based on multimodal data processing to generate a robust global path; The SAC algorithm combines local features extracted by the residual convolutional network (such as joint state deviation and small-sized obstacles) and dynamic constraints embedded in the physical information neural network to optimize the smoothness and real-time performance of local trajectories within the global framework. The collaboration of the two enables the system to adapt to changes in macroscopic scenarios while ensuring the accuracy of local operations. The optimized local control parameters are transmitted to the control module; the control module converts the parameters into drive signals to control the robotic arm to perform gripping, transporting, and placing actions; During execution, the perception layer 2 collects the joint states of the robotic arm, the force on the end effector, and scene data from the depth camera 3 in real time, and feeds them back to the visual language big model 4 and the virtual simulation layer 5 to dynamically fine-tune key nodes in the global path and local trajectory parameters until the task is completed.

[0062] In summary:

[0063] This invention addresses the following technical problems: In existing technologies, data from state perception sensors and environmental perception sensors are often processed independently and not deeply coupled with the instruction parsing module. This results in the robotic arm being unable to adjust its operational strategy in real time based on its own state and environmental changes, easily leading to operational deviations or collision risks. Furthermore, existing systems often rely on single limit switches for safety protection, lacking a predictive mechanism that combines multi-dimensional data such as speed and torque. In scenarios involving workpiece position deviations or sudden obstacles, the dynamic correction and safety protection capabilities are insufficient. By adopting the technical solutions of the above embodiments and through the aforementioned settings, this application can certainly solve the above-mentioned technical problems and simultaneously achieve the following technical effects:

[0064] 1. This invention relies on the fusion advantages of the visual language big model 4 and the speech processing sub-model to accurately parse complex natural language instructions. Combined with real-time scene perception data, it achieves deep coupling between semantics and scene, and can adapt to the instruction requirements of various different work scenarios. It effectively solves the limitations of traditional systems that rely on preset programs and whose instruction parsing is disconnected from the scene.

[0065] 2. This invention adopts a dual guarantee mechanism of real-time feedback from multiple sensors combined with virtual simulation verification. With the collaborative optimization of residual convolutional networks, physical information neural networks and SAC algorithms, trajectory parameters can be dynamically adjusted to correct deviations and avoid risks, making the operation accuracy and trajectory optimization accuracy significantly better than traditional solutions. Through the dual protection system constructed by intelligent obstacle segmentation and limit switches, the risk of collision can be effectively reduced, and the dynamic correction response efficiency is significantly better than traditional single sensor control solutions.

[0066] 3. This invention utilizes a lightweight architecture combining residual convolutional networks and physical information neural networks to meet real-time control requirements. The residual structure ensures efficient feature extraction, while the physical information neural network, by embedding robotic arm dynamics constraints, ensures the physical rationality of the optimization results. Coupled with an online fine-tuning mechanism, it can quickly adapt to different workpiece characteristics and scene changes, significantly improving scene adaptability compared to traditional optimization schemes. Furthermore, the system adopts a modular design, facilitating later maintenance and functional upgrades. Its applicable scenarios cover multiple fields such as industry, logistics, and services, demonstrating broad application prospects.

[0067] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0068] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A robotic arm control system based on multi-sensor fusion and a large language model, characterized in that, It includes an input module (1), a perception layer (2), a depth camera (3), a large visual language model (4), a virtual simulation layer (5), and an execution layer (6). The input module (1) is a speech recognition microphone array. The input module (1) is used to receive natural language instructions and convert them into digital signals. The perception layer (2) includes a state perception unit and an environment perception unit. The environment perception unit integrates a depth camera (3). The state perception unit is associated with the sensors of the execution layer (6). The visual language big model (4) and the virtual simulation layer (5) together constitute the decision layer.

2. The robotic arm control system based on multi-sensor fusion and large language model according to claim 1, characterized in that, The execution layer (6) includes a robotic arm body and a control module. Speed ​​sensors and angle sensors are installed at each joint of the robotic arm body. Force sensors are installed on the end effector of the robotic arm body. The robotic arm body and the control module are connected through drive lines.

3. The robotic arm control system based on multi-sensor fusion and large language model according to claim 2, characterized in that, The virtual simulation layer (5) integrates the SAC reinforcement learning algorithm, residual convolutional network and physical information neural network module. The physical information neural network module is embedded with the kinematic equation constraint of the robot arm body. The control module is an MCU microcontroller, which is connected to the virtual simulation layer (5) via CAN bus.

4. The robotic arm control system based on multi-sensor fusion and large language model according to claim 2, characterized in that, The visual language big model (4) is connected to the input module (1) and the perception layer (2) via ROS communication, and receives language instructions and multi-sensor data in real time. The virtual simulation layer (5) communicates with the visual language big model (4) via Ethernet, receives the decision information output by the visual language big model (4) and completes the optimization, and then transmits the optimized control parameters to the execution layer (6).

5. The robotic arm control system based on multi-sensor fusion and large language model according to claim 4, characterized in that, The data from the depth camera (3), the speed sensor of the robotic arm body, the angle sensor of the robotic arm body, and the force sensor of the end effector are converted from analog to digital and then aggregated to the data acquisition card. They are then transmitted to the visual language big model (4) via ROS communication for real-time interaction.

6. A robotic arm control method based on multi-sensor fusion and large language model, applicable to the robotic arm control system based on multi-sensor fusion and large language model as described in any one of claims 1-5, characterized in that, Includes the following steps: S1: Receive text or voice commands through input module (1) and process the voice commands into standardized text through the voice processing sub-model; S2: The perception layer (2) collects real-time data on the status of the robotic arm itself and the working environment, and adds a unified timestamp after preprocessing. S3: The visual language big model of the decision layer (4) fuses multimodal data and parses instructions to output the planned global path; S4: The robotic arm body performs actions according to the optimized control parameters and feeds back data to the perception layer (2) in real time to form a closed-loop control.

7. The robotic arm control method based on multi-sensor fusion and large language model according to claim 6, characterized in that, The specific steps in S4 are as follows: Step 1: The optimized control parameters are transmitted to the control module; Step 2: The control module converts the parameters into drive signals to control the robotic arm to perform grasping, transporting, and placing actions; Step 3: During the execution process, the perception layer (2) collects the joint status of the robotic arm body, the force of the end effector and the scene data of the depth camera (3) in real time, and feeds them back to the visual language big model (4) and the virtual simulation layer (5) to dynamically fine-tune the key nodes of the global path and the local trajectory parameters until the task is completed.

8. The robotic arm control method based on multi-sensor fusion and large language model according to claim 7, characterized in that, The virtual simulation layer (5) verifies the feasibility of the path through the Mujoco engine, and performs local trajectory optimization by combining the SAC algorithm, residual convolutional network and physical information neural network module. The residual convolutional network contains depth-separable convolution and inverted residual blocks, which are used to extract environmental obstacles and the state features of the robotic arm.

Images