Robotic arm control method and device based on visual language model, and medium

Abstract

The application discloses a mechanical arm control method and device based on a visual language model and a medium, relates to the technical field of robot control, and comprises the following steps: obtaining fused space-time features based on a voice control instruction, a visual image and mechanical arm state data; inputting the fused space-time features, the visual image and the mechanical arm state data into a function head to obtain an initial mechanical arm control instruction, a current grabbing state classification result and a corresponding first occurrence probability, and a predicted grabbing state classification result and a corresponding second occurrence probability; and feeding back the above information to an action decoder, which adjusts the initial mechanical arm control instruction based on the feedback information and outputs an adjusted mechanical arm control instruction. The application is used to solve the problems of low precision and poor reliability in robot grabbing control in the prior art, and realizes high-precision and high-reliability grabbing operation.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a method, device and medium for controlling a robotic arm based on a visual language model. Background Technology

[0002] With the widespread application of robotics technology in industrial manufacturing, logistics sorting, and service robots, the level of intelligent operation of robotic arms has become a core key to improving operational efficiency and reliability. Among these, Visual Language Models (VLMs), with their powerful visual understanding and language command interaction capabilities, are widely used in the field of robotic arm motion control, forming Visual Language Action (VLA) robot systems. The widespread adoption of VLM bases (such as LLaVA, OpenVLA, and MiniGPT-4) has further lowered the development threshold for VLA robots, enabling robotic arms to achieve target recognition, motion planning, and trajectory control through language commands, significantly improving operational flexibility and convenience.

[0003] Existing VLA robot systems based on VLM bases focus on visual-language alignment, motion decoding, and motion control. They plan and execute robotic arm movements based solely on verbal instructions and single-frame visual information. Furthermore, the grasping state estimation and robot control are independent of each other, resulting in complex systems, high latency, and poor reliability. Consequently, they cannot meet the requirements for high-precision and high-reliability grasping operations. Summary of the Invention

[0004] In response to the aforementioned problems and technical requirements, the applicant proposes a robotic arm control method, device, and medium based on a visual language model to solve the problems of low accuracy and poor reliability in existing technologies for robot grasping control, thereby enabling high-precision and high-reliability completion of grasping operations.

[0005] This application provides a robotic arm control method based on a visual language model. The visual language model includes a data processing module and a functional head. The functional head includes a current state prediction head, a future state prediction head, and an action decoder. The method includes: The system acquires voice control commands from the user to perform grasping operations, and the robotic arm uses real-time visual images and real-time robotic arm status data to grasp the target based on the voice control commands. Voice control commands, visual images, and robotic arm status data are input into the data processing module in real time to obtain the fused spatiotemporal features output by the data processing module. The function head is fed with spatiotemporal features, visual images and robotic arm state data to obtain the initial robotic arm control command generated by the motion decoder, the current grasping state classification result and the corresponding first occurrence probability output by the current state prediction head, and the predicted grasping state classification result and the corresponding second occurrence probability at the future preset interval output by the future state prediction head. The current grasping state classification result, the first occurrence probability, the predicted grasping state classification result and the second occurrence probability are fed back to the motion decoder. The motion decoder adjusts the initial robotic arm control command based on the feedback information and outputs the adjusted robotic arm control command.

[0006] According to the robotic arm control method based on visual language model provided in the embodiments of this application, the functional head further includes: a physical prior constraint unit and a prediction unit; The data, which integrates spatiotemporal features, visual images, and robotic arm state data, is input into the functional head to obtain the current grasping state classification result and corresponding first probability of occurrence output by the current state prediction head, and the predicted grasping state classification result and corresponding second probability of occurrence output by the future state prediction head at a preset future interval, including: The spatiotemporal features are input into the prediction unit to obtain the prediction parameters output by the prediction unit. The prediction parameters include the contact reaction force and contact reaction torque of the target to be grasped on the robotic arm. The system integrates spatiotemporal features, visual images, and robotic arm state data into the functional head to obtain the current grasping state classification result, first occurrence probability, and first prediction label output by the current state prediction head, as well as the predicted grasping state classification result, second occurrence probability, and second prediction label output by the future state prediction head. Based on the first and second prediction tags, the robotic arm state data and prediction parameters are input into the physical prior constraint unit to obtain the mechanical rule determination result output by the physical prior constraint unit. Given that the determination result of the mechanical rule conforms to the physical law, we obtain the current grasping state classification result and the corresponding first occurrence probability, as well as the predicted grasping state classification result and the corresponding second occurrence probability.

[0007] According to the robotic arm control method based on a visual language model provided in the embodiments of this application, after obtaining the mechanical rule determination result output by the physical prior constraint unit, the method further includes: When the determination result of the mechanical rule does not conform to the physical law, based on the new visual image and the state data of the robotic arm, the step of inputting the voice control command, visual image and robotic arm state data into the data processing module in real time is executed to obtain the fused spatiotemporal features output by the data processing module.

[0008] According to the robotic arm control method based on visual language model provided in the embodiments of this application, the spatiotemporal features are integrated, including the physical properties of the target to be grasped, and the physical properties include: deformability; After obtaining the mechanical rule determination results output by the physical prior constraint unit, the following is also included: Given that the mechanical rule determination result conforms to physical laws, and given that the physical properties are variable, the image data is input into the physical prior constraint unit to obtain the predicted contact force output by the physical prior constraint unit. The predicted contact force and spatiotemporal features are spliced ​​together and input into a preset spatiotemporal convolutional network to obtain the constraint prediction classification result and the corresponding third occurrence probability output by the spatiotemporal convolutional network. Assuming the mechanical rule determination result conforms to physical laws, the current grasping state classification result and its corresponding first probability of occurrence are obtained, along with the predicted grasping state classification result and its corresponding second probability of occurrence, including: If the consistency between the preset predicted classification result and the current grasping state classification result, and the consistency between the first occurrence probability and the third occurrence probability both meet the preset consistency criteria, then the current grasping state classification result and the corresponding first occurrence probability, as well as the predicted grasping state classification result and the corresponding second occurrence probability, are obtained.

[0009] According to the robotic arm control method based on a visual language model provided in the embodiments of this application, the robotic arm state data and predicted parameters are input into a physical prior constraint unit to obtain the mechanical rule determination result output by the physical prior constraint unit, including: Input the robotic arm state data and prediction parameters into the mechanical rule determination formula to obtain the mechanical rule determination result output by the mechanical rule determination formula; The formulas for determining mechanical rules include: ; in, This indicates the first result in the determination of the mechanical rules. Indicates the clamping force of the grippers. Represents the contact reaction force, The gravity of the target to be grasped is a constant. This represents the second result in the determination of mechanical rules. Indicates joint torque. This indicates the contact reaction torque.

[0010] According to the robotic arm control method based on a visual language model provided in the embodiments of this application, image data is input into a physical prior constraint unit to obtain the predicted contact force output by the physical prior constraint unit, including: The object's contact deformation and deformation rate are obtained by parsing the image data and input into the soft contact dynamics formula to obtain the predicted contact force output by the soft contact dynamics formula; The soft contact dynamics formulas include: ; in, Indicates the predicted contact force. This represents the contact stiffness coefficient, which is a constant. This represents the contact damping coefficient, determined based on the material of the target to be grasped. Indicates the deformation of an object upon contact. The nonlinear deformation exponent is a constant. Indicates the deformation rate.

[0011] According to the visual language model-based robotic arm control method provided in the embodiments of this application, the motion decoder adjusts the initial robotic arm control commands based on feedback information, including: If the probability of slight relative slippage and severe slippage in any one or more of the current grasping state classification results and the predicted grasping state classification results is greater than the preset probability, increase the gripper clamping force and reduce the end effector speed. If the probability of abnormal gripping force in any one or more of the current gripping state classification results and the predicted gripping state classification results is greater than the preset probability, adjust the gripping force of the gripper to a preset reasonable range. If any one or more of the current grasping state classification results and the predicted grasping state classification results indicate that the object is about to fall off, stop the current action, plan a new grasping trajectory, and repeat the grasping action.

[0012] According to the robotic arm control method based on visual language model provided in the embodiments of this application, the data processing module includes: an input unit and a VLM backbone network; Voice control commands, visual images, and robotic arm status data are input into the data processing module to obtain the fused spatiotemporal features output by the data processing module, including: Voice control commands, visual images, and robotic arm status data are input into the input unit. The input unit performs preset preprocessing operations on the input data and inputs the processed data into the VLM backbone network. The voice control commands are tokenized by the language encoder to obtain language feature vectors. The visual image is extracted by the visual encoder to obtain visual feature vectors. The robotic arm status data is extracted by the ontology temporal feature projection layer to obtain ontology temporal feature vectors. The language feature vectors, visual feature vectors, and ontology temporal feature vectors are input into the spatiotemporal Transformer encoder and adaptively fused based on a three-way cross-attention mechanism to obtain fused spatiotemporal features.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the visual language model-based robotic arm control method as described above.

[0014] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the visual language model-based robotic arm control method as described above.

[0015] The robotic arm control method, device, and medium based on a visual language model provided in this application embodiment include: a data processing module and a function head. The function head includes: a current state prediction head, a future state prediction head, and an action decoder. The method acquires user-inputted voice control commands requiring the robotic arm to perform grasping operations. The robotic arm performs target grasping based on real-time visual images and real-time robotic arm state data. The voice control commands, visual images, and robotic arm state data are input into the data processing module in real-time to obtain fused spatiotemporal features output by the data processing module. The fused spatiotemporal features, visual images, and robotic arm state data are input into the function head to obtain the initial robotic arm control commands generated by the action decoder and the current state prediction head. The current grasping state classification result and its corresponding first probability of occurrence are output, as well as the predicted grasping state classification result and its corresponding second probability of occurrence at a future preset interval output by the future state prediction head. The current grasping state classification result, the first probability of occurrence, the predicted grasping state classification result, and the second probability of occurrence are fed back to the motion decoder. The motion decoder adjusts the initial robotic arm control command based on the feedback information and outputs the adjusted robotic arm control command. This application obtains fused spatiotemporal features by fusing multimodal data, and predicts the current grasping state and the future grasping state based on the fused spatiotemporal features, thereby further adjusting the initial robotic arm control command to ensure the accuracy of the robotic arm control command output, and achieves the goal of completing the grasping operation with high precision and high reliability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the robotic arm control method based on a visual language model provided in an embodiment of this application. Figure 2This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0019] This application provides a robotic arm control method based on a visual language model. The visual language model includes a data processing module and a functional head. The functional head includes a current state prediction head, a future state prediction head, and an action decoder. This method can be applied to smart terminals and servers. This application uses the application of this method in a server as an example for illustration. Other descriptions in the embodiments are illustrative and not intended to limit the scope of protection of this application, and will not be described in detail thereafter. The specific implementation of the method is as follows... Figure 1 As shown: Step 101: Obtain the voice control command input by the user requesting the robotic arm to perform a grasping operation. The robotic arm performs target grasping based on the voice control command, and obtains real-time visual images and real-time robotic arm status data.

[0020] Step 102: Input the voice control commands, visual images and robotic arm status data into the data processing module in real time to obtain the fused spatiotemporal features output by the data processing module.

[0021] For example, the voice control command is to grab the red wooden block on the table and place it in the designated location. Visual images include continuous visual frames (including RGB and depth images) captured by industrial cameras during the robotic arm's grasping process. Robotic arm status data includes continuous data collected by the robotic arm's built-in sensors (joint encoders, torque sensors, gripper force sensors, and pose sensors, etc.), specifically including joint angles, joint angular velocities, joint torques, gripper opening, gripper clamping force, end effector pose, and end effector velocity.

[0022] Step 103: Input the fused spatiotemporal features, visual images and robotic arm state data into the function head to obtain the initial robotic arm control command generated by the action decoder, the current grasping state classification result and the corresponding first occurrence probability output by the current state prediction head, and the predicted grasping state classification result and the corresponding second occurrence probability at the future preset interval output by the future state prediction head.

[0023] Step 104: Feed back the current grasping state classification result, the first occurrence probability, the predicted grasping state classification result and the second occurrence probability to the motion decoder. The motion decoder adjusts the initial robotic arm control command based on the feedback information and outputs the adjusted robotic arm control command.

[0024] The robotic arm control commands include: joint angle control commands, gripper clamping force control commands, and end effector trajectory control commands.

[0025] The robotic arm control method based on a visual language model provided in this application acquires a user-inputted voice control command requesting the robotic arm to perform a grasping operation. The robotic arm then performs target grasping based on the voice control command, capturing real-time visual images and real-time robotic arm state data. The voice control command, visual images, and robotic arm state data are input into a data processing module in real-time to obtain fused spatiotemporal features output by the data processing module. These fused spatiotemporal features, visual images, and robotic arm state data are then input into a function head to obtain the initial robotic arm control command generated by the action decoder, the current grasping state classification result and corresponding first occurrence probability output by the current state prediction head, and future... The state prediction head outputs the predicted grasping state classification result and the corresponding second occurrence probability at a future preset interval. The current grasping state classification result, the first occurrence probability, the predicted grasping state classification result, and the second occurrence probability are fed back to the motion decoder. The motion decoder adjusts the initial robotic arm control command based on the feedback information and outputs the adjusted robotic arm control command. This application obtains fused spatiotemporal features by fusing multimodal data, and predicts the current grasping state and future grasping state based on the fused spatiotemporal features, thereby further adjusting the initial robotic arm control command to ensure the accuracy of the robotic arm control command output, and achieves the goal of completing the grasping operation with high precision and high reliability.

[0026] In one specific embodiment, the data processing module includes an input unit and a VLM backbone network.

[0027] The specific implementation of inputting voice control commands, visual images, and robotic arm status data into the data processing module to obtain the fused spatiotemporal features output by the data processing module includes: Voice control commands, visual images, and robotic arm status data are input into the input unit. The input unit performs preset preprocessing operations on the input data and inputs the processed data into the VLM backbone network. The voice control commands are tokenized by the language encoder to obtain language feature vectors. The visual image is extracted by the visual encoder to obtain visual feature vectors. The robotic arm status data is extracted by the ontology temporal feature projection layer to obtain ontology temporal feature vectors. The language feature vectors, visual feature vectors, and ontology temporal feature vectors are input into the spatiotemporal Transformer encoder and adaptively fused based on a three-way cross-attention mechanism to obtain fused spatiotemporal features.

[0028] Specifically, the preprocessing operations include: denoising, normalizing, and unifying the size of the visual images to form visual temporal data of N consecutive frames. For example, the value of N ranges from 10 to 50 frames, and the sampling frequency is 10 to 30 Hz.

[0029] The robotic arm status data is normalized and outlier removal preprocessed to form body time-series data synchronized with the visual time-series, with the sampling frequency consistent with the visual time-series.

[0030] Specifically, the VLM backbone network includes: a visual encoder, a language encoder, a spatiotemporal Transformer encoder, and an ontology temporal feature projection layer.

[0031] Specifically, the ontology temporal feature projection layer maps the preprocessed robotic arm state data to the latent feature space of the VLM base to obtain the ontology temporal feature vector, and performs spatial unification operations on visual features, language features and ontology temporal features.

[0032] The spatiotemporal Transformer encoder incorporates a three-way cross-attention mechanism to obtain fused spatiotemporal features, achieving deep correlation and synergy among the three types of features.

[0033] Specifically, the three-way cross-attention mechanism is used to achieve adaptive association and deep fusion of visual features, linguistic features, and ontological temporal features. The specific implementation method is as follows: visual features, linguistic features, and ontological temporal features are respectively used as queries, keys, and values. The contribution of the three types of features is dynamically allocated through attention weight calculation in order to achieve adaptive association and deep fusion of features.

[0034] In one specific embodiment, the functional head further includes a physical prior constraint unit and a prediction unit.

[0035] The specific implementation of inputting spatiotemporal features, visual images, and robotic arm state data into the functional head to obtain the current grasping state classification result and corresponding first probability of occurrence output by the current state prediction head, and the predicted grasping state classification result and corresponding second probability of occurrence output by the future state prediction head at a future preset interval, includes: The spatiotemporal features are input into the prediction unit to obtain the prediction parameters output by the prediction unit. The spatiotemporal features, visual images, and robotic arm state data are input into the functional head to obtain the current grasping state classification result, first probability of occurrence, and first prediction label output by the current state prediction head, and the predicted grasping state classification result, second probability of occurrence, and second prediction label output by the future state prediction head. Based on the first and second prediction labels, the robotic arm state data and prediction parameters are input into the physical prior constraint unit to obtain the mechanical rule determination result output by the physical prior constraint unit. If the mechanical rule determination result is determined to conform to the physical laws, the current grasping state classification result and the corresponding first probability of occurrence, as well as the predicted grasping state classification result and the corresponding second probability of occurrence, are obtained.

[0036] The predicted parameters include the contact reaction force and contact reaction torque of the target to be grasped on the robotic arm.

[0037] Specifically, the current state prediction head outputs the current state classification result corresponding to the current moment based on the fused spatiotemporal features within a preset first sliding window.

[0038] The first sliding window is a preset frame preceding the current frame, such as frames 3 to 10.

[0039] The classification results include: stable gripping, empty gripping, slight relative slippage, severe slippage, imminent detachment, physical fall, abnormal gripping force, and collision.

[0040] Specifically, the future state prediction head outputs the predicted capture state classification result corresponding to the future time based on the fused spatiotemporal features within a preset second sliding window.

[0041] The second sliding window consists of preset frames before the current frame and preset frames after the current frame, and the predicted future time range is the future time corresponding to the preset frames after the current frame.

[0042] In one specific embodiment, the specific implementation of inputting the robotic arm state data and prediction parameters into the physical prior constraint unit to obtain the mechanical rule determination result output by the physical prior constraint unit includes: Input the robotic arm's state data and predicted parameters into the mechanical rule determination formula to obtain the mechanical rule determination result output by the mechanical rule determination formula.

[0043] The formula for determining the mechanical rules is shown in formula (1): ...(1) in, This indicates the first result in the determination of the mechanical rules. Indicates the clamping force of the grippers. Represents the contact reaction force, The gravity of the target to be grasped is a constant. This represents the second result in the determination of mechanical rules. Indicates joint torque. This indicates the contact reaction torque.

[0044] Specifically, if both the first result and the second result are less than the preset value, it is determined to conform to physical laws; if either or both of the first result and the second result are greater than or equal to the preset value, it is determined to not conform to physical laws.

[0045] The preset value is the preset allowable threshold for mechanical deviation.

[0046] Specifically, after the functional head completes the classification result prediction, it does not output it directly. Instead, it uses the physical prior constraint unit to determine the physical laws. Only when the physical prior constraint unit determines that the physical laws are consistent with the physical laws is the classification result and the corresponding probability of occurrence output output, thus ensuring the accuracy of the data output.

[0047] In one specific embodiment, the spatiotemporal features are integrated into the physical properties of the target to be captured, including deformability and non-deformability.

[0048] After obtaining the mechanical rule determination result output by the physical prior constraint unit, and assuming that the physical property is deformable and the mechanical rule determination result conforms to physical laws, the image data is input into the physical prior constraint unit to obtain the predicted contact force output by the physical prior constraint unit. The predicted contact force and the fused spatiotemporal features are then concatenated and input into a preset spatiotemporal convolutional network to obtain the constraint prediction classification result and the corresponding third occurrence probability output by the spatiotemporal convolutional network. If the consistency between the preset prediction classification result and the current grasping state classification result, and the consistency between the first occurrence probability and the third occurrence probability both meet the preset consistency criteria, the current grasping state classification result and the corresponding first occurrence probability, as well as the predicted grasping state classification result and the corresponding second occurrence probability, are obtained.

[0049] Specifically, after the functional head completes the classification prediction, it does not output the results directly. Instead, it uses physical prior constraint units to determine the physical laws. If the physical prior constraint unit's determination conforms to the physical laws, it determines the physical attributes of the target to be captured. If the physical attributes are immutable, it outputs the classification results and the corresponding probability of occurrence. If the physical attributes are deformable, it can be further verified using physical prior constraint units and a spatiotemporal convolutional network. If the verification is consistent, it outputs the classification results and the corresponding probability of occurrence, ensuring the accuracy of the data output.

[0050] In one specific embodiment, the specific implementation of inputting image data into the physical prior constraint unit to obtain the predicted contact force output by the physical prior constraint unit includes: The image data is analyzed to obtain the object's contact deformation and deformation rate, which are then input into the soft contact dynamics formula to obtain the predicted contact force output by the soft contact dynamics formula.

[0051] The formula for soft contact dynamics is given in formula (2): ...(2) in, Indicates the predicted contact force. This represents the contact stiffness coefficient, which is a constant. This represents the contact damping coefficient, determined based on the material of the target to be grasped. Indicates the deformation of an object upon contact. The nonlinear deformation exponent is a constant. Indicates the deformation rate.

[0052] In one specific embodiment, after obtaining the mechanical rule determination result output by the physical prior constraint unit, if it is determined that the mechanical rule determination result does not conform to the physical laws, based on the new visual image and robotic arm state data, a step is executed to input the voice control command, visual image and robotic arm state data into the data processing module in real time to obtain the fused spatiotemporal features output by the data processing module.

[0053] In one specific embodiment, the implementation of the motion decoder adjusting the initial robotic arm control commands based on feedback information includes: If the probability of slight relative slippage and severe slippage in any one or more of the current grasping state classification results and the predicted grasping state classification results is greater than the preset probability, increase the gripper clamping force and reduce the end effector speed.

[0054] If the probability of abnormal gripping force in any one or more of the current gripping state classification results and the predicted gripping state classification results is greater than the preset probability, adjust the gripping force of the gripper to a preset reasonable range.

[0055] If any one or more of the current grasping state classification results and the predicted grasping state classification results indicate that the object is about to fall off, stop the current action, plan a new grasping trajectory, and repeat the grasping action.

[0056] This application feeds back the output of the function head to the motion decoder to achieve stable target grasping and realizes intelligent compliant control based on grasping state awareness.

[0057] In one specific embodiment, a visual language model is pre-trained, specifically including: A transfer learning strategy is adopted, based on the pre-trained weights of the VLM base, freezing some of the underlying visual and language coding layers, and training the newly added ontology temporal feature projection layer, three-way cross attention layer, grasping state temporal multi-class estimation head (functional head), and action decoder fine-tuning layer, thereby reducing training costs and improving training efficiency.

[0058] Construct a grasping state time series dataset, which includes visual time series, ontology time series, language commands and corresponding grasping state annotations for different objects and grasping scenarios, for model training and validation.

[0059] A hybrid loss function (cross-entropy loss + temporal prediction loss) is used to train the model end-to-end, optimizing the feature fusion effect and state estimation accuracy, and ensuring the synchronization and accuracy of motion control and state estimation.

[0060] Specifically, a physical loss is constructed based on mechanical equilibrium and soft contact dynamics rules to constrain the model training process, allowing the network parameters to converge iteratively in a direction that conforms to physical laws.

[0061] The formula for calculating the physical consistency loss is shown in formula (3): ...(3) in, Indicates the loss of physical consistency. This represents the contact loss weighting coefficient, which is a constant.

[0062] This application achieves single-model synchronous inference and feature sharing for visual language understanding, robotic arm motion control, grasping state timing detection and prediction by fusing features from multimodal data, which significantly reduces system complexity and inference latency.

[0063] This application adopts a sliding window timing multi-class head to simultaneously detect the current grasping state and predict the future state. It can provide early warning of abnormalities such as object falling off or sliding, giving the robotic arm sufficient time to adjust its control strategy, greatly improving the reliability of grasping operations and reducing the grasping failure rate.

[0064] This application constructs a two-way feedback mechanism between grasping state and motion control. The state estimation results are fed back to the motion decoder in real time to dynamically correct the control commands, enabling the robotic arm to adaptively adjust according to changes in the grasping state and adapt to complex grasping scenarios (such as object sliding and deformation).

[0065] This application is based on a secondary improvement of the VLM base and utilizes its pre-trained weights for transfer learning, reducing model training costs and R&D cycles; it eliminates the need for additional deployment of independent grasping and detection models and hardware, reducing system deployment costs and making it easy to implement on industrial robotic arms, service robots and other equipment.

[0066] Figure 2 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 2 As shown, the electronic device may include a processor 201, a communications interface 202, a memory 203, and a communication bus 204. The processor 201, communications interface 202, and memory 203 communicate with each other via the communication bus 204. The processor 201 can call logical instructions from the memory 203 to execute a robotic arm control method based on a visual language model.

[0067] Furthermore, the logical instructions in the aforementioned memory 203 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the robotic arm control method based on the visual language model provided by the above methods.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the robotic arm control method based on the visual language model provided in the above embodiments.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A robotic arm control method based on a visual language model, characterized in that, The visual language model includes a data processing module and a functional head. The functional head includes a current state prediction head, a future state prediction head, and an action decoder. The method includes: The system acquires voice control commands from the user to perform grasping operations, and the robotic arm uses real-time visual images and real-time robotic arm status data to grasp the target based on the voice control commands. Voice control commands, visual images, and robotic arm status data are input into the data processing module in real time to obtain the fused spatiotemporal features output by the data processing module. The function head is fed with spatiotemporal features, visual images and robotic arm state data to obtain the initial robotic arm control command generated by the motion decoder, the current grasping state classification result and the corresponding first occurrence probability output by the current state prediction head, and the predicted grasping state classification result and the corresponding second occurrence probability at the future preset interval output by the future state prediction head. The current grasping state classification result, the first occurrence probability, the predicted grasping state classification result and the second occurrence probability are fed back to the motion decoder. The motion decoder adjusts the initial robotic arm control command based on the feedback information and outputs the adjusted robotic arm control command.

2. The robotic arm control method based on a visual language model according to claim 1, characterized in that, The functional head also includes: a physical prior constraint unit and a prediction unit; The data, which integrates spatiotemporal features, visual images, and robotic arm state data, is input into the functional head to obtain the current grasping state classification result and corresponding first probability of occurrence output by the current state prediction head, and the predicted grasping state classification result and corresponding second probability of occurrence output by the future state prediction head at a preset future interval, including: The spatiotemporal features are input into the prediction unit to obtain the prediction parameters output by the prediction unit. The prediction parameters include the contact reaction force and contact reaction torque of the target to be grasped on the robotic arm. The system integrates spatiotemporal features, visual images, and robotic arm state data into the functional head to obtain the current grasping state classification result, first occurrence probability, and first prediction label output by the current state prediction head, as well as the predicted grasping state classification result, second occurrence probability, and second prediction label output by the future state prediction head. Based on the first and second prediction tags, the robotic arm state data and prediction parameters are input into the physical prior constraint unit to obtain the mechanical rule determination result output by the physical prior constraint unit. Given that the determination result of the mechanical rule conforms to the physical law, we obtain the current grasping state classification result and the corresponding first occurrence probability, as well as the predicted grasping state classification result and the corresponding second occurrence probability.

3. The robotic arm control method based on a visual language model according to claim 2, characterized in that, After obtaining the mechanical rule determination results output by the physical prior constraint unit, the following is also included: When the determination result of the mechanical rule does not conform to the physical law, based on the new visual image and the state data of the robotic arm, the step of inputting the voice control command, visual image and robotic arm state data into the data processing module in real time is executed to obtain the fused spatiotemporal features output by the data processing module.

4. The robotic arm control method based on a visual language model according to claim 2, characterized in that, The spatiotemporal features include the physical properties of the target to be captured, including deformability; After obtaining the mechanical rule determination results output by the physical prior constraint unit, the following is also included: Given that the mechanical rule determination result conforms to physical laws, and given that the physical properties are variable, the image data is input into the physical prior constraint unit to obtain the predicted contact force output by the physical prior constraint unit. The predicted contact force and spatiotemporal features are spliced ​​together and input into a preset spatiotemporal convolutional network to obtain the constraint prediction classification result and the corresponding third occurrence probability output by the spatiotemporal convolutional network. Assuming the mechanical rule determination result conforms to physical laws, the current grasping state classification result and its corresponding first probability of occurrence are obtained, along with the predicted grasping state classification result and its corresponding second probability of occurrence, including: If the consistency between the preset predicted classification result and the current grasping state classification result, and the consistency between the first occurrence probability and the third occurrence probability both meet the preset consistency criteria, then the current grasping state classification result and the corresponding first occurrence probability, as well as the predicted grasping state classification result and the corresponding second occurrence probability, are obtained.

5. The robotic arm control method based on a visual language model according to claim 2, characterized in that, The robotic arm's state data and predicted parameters are input into the physical prior constraint unit to obtain the mechanical rule determination results output by the physical prior constraint unit, including: Input the robotic arm state data and prediction parameters into the mechanical rule determination formula to obtain the mechanical rule determination result output by the mechanical rule determination formula; The formulas for determining mechanical rules include: ； in, This indicates the first result in the determination of the mechanical rules. Indicates the clamping force of the grippers. Represents the contact reaction force, The gravity of the target to be grasped is a constant. This represents the second result in the determination of mechanical rules. Indicates joint torque. This indicates the contact reaction torque.

6. The robotic arm control method based on a visual language model according to claim 4, characterized in that, The image data is input into the physical prior constraint unit to obtain the predicted contact force output by the physical prior constraint unit, including: The image data is analyzed to obtain the object's contact deformation and deformation rate, which are then input into the soft contact dynamics formula to obtain the predicted contact force output by the soft contact dynamics formula. The soft contact dynamics formulas include: ； in, Indicates the predicted contact force. This represents the contact stiffness coefficient, which is a constant. This represents the contact damping coefficient, determined based on the material of the target to be grasped. Indicates the deformation of an object upon contact. The nonlinear deformation exponent is a constant. Indicates the deformation rate.

7. The robotic arm control method based on a visual language model according to any one of claims 1-6, characterized in that, The motion decoder adjusts the initial robotic arm control commands based on feedback information, including: If the probability of slight relative slippage and severe slippage in any one or more of the current grasping state classification results and the predicted grasping state classification results is greater than the preset probability, increase the gripper clamping force and reduce the end effector speed. If the probability of abnormal gripping force in any one or more of the current gripping state classification results and the predicted gripping state classification results is greater than the preset probability, adjust the gripping force of the gripper to a preset reasonable range. If any one or more of the current grasping state classification results and the predicted grasping state classification results indicate that the object is about to fall off, stop the current action, plan a new grasping trajectory, and repeat the grasping action.

8. The robotic arm control method based on a visual language model according to any one of claims 1-6, characterized in that, The data processing module includes: an input unit and a VLM backbone network; Voice control commands, visual images, and robotic arm status data are input into the data processing module to obtain the fused spatiotemporal features output by the data processing module, including: Voice control commands, visual images, and robotic arm status data are input into the input unit. The input unit performs preset preprocessing operations on the input data and inputs the processed data into the VLM backbone network. The voice control commands are tokenized by the language encoder to obtain language feature vectors. The visual image is extracted by the visual encoder to obtain visual feature vectors. The robotic arm status data is extracted by the ontology temporal feature projection layer to obtain ontology temporal feature vectors. The language feature vectors, visual feature vectors, and ontology temporal feature vectors are input into the spatiotemporal Transformer encoder and adaptively fused based on a three-way cross-attention mechanism to obtain fused spatiotemporal features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the robotic arm control method based on a visual language model as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the visual language model-based robotic arm control method as described in any one of claims 1 to 8.

Images