Mechanical arm grabbing prediction method, electronic equipment and storage medium

Abstract

The invention provides a mechanical arm grabbing prediction method, electronic equipment and a storage medium, and belongs to the technical field of mechanical arm vision. A mechanical arm grabbing posture correction platform is built, and an RGB image of an actuator posture is obtained; an LLaMA-Adapter is adopted to construct a multi-mode large model; visual features of the RGB image are extracted through a CLIP model, a pre-trained LLaMA word segmentation device is used for analyzing visual feature prompts, and text information and image information of RGB image content are obtained; on the basis of a continuous thinking fine-tuning reasoning grabbing strategy, the operation posture, corresponding to the processed text information and image information, of an end effector of the mechanical arm is predicted; and operation posture operation data of an end effector of the mechanical arm are obtained in real time, and the multi-modal large model is verified based on an index moving average method. According to the method, a continuous strategy learning method is provided, the adaptability of the model to current scene configuration is enhanced, the frequency of expert intervention is reduced, and the mechanical arm meets the use requirement.

Description

Technical Field

[0001] The present invention belongs to the field of robot vision technology, and in particular relates to a robot grasping prediction method based on a self-correcting multi-modal large model, an electronic device and a storage medium. Background Art

[0002] In the existing technology, robotic arm visual grasping technology has been widely used in many fields. In practical applications, the environment in which the robotic arm is located is often complex and changeable, with interference factors such as lighting changes, occlusion, and reflection. These factors will affect the quality of the image and thus reduce the accuracy and reliability of visual feedback.

[0003] The rapid movement of the robot arm may cause image blur, especially when grasping at high speed. Image blur will seriously affect the accuracy of visual feedback and increase the risk of grasping failure. Current robot arm visual grasping systems are often optimized for specific tasks or objects and lack sufficient flexibility and adaptability. This also causes the robot manipulation strategy to fail to meet the motion performance requirements, affecting the use of the robot arm. Summary of the invention

[0004] The present invention provides a robot arm grasping prediction method, which provides a continuous strategy learning method, enhances the adaptability of the model to the current scene configuration, reduces the frequency of expert intervention, and enables the robot arm to meet usage requirements.

[0005] Methods include:

[0006] S101: Build a robot gripping posture prediction platform, use the Franka Panda robot as the robot posture prediction model, and obtain the RGB image of the actuator posture;

[0007] S102: Use LLaMA-Adapter to construct a multimodal large model;

[0008] S103: extract visual features of the RGB image through the CLIP model of the multimodal large model, and use the pre-trained LLaMA word segmenter to parse the visual feature prompts to obtain text information and image information of the RGB image content;

[0009] S104: fine-tuning the reasoning grasping strategy based on continuous thinking, predicting the operation posture of the end effector of the robot arm corresponding to the processed text information and image information;

[0010] S105: Acquire the operation posture data of the end effector of the robot arm in real time, and verify the multi-modal large model based on the exponential moving average method.

[0011] It should be further explained that, in step S101, the SAPIEN dataset and the PartNet-Mobility dataset are used to build a robot arm grasping posture correction platform.

[0012] It should be further explained that in step S101, the robot arm grasping posture correction platform is configured with a VulkanRenderer high-efficiency renderer.

[0013] It should be further explained that in the method, the dataset loader provided by the SAPIEN dataset is used to load the object model in the dataset and the URDF files of the Franka Panda robot and actuator.

[0014] It should be further explained that, in step S103, the CLIP model is configured with a text encoder and an image encoder;

[0015] The text encoder is selected as the training network, and the image encoder uses a deep convolutional network as the training network. The input form of the image encoder is [n, h, w, c], where n is the batch size and h, w, c are the image sizes.

[0016] It should be further explained that, in step S103, the CLIP model pre-trains the text information and the image information to obtain the association relationship between the image content and the natural language description;

[0017] Pre-training includes the following steps:

[0018] S1031: Processing text information and image information into feature vectors;

[0019] S1032: construct a relationship matrix, where each element in the relationship matrix is ​​the cosine similarity between each image feature vector and text feature vector;

[0020] S1033: The loss function formula used in the pre-trained contrastive learning method is as follows:

[0021]

[0022] τ is the set hyperparameter; q is the encoding feature; k is the encoding sample; k + Highly matched samples.

[0023] It should be further explained that, in the method, the grasping motion types are divided into rotation and translation, and the operation posture movement orientation of the end effector of the robot arm is obtained based on the Affordance map model;

[0024] Affordance map model A∈R H×W , based on the following formula:

[0025]

[0026] Based on D∈R H×W Calculate the Euclidean distance of the robot's position before and after movement, and A is the probability of mobility of each pixel.

[0027] It should be further explained that in step S105, the strategy formula of the exponential moving average method is μ τ =αμ τ-1 +(1-α)μ τ

[0028] Among them, τ is the time step, μ represents the multimodal large model, and α = 0.99.

[0029] According to another embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the robot arm grasping prediction method when executing the program.

[0030] According to another embodiment of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the robot arm grasping prediction method are implemented.

[0031] It can be seen from the above technical solutions that the present invention has the following advantages:

[0032] The robot grasping prediction method provided by the present invention can more comprehensively understand the grasping scene through the visual features and text information of the RGB image, so as to more accurately predict the optimal operating posture of the robot end effector. The multimodal fusion method can better capture the subtle differences in complex environments than a single modality, and improve the accuracy and robustness of the prediction.

[0033] The present invention utilizes the pre-trained LLaMA word segmenter and CLIP model, which can process input data from different sources and formats, enhances the generalization ability of the model, and can also adapt to different working environments and task requirements.

[0034] The robot grasping prediction method can achieve online learning and self-optimization of the model by acquiring the operation posture data of the robot end effector in real time and verifying the multi-modal large model using the exponential moving average method. It can ensure that the model still maintains a high prediction accuracy after long-term operation.

[0035] The method improves the efficiency and reliability of industrial automation production by automatically predicting and correcting the grasping posture of the robot arm. It integrates technologies from multiple fields such as deep learning, computer vision, and natural language processing to improve prediction accuracy, enhance generalization ability, and achieve real-time self-correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0037] Figure 1 A flowchart of the robot grasp prediction method;

[0038] Figure 2 Schematic diagram of an electronic device. DETAILED DESCRIPTION

[0039] The following describes in detail the robot arm grasping prediction method involved in the present application. For the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0040] It should be understood that when used in the present specification, the term "includes" indicates the presence of the described features, integral bodies, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integral bodies, steps, operations, elements, components and / or their collections. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0041] It should be understood that the "one or more" mentioned in this application refers to one, two or more than two, and the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B. The "and / or" in this article is just a way to describe the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0042] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" etc. that appear in different places in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1 The figure is a flowchart of a robot arm grasping prediction method in a specific embodiment, the method comprising:

[0045] S101: Build a robotic arm grasping posture correction platform, use the Franka Panda robot as the robotic arm posture prediction model, and obtain the RGB image of the actuator posture.

[0046] In some embodiments, a Franka Panda robot is used as an experimental platform, which is known for its high accuracy and flexibility and is suitable for complex robotic arm posture prediction tasks.

[0047] In this embodiment, a high-resolution camera is installed in the working area of ​​the robot arm to capture the RGB image of the actuator position. At the same time, necessary sensors (such as force sensors, position sensors, etc.) are installed to obtain more comprehensive robot arm operation status data. The camera and sensors are connected to a computer or data processing center to ensure real-time data transmission and processing.

[0048] In some specific embodiments, a Franka Panda robot is selected as an experimental platform, and a high-resolution camera is installed in the working area of ​​the robotic arm to ensure that the RGB image of the actuator posture can be clearly captured.

[0049] The robot gripping posture correction platform is configured to receive and process data from cameras and sensors. A data interface is established to ensure that the camera and sensor can transmit data to the data processing center in real time and stably.

[0050] In some embodiments, SAPIEN is a physics-based simulation environment for home scene robots. The robotic arm grasping posture correction platform can use the SAPIEN dataset and the PartNet-Mobility dataset to set up an interactive environment, use the dataset loader provided by SAPIEN, load the object model and the Franka Panda robot and end effector URDF file in the dataset, use the efficient renderer VulkanRenderer based on efficient rasterization, randomly select a contact point on the movable part, and use the opposite direction of its normal vector as the actuator direction to interact with the target, and successfully implement the operation and record it as a successful sample.

[0051] Optionally, record offline sampling of about 10,000 successful operation samples, covering 20 categories.

[0052] S102: Use LLaMA-Adapter to construct a multimodal large model.

[0053] In some embodiments, the robot arm grasping posture correction platform uses LLaMA-Adapter as the basic model, which can support efficient fine-tuning and can process text information and image information of multi-modal input.

[0054] This embodiment loads the pre-trained LLaMA model and freezes most of the parameters, and only fine-tunes the adaptation layer in the LLaMA-Adapter to reduce the amount of calculation and avoid overfitting, thereby improving the efficiency of self-correction.

[0055] The model set in this embodiment accepts RGB images and text instructions as inputs, uses the visual encoder of the CLIP model (or similar models) to extract the visual features of the image, and uses the pre-trained LLaMA word segmenter to process the text instructions.

[0056] For the multimodal large model constructed by LLaMA-Adapter, a small amount of self-instruct data is used to train LLaMA-Adapter to adapt it to the task of robot arm posture prediction.

[0057] S103: The visual features of the RGB image are extracted through the CLIP model of the multimodal large model, and the visual feature cues are parsed using the pre-trained LLaMA word segmenter to obtain the text information and image information of the RGB image content.

[0058] In some embodiments, the RGB image captured by the camera is passed as input to the visual encoder of the CLIP model. The CLIP model encodes the image and extracts a high-level visual feature vector. The visual feature vector is passed to the multimodal large model and combined with the textual prompts processed by the pre-trained LLaMA word segmenter.

[0059] In this way, within the multimodal large model, visual features and textual prompts are fused through a specific fusion mechanism to form a unified input representation.

[0060] S104: Based on continuous thinking, fine-tuning the reasoning grasping strategy, predicting the operation posture of the robot end effector corresponding to the processed text information and image information.

[0061] In some embodiments, in LLaMA-Adapter, the visual features extracted by CLIP are fused with textual prompts and processed by an adaptive prompt layer inside the model.

[0062] The model of this embodiment can predict the optimal operating posture of the end effector of the robot arm based on the fused input information. Optionally, it can include the position of the contact point, the direction and strength of the gripper, etc.

[0063] This embodiment also uses continuous thinking (such as a fast and slow system that mimics the human thinking process) to fine-tune the model to optimize the prediction results. For example, when the prediction is wrong, the model can reflect on itself and generate a new prediction.

[0064] In this embodiment, the multimodal large model predicts the optimal operating posture of the end effector of the robot arm based on the fused input representation, wherein the prediction results include key parameters such as the position of the contact point, the direction and force of the gripper, etc.

[0065] This embodiment fine-tunes the prediction strategy by introducing a continuous thinking mechanism (such as a fast and slow system model). When there is a deviation between the prediction result and the actual execution result, the model can reflect on itself and generate a new prediction strategy. Through continuous learning and optimization, the model's adaptability to specific scenarios and tasks is improved.

[0066] S105: Acquire the operation posture data of the end effector of the robot arm in real time, and verify the multi-modal large model based on the exponential moving average method.

[0067] In some embodiments, the operation posture and operation data of the end effector of the robot arm are obtained in real time through sensors. The actual operation data is compared with the model prediction results, and the exponential moving average method (EMA) is used to smooth the prediction performance of the model. EMA can help the model better adapt to short-term data fluctuations while retaining long-term trends.

[0068] This embodiment can make necessary adjustments to the multimodal large model based on the verification results. If the prediction error is large, some parameters of the model can be retrained or fine-tuned. At the same time, the successfully corrected samples are used for continuous strategy learning to improve the adaptability of the model to specific scenario configurations.

[0069] In some specific embodiments, the operation posture and operation data of the end effector of the robot arm can be obtained in real time through sensors, and the collected data can be preprocessed and cleaned to ensure the accuracy and integrity of the data.

[0070] Afterwards, the exponential moving average method (EMA) is used to smooth the prediction performance of the multimodal large model. The actual operation data is compared and analyzed with the model prediction results, and indicators such as prediction error and accuracy are calculated.

[0071] This embodiment can also adjust and continuously optimize feedback. Specifically, necessary adjustments and optimizations can be made to the multimodal large model based on the verification results. Successfully calibrated samples can be used for continuous strategy learning to improve the model's adaptability to specific scenario configurations. The model is regularly updated and iterated to meet the use requirements of the robotic arm.

[0072] Through the above steps, a robotic arm posture self-correction platform based on a multimodal large model can be constructed. The platform can predict and correct the operating posture of the robotic arm in real time, thereby improving the accuracy and stability of the robotic arm operation.

[0073] In one embodiment of the present invention, based on the robot arm grasping prediction method, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0074] In this embodiment, the multimodal pre-training implementation process uses LLaMA-Adapter to construct a multimodal large model (MLLM), extracts visual features of RGB images through the CLIP model, and uses the pre-trained LLaMA word segmenter to encode text prompts.

[0075] The CLIP model of this embodiment includes a text encoder and an image encoder. The text encoder selects TextTransformer as the training network, and the image encoder uses a deep convolutional network as the training network. The input form of the image encoder is [n, h, w, c], where n is the batch size, and h, w, c are the image sizes.

[0076] For example, 224×224×3; the input format of the text encoder is [n, l]. Since it is an image-text pair, the batch in the text encoder is the same as the batch in the image encoder, and l is the sequence length. Then the similarity between the text vector and the image vector is calculated to predict whether they are a pair. The contrastive learning method is used to pre-train CLIP. The CLIP model obtains the association between image content and natural language description by pre-training a large number of image-text pairs.

[0077] The specific steps of the pre-training stage of this embodiment are as follows:

[0078] ①. The input text and image are processed into feature vectors by their respective Encoders;

[0079] ②. Construct a relationship matrix. Each element in the relationship matrix is ​​the cosine similarity between each image feature vector and other text feature vectors. The elements on the main diagonal of the matrix are matched (the image and text features completely correspond), and the elements in other places are not matched.

[0080] ③. The loss function formula used in the pre-training contrastive learning method is as follows:

[0081]

[0082] τ is the set hyperparameter; q is the encoding feature; k is the encoding sample; k + Highly matched samples.

[0083] In some specific embodiments, the reasoning capability of MLLM is utilized to configure a continuous mind fine-tuning reasoning grasping strategy (ManipLLM) to achieve robust and interpretable actuator pose prediction.

[0084] In the process of pre-collecting training data, this embodiment captures RGB images and their corresponding successful manipulation end-effector postures in the simulator. During the inference process, we focus on predicting the 2D coordinates [x, y] of the contact pixels on the image, and then convert them into 3D coordinates using depth information. At the same time, we obtain the left side (Z-axis direction) of the gripper from its upward and forward directions based on the geometric relationship.

[0085] The MLLM of this embodiment aligns visual features with the embedding space of the large language model (LLM) through a projection layer, enabling LLaMA to perform multimodal understanding and generate corresponding answers. During training, we only fine-tune the injection adapter in the LLM while keeping the main pre-training parameters unchanged to retain the strong capabilities of the existing MLLM and enhance the model's capabilities in manipulation and failure correction.

[0086] Here, we can use the reasoning ability of MLLM to configure the continuous thinking fine-tuning reasoning grasping strategy (ManipLLM) to achieve robust and explainable actuator pose prediction. The continuous thinking fine-tuning reasoning grasping strategy can include: understanding the operation category of the end effector of the manipulator, prior force field reasoning, and posture prediction centered on the end effector of the manipulator.

[0087] For the operation classification of the end effector of the robot arm, based on object category identification (OCI), a deep learning method can be used to classify different categories of manipulated targets according to their geometric properties.

[0088] This embodiment also uses a priori force field reasoning. The grasping motion types are divided into "rotation" and "translation", and the corresponding affordance map model is collected. This stage aims to make the model understand which object areas can be manipulated. First, find the moving parts of the object and move the object parts along the axis. Affordance map model A∈R H×W Obtained as follows

[0089]

[0090] D∈R H×WCalculate the Euclidean distance of the 3D position (corresponding to each pixel) before and after the movement. A is the probability of motion of each pixel.

[0091] In a specific embodiment of the present application, object-centered posture prediction, after collecting training data, records the RGB image and the corresponding actuator posture as the model input and the reward value of the result, and predicts the 2D coordinate posture [x, y] of the end effector through the RGB image and text prompt information.

[0092] The depth frame information provided by the depth camera is used to convert the depth value into the Z coordinate in space through intrinsic parameter transformation. At the same time, the Z-axis direction of the gripper is obtained from its upward and forward directions according to the geometric relationship, and the accurate initial contact actuator pose interpretation is generated by reasoning.

[0093] In this embodiment, since the relative position of the robot and the object changes during each operation, a continuous strategy learning method is configured to enable the robot gripping posture prediction platform to have manipulation and fault correction capabilities. This method aims to enhance the posture prediction ability without the need for expert feedback prompts. Therefore, the exponential moving average method (EMA) is explored to continuously learn from new data and obtain successfully corrected samples. The strategy formula is: μ τ =αμ τ-1 +(1-α)μ τ .

[0094] Wherein, τ is the time step, and μ represents the model of the present invention. The update weight α=0.99 can be set based on. The effectiveness of the EMA scheme in continuous learning of actions is evaluated by performing repeated closed-loop correction and continuous policy learning sessions for each scenario configuration.

[0095] In this way, the present invention realizes the use of a multimodal large-scale model to simultaneously predict the operating posture of the end effector of the robot arm, and can autonomously identify and correct failed operating actions.

[0096] The present application also provides an electronic device, which is used to implement the steps of the robot arm grasping prediction method.

[0097] The electronic device of the present embodiment is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0098] Figure 21 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 includes but is not limited to: a network module 502, an audio output unit 503, an input unit 504, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 501, and a power module 511.

[0099] The processor 501 may include one or more processing units, for example, the processor 501 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0100] The input unit 504 is used to receive audio or video signals. The input unit 504 may include a graphics processing unit (GPU) and a microphone, and the graphics processor processes image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame may be displayed on the display unit 506.

[0101] The user input unit 507 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 507 includes a touch panel and other input devices.

[0102] The present application also provides a storage medium storing a program product capable of implementing the above-mentioned robot arm grasping prediction method of this specification. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0103] The storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0104] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robotic arm grasping prediction method, characterized in that the method include: S101: Build a robotic arm grasping posture prediction platform, use the Franka Panda robot as the robotic arm posture prediction model, and obtain RGB images of the actuator posture; S102: Use LLaMA-Adapter to construct a large multimodal model; S103: Extract visual features of the RGB image using the CLIP model of the multimodal large model, and use the pre-trained LLaMA word segmenter to parse the visual feature cues to obtain textual information and image information of the RGB image content; S104: Based on continuous thinking, fine-tuning the reasoning grasping strategy is performed to predict the operation posture of the robot end effector corresponding to the processed text information and image information; S105: Acquire the operating posture data of the end effector of the robotic arm in real time, and verify the multimodal large model based on the exponential moving average method.

2. The robot arm grasping prediction method according to claim 1, characterized in that: In step S101, the SAPIEN dataset and the PartNet-Mobility dataset are used to build a robotic arm grasping posture correction platform.

3. The robot arm grasping prediction method according to claim 1, characterized in that: In step S101, the robotic arm grasping posture correction platform is configured with a VulkanRenderer high-efficiency renderer.

4. The robot arm grasping prediction method according to claim 2, characterized in that: In this method, the dataset loader provided by the SAPIEN dataset is used to load the object model in the dataset and the URDF files of the Franka Panda robot and actuator.

5. The robot arm grasping prediction method according to claim 1, characterized in that: In step S103, the CLIP model is configured with a text encoder and an image encoder; The text encoder is selected as the training network, and the image encoder uses a deep convolutional network as the training network. The input form of the image encoder is [n, h, w, c], where n is the batch size and h, w, c are the image sizes.

6. The robot arm grasping prediction method according to claim 5, characterized in that: In step S103, the CLIP model is pre-trained on the text information and image information to obtain the association between the image content and the natural language description; Pre-training includes the following steps: S1031: Processing text information and image information into feature vectors; S1032: Construct a relationship matrix, where each element in the relationship matrix is ​​the cosine similarity between each image feature vector and text feature vector; S1033: The loss function formula used in the pre-training contrastive learning method is as follows: τ is the set hyperparameter; q is the encoding feature; k is the encoding sample; k + Highly matched samples.

7. The robot arm grasping prediction method according to claim 5, characterized in that: In this method, the grasping motion types are divided into rotation and translation, and the operation posture and movement orientation of the robot end effector are obtained based on the affordance map model; Affordance map model A∈R H×W , obtained based on the following formula: Based on D∈R H×W Calculate the Euclidean distance between the positions of the robot before and after movement, where A is the probability of motion of each pixel.

8. The robot arm grasping prediction method according to claim 5, characterized in that: In step S105, the strategy formula of the exponential moving average method is μ τ =αμ τ-1 +(1-α)μ τ Where τ is the time step, μ represents the multimodal large model, and α = 0.

99.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the robot arm grasping prediction method according to any one of claims 1 to 8 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the robot arm grasping prediction method according to any one of claims 1 to 8 are implemented.

Images