Mechanical arm motion planning method based on vision-language multi-modal data

Abstract

The invention provides a mechanical arm motion planning method based on vision-language multi-modal data. The method comprises the following steps: acquiring a human language instruction, and controlling an image acquisition device of the robot to acquire an environment image; generating multi-modal data based on the human language instruction and the environment image by utilizing a multi-modal data generation module; encoding the human language instruction by utilizing a language model to obtain a text language vector; performing absolute position coding on the environment image to obtain a visual coding vector; fusing the text language vector and the visual coding vector to obtain multi-modal data; and inputting the multi-modal data into the trained multi-modal motion control model to obtain a motion planning track of the mechanical arm. A target track generated by the mechanical arm under manual control is used for reinforcement learning to train a multi-modal motion control model, the motion planning path and strategy of the mechanical arm are optimized, and the mechanical arm can achieve the motion effect approaching human beings.

Description

Technical Field

[0001] This invention relates to the field of intelligent mechanical control technology, and in particular to a method for planning the motion of a robotic arm based on visual-linguistic multimodal data. Background Technology

[0002] When a large model is integrated into a robot, complex instructions can be transformed into specific action plans. Humans then only need to issue the corresponding commands, and the robot can automatically complete the actions and tasks through the motion control planner. However, when the robot performs more complex actions, such as folding clothes, washing dishes, or picking up objects, the robotic arm's motion paths may be inconsistent or suboptimal. This results in a gap between the robot's robotic arm movements and human movements, preventing it from completing tasks as quickly and efficiently as a human. Summary of the Invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art and provide a method for motion planning of robotic arms based on vision-language multimodal data.

[0004] To achieve the above-mentioned objectives of the present invention, according to a first aspect of the present invention, a method for planning the motion of a robotic arm based on visual-linguistic multimodal data is provided, comprising: acquiring human language commands and controlling the robot's image acquisition device to acquire environmental images; generating multimodal data based on the human language commands and environmental images using a multimodal data generation module; inputting the multimodal data into a trained multimodal motion control model to obtain the motion planning trajectory of the robotic arm; wherein the multimodal data generation module performs the following: encoding the human language commands using a language model to obtain text language vectors; performing absolute position encoding on the environmental images to obtain visual encoding vectors; and fusing the text language vectors and visual encoding vectors to obtain multimodal data.

[0005] To achieve the above-mentioned objectives of the present invention, according to a second aspect of the present invention, the present invention provides a robotic arm motion planning device based on vision-language multimodal data, used to implement the robotic arm motion planning method based on vision-language multimodal data described in the first aspect of the present invention, comprising: an input module for acquiring human language commands and controlling the robot's image acquisition device to acquire environmental images; a multimodal data generation module for generating multimodal data based on human language commands and environmental images, including: encoding human language commands using a language model to obtain text language vectors; performing absolute position encoding on the environmental images to obtain visual encoding vectors; fusing text language vectors and visual encoding vectors to obtain multimodal data; and a trajectory generation module for inputting the multimodal data into a trained multimodal motion control model to obtain the motion planning trajectory of the robotic arm.

[0006] To achieve the above-mentioned objectives of the present invention, according to a third aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the robotic arm motion planning method based on vision-language multimodal data as described in the first aspect of the present invention.

[0007] To achieve the above-mentioned objectives of the present invention, according to a fourth aspect of the present invention, the present invention provides a robot including an image acquisition device, a motion control planner and at least one robotic arm, as well as the robotic arm motion planning device based on visual-linguistic multimodal data as described in the second aspect of the present invention.

[0008] This invention uses multimodal data as input to a multimodal motion control model, which can improve the model's environmental understanding and enhance task completion quality. Furthermore, during the training process, a digital twin bionic space is established, and reinforcement learning is used to train and optimize the robotic arm's motion planning path and strategy using the target trajectory generated by the robotic arm under human control. Correct motion trajectories are rewarded accordingly, enabling the robotic arm to mimic human movement and complete tasks quickly and efficiently. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a robotic arm motion planning method based on visual-linguistic multimodal data in a preferred embodiment of the present invention.

[0010] Figure 2 This is a diagram illustrating the training process of a multimodal motion control model in one example of this invention.

[0011] Figure 3 This is a schematic diagram of the hardware structure of an electronic device in a preferred embodiment of the present invention. Detailed Implementation

[0012] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0013] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0014] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0015] This invention discloses a robotic arm motion planning method based on vision-language multimodal data. In a preferred embodiment, such as... Figure 1 As shown, the method includes:

[0016] Step S1: Obtain human language commands to control the robot's image acquisition device to acquire environmental images. The image acquisition device is preferably, but not limited to, the robot's built-in camera. In one example, the length of the human language command is 16, and the size of the environmental image is preferably, but not limited to, 512*512*3. After receiving the human language command, the robot activates its camera to scan objects in the environment and obtain multiple environmental images from different perspectives. For example, 512*512*3 represents three environmental images of size 512*512 from different perspectives.

[0017] Step S2: Use the multimodal data generation module to generate multimodal data based on human language instructions and environmental images.

[0018] The multimodal data generation module executes as follows:

[0019] Step S21: Encode human language instructions using a language model to obtain text language vectors. For better performance with Chinese, the preferred language model is the Baidu Language Big Data Model. This model is used to embeddate and encode the text of human language instructions, generating token vectors, i.e., text language vectors. Alternatively, the language model can be a BERT model.

[0020] Step S22: Perform absolute position encoding on the environmental image to obtain a visual encoding vector. Variable-interval time-of-flight (VIT) encoding is used to embed the location of the environmental image. Each pixel or patch in the environmental image is assigned a learnable embedding vector, resulting in a related image + location sequence, which serves as the visual encoding vector.

[0021] Step S23: Fuse text language vectors and visual encoding vectors to obtain multimodal data. The environmental images in the multimodal data correspond to human language command text. Preferably, the multimodal data is then input into a large-scale text-based model, enabling the model to understand the corresponding speech commands and simultaneously correspond them to the physical state of the environment.

[0022] In this embodiment, the multimodal data generation module can also select a lightweight vision-language pre-training model, such as the CLIP (Contrastive Language-Image Pre-training) model, to perform self-supervised learning using the lightweight vision-language pre-training model, so that the text language vectors and visual encoding vectors correspond more accurately.

[0023] Step S3: Input the multimodal data into the trained multimodal motion control model to obtain the motion planning trajectory of the robotic arm.

[0024] In a preferred embodiment, refer to Figure 2 The training process of a multimodal motion control model includes:

[0025] Step A: Construct the neural network for the multimodal motion control model. The neural network preferably uses, but is not limited to, existing transformer networks. Construct a digital twin bionic space: Based on the real environment and the joint coordinates of the robot arm, establish a 1:1 digital twin bionic space using digital twin technology.

[0026] Step B involves pre-training the neural network to obtain a pre-trained neural network.

[0027] Step C involves collecting multiple reinforcement learning language commands and acquiring environmental images corresponding to each reinforcement learning language command within the digital twin bionic space. The reinforcement learning language commands are the language commands used during human control within the digital twin bionic space; environmental images are also acquired during this process.

[0028] Step D: The multimodal data generation module generates reinforcement multimodal data corresponding to each reinforcement learning language instruction based on each reinforcement learning language instruction and the corresponding environmental image.

[0029] Step E: In the constructed digital twin bionic space, the operator controls the robotic arm to execute actions according to each reinforcement learning language instruction. The operator wears a manipulator to perform various remote operations on the robotic arm, recording the actual motion data and operation planning strategies of the robotic arm under human control within the digital twin bionic space. The joint position coordinates of the robotic arm include the spatial position coordinates of each joint on the robotic arm.

[0030] Step F involves recording the motion trajectory generated by the robotic arm executing each reinforcement learning language instruction under human control, and denoting this trajectory as the target motion trajectory. The operator remotely controls the robotic arm via a manipulator to complete the required tasks. During the object grasping process, the computer records the coordinate changes of each joint of the robotic arm and generates corresponding multimodal data.

[0031] Step G: Construct a reinforcement learning sample set. A reinforcement learning sample includes reinforcement multimodal data and target motion trajectory corresponding to a reinforcement learning language instruction.

[0032] Step H involves using a reinforcement learning sample set to train the pre-trained neural network to obtain a multimodal motion control model.

[0033] In this embodiment, such as Figure 2 As shown, the multimodal motion control model adopts a joint training method of large model + reinforcement learning, which makes the robotic arm more similar to a real human in the process of completing tasks, and can perform motion control planning and decision-making more efficiently.

[0034] First, the BERT large language model is used to analyze human voice commands, and corresponding task decomposition and execution planning are performed. At the same time, a lightweight vision-language model is used to analyze and model the characteristics and parameters of the on-site environment, and then merge them into multimodal data.

[0035] By pre-training the neural network, a pre-trained neural network can be obtained.

[0036] Establish a digital bionic space and a digital robotic arm, collect motion paths and strategy data for human task completion, and form corresponding multimodal data and models.

[0037] The initial multimodal data of the robotic arm is combined with human multimodal data for joint reinforcement learning training to ultimately generate a multimodal motion control model that completes the task.

[0038] This multimodal motion control model allows robots to perform various complex tasks in a way that is closer to that of humans.

[0039] In a preferred embodiment, step B involves pre-training the neural network to obtain a pre-trained neural network, including:

[0040] Step B1: Collect multiple pre-trained language instructions and acquire the environmental image corresponding to each pre-trained language instruction.

[0041] Step B2: The multimodal data generation module generates pre-trained multimodal data corresponding to each pre-trained language instruction based on each pre-trained language instruction and the corresponding environmental image.

[0042] Step B3: Use the robot's motion control planner to generate n trajectories for each pre-trained language instruction, where n is a positive integer. Sampling is performed on the n trajectories using a Gaussian distribution to obtain the motion space reference trajectory corresponding to each pre-trained language instruction. The motion control planner is preferably, but not limited to, the existing fifth-order polynomial algorithm.

[0043] Step B4: Construct a pre-training sample set. A pre-training sample includes pre-training multimodal data and motion space reference trajectory corresponding to a pre-training language instruction.

[0044] Step B5: Train the neural network using the pre-training sample set to obtain a pre-trained neural network. During the training process of each pre-training sample, input the pre-training multimodal data of the pre-training sample into the neural network. The neural network outputs a heuristic trajectory. Calculate a first tolerance value based on the heuristic trajectory and the motion space reference trajectory. Adjust the network parameters of the neural network according to the magnitude of the first tolerance value.

[0045] In this embodiment, preferably, the formula for calculating the first tolerance value is:

[0046]

[0047] Where T represents the trajectory length, i.e. the number of trajectory points included in the trajectory, and the trajectory point index t∈[1,T]; This represents the joint position coordinates of the robotic arm at the t-th trajectory point in the motion space reference trajectory; This represents the joint position coordinates of the robotic arm at the t-th trajectory point in the heuristic trajectory.

[0048] In this embodiment, preferably, the motion space reference trajectory Track new Obtain it using the following formula:

[0049]

[0050] in, This represents a Gaussian distribution with a standard deviation of 1. This indicates Gaussian distribution sampling. The value represents the average joint position coordinates of the robotic arm at the t-th trajectory point of the n trajectories generated by the motion control planner for each pre-trained language instruction; CON represents the operation of concatenating the vectors before and after the motion trajectory.

[0051] In a preferred embodiment, during step H, in the reinforcement learning training process for each reinforcement learning sample:

[0052] The reinforcement learning samples' reinforcement multimodal data are input into a pre-trained neural network, which outputs a motion planning trajectory. A second tolerance value is calculated based on the target motion trajectory and the motion planning trajectory of the reinforcement learning samples. If the second tolerance value is less than or equal to a second threshold, the reward is increased; if the second tolerance value is greater than the second threshold, the reward is decreased. The network parameters of the pre-trained neural network are then adjusted.

[0053] In this embodiment, preferably, the formula for calculating the second tolerance value is:

[0054]

[0055] in, The joint position coordinates of the robotic arm represent the t-th trajectory point of the target motion trajectory; The coordinates of the joint position of the robotic arm represent the t-th trajectory point of the motion planning trajectory; T represents the trajectory motion length, and the trajectory point index t∈[1,T].

[0056] In this embodiment, more preferably, during reinforcement learning training, the robot can adjust and optimize its motion trajectory based on current environmental information parameters and reward function values, update the joint pose parameters of the real robotic arm in real time, and ultimately complete the corresponding work task. Preferably, the comprehensive reward is R, satisfying:

[0057]

[0058] Where ρ1 represents the first weight, ρ2 represents the second weight, ρ3 represents the third weight, and ρ4 represents the fourth weight; ρ1, ρ2, ρ3, and ρ4 can all be preset. q i q represents the joint velocity of the robotic arm; j τ represents the acceleration of the robotic arm joints; τ is the torque; action t The target trajectory. Represents the joint position coordinates of the target trajectory, v xy This represents the joint position coordinates of the motion planning trajectory. K is the first coefficient and is a positive number.

[0059] This invention also discloses a robotic arm motion planning device based on vision-language multimodal data, used to implement the aforementioned robotic arm motion planning method based on vision-language multimodal data. In a preferred embodiment, the device includes: an input module for acquiring human language commands and controlling the robot's image acquisition device to acquire environmental images; a multimodal data generation module for generating multimodal data based on human language commands and environmental images, including: encoding human language commands using a language model to obtain text language vectors; performing absolute position encoding on the environmental images to obtain visual encoding vectors; fusing text language vectors and visual encoding vectors to obtain multimodal data; and a trajectory generation module for inputting the multimodal data into a trained multimodal motion control model to obtain the motion planning trajectory of the robotic arm.

[0060] In this embodiment, the input module, the multimodal data generation module, and the trajectory generation module correspond one-to-one with steps S1, S2, and S3 of the above method, and will not be described again here.

[0061] This invention also discloses a robot, including an image acquisition device, a motion control planner, and at least one robotic arm, as well as the aforementioned robotic arm motion planning device based on visual-language multimodal data. The image acquisition device and the motion control planner are respectively connected to the robotic arm motion planning device.

[0062] The present invention also discloses an electronic device, in one embodiment of which the electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the robotic arm motion planning method based on vision-language multimodal data provided by the present invention.

[0063] like Figure 3 The diagram shown is a schematic representation of an electronic device for a robotic arm motion planning method based on vision-language multimodal data, according to an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for a robotic arm motion planning method based on vision-language multimodal data.

[0064] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a robotic arm motion planning method based on vision-language multimodal data) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0065] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a robotic arm motion planning method program based on vision-language multimodal data, but also to temporarily store data that has been output or will be output.

[0066] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0067] Communication interface 13 is used for communication between the aforementioned electronic device and other electronic devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0068] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements. For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0069] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0070] Furthermore, if the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, a computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0071] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0072] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for planning the motion of a robotic arm based on visual-linguistic multimodal data, characterized in that, include: It acquires human language commands to control the robot's image acquisition device to capture environmental images; The multimodal data generation module generates multimodal data based on human language commands and environmental images; Multimodal data is input into a trained multimodal motion control model to obtain the motion planning trajectory of the robotic arm; The multimodal data generation module performs the following steps: encoding human language instructions using a language model to obtain text language vectors; encoding the absolute position of environmental images to obtain visual encoding vectors; and fusing the text language vectors and visual encoding vectors to obtain multimodal data.

2. The robotic arm motion planning method based on vision-language multimodal data as described in claim 1, characterized in that, The training process of the multimodal motion control model includes: A neural network for constructing a multimodal motion control model was developed, and a digital twin bionic space was constructed. The neural network is pre-trained to obtain a pre-trained neural network; multiple reinforcement learning language instructions are collected, and environmental images corresponding to each reinforcement learning language instruction in the digital twin bionic space are acquired; The multimodal data generation module generates reinforcement multimodal data corresponding to each reinforcement learning language instruction based on each reinforcement learning language instruction and the corresponding environment image; In the constructed digital twin bionic space, a robotic arm is manually controlled to perform actions according to each reinforcement learning language instruction; Record the motion trajectory generated by the robotic arm executing each reinforcement learning language instruction under human control, and denote it as the target motion trajectory; Construct a reinforcement learning sample set. A reinforcement learning sample includes reinforcement multimodal data and target motion trajectory corresponding to a reinforcement learning language instruction. A multimodal motion control model is obtained by using a reinforcement learning sample set to train a pre-trained neural network.

3. The robotic arm motion planning method based on vision-language multimodal data as described in claim 2, characterized in that, Pre-training a neural network to obtain a pre-trained neural network includes: Collect multiple pre-trained language instructions and acquire the environmental image corresponding to each pre-trained language instruction; The multimodal data generation module generates pre-trained multimodal data corresponding to each pre-trained language instruction based on each pre-trained language instruction and the corresponding environmental image; The robot's motion control planner generates n trajectories for each pre-trained language instruction, and Gaussian distribution sampling is performed on the n trajectories to obtain the motion space reference trajectory corresponding to each pre-trained language instruction; Construct a pre-training sample set. Each pre-training sample includes pre-training multimodal data and motion space reference trajectory corresponding to a pre-training language instruction. The neural network is trained using a pre-training sample set to obtain a pre-trained neural network. During the training process of each pre-training sample, the pre-training multimodal data of the pre-training sample is input into the neural network, and the neural network outputs a heuristic trajectory. A first tolerance value is calculated based on the heuristic trajectory and the motion space reference trajectory, and the network parameters of the neural network are adjusted according to the magnitude of the first tolerance value.

4. The robotic arm motion planning method based on vision-language multimodal data as described in claim 3, characterized in that, The formula for calculating the first tolerance value is: Where T represents the trajectory length, i.e. the number of trajectory points included in the trajectory, and the trajectory point index t∈[1,T]; This represents the joint position coordinates of the robotic arm at the t-th trajectory point in the motion space reference trajectory; This represents the joint position coordinates of the robotic arm at the t-th trajectory point in the heuristic trajectory.

5. The robotic arm motion planning method based on vision-language multimodal data as described in claim 3, characterized in that, The motion space reference trajectory Track new Obtain it using the following formula: in, This represents a Gaussian distribution with a standard deviation of 1. This indicates Gaussian distribution sampling. The value represents the average joint position coordinates of the robotic arm at the t-th trajectory point of the n trajectories generated by the motion control planner for each pre-trained language instruction; CON represents the operation of concatenating the vectors before and after the motion trajectory.

6. The robotic arm motion planning method based on vision-language multimodal data as described in any one of claims 2-5, characterized in that, During the reinforcement learning training process for each reinforcement learning sample: The reinforcement learning samples are input into the reinforcement multimodal data, and the pre-trained neural network outputs the motion planning trajectory. The second tolerance value is calculated based on the target motion trajectory and motion planning trajectory of the reinforcement learning samples. If the second tolerance value is less than or equal to the second threshold, the reward is increased; if the second tolerance value is greater than the second threshold, the reward is decreased. The network parameters of the pre-trained neural network are adjusted.

7. The robotic arm motion planning method based on vision-language multimodal data as described in claim 6, characterized in that, The formula for calculating the second tolerance value is: in, The joint position coordinates of the robotic arm represent the t-th trajectory point of the target motion trajectory; The coordinates of the joint position of the robotic arm represent the t-th trajectory point of the motion planning trajectory; T represents the trajectory motion length, and the trajectory point index t∈[1,T].

8. A robotic arm motion planning device based on vision-language multimodal data, used to implement the robotic arm motion planning method based on vision-language multimodal data as described in any one of claims 1-7, characterized in that, include: The input module acquires human language commands and controls the robot's image acquisition device to capture environmental images. The multimodal data generation module generates multimodal data based on human language instructions and environmental images, including: encoding human language instructions using a language model to obtain text language vectors; performing absolute position encoding on environmental images to obtain visual encoding vectors; and fusing text language vectors and visual encoding vectors to obtain multimodal data. The trajectory generation module inputs multimodal data into a trained multimodal motion control model to obtain the motion planning trajectory of the robotic arm.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the robotic arm motion planning method based on vision-language multimodal data as described in any one of claims 1 to 7.

10. A robot, characterized in that, It includes an image acquisition device, a motion control planner, and at least one robotic arm, as well as the robotic arm motion planning device based on visual-linguistic multimodal data as described in claim 8.

Images