Mechanical arm control method and device based on pre-training large model

Abstract

The invention provides a mechanical arm control method and device based on a pre-training large model. The method comprises the steps that working information of a mechanical arm is obtained; inputting the working information into a large model to obtain a planned path; the large model is obtained by finely adjusting a pre-trained large model based on training data; and controlling the mechanical arm to execute the planned path. According to the method, path planning is carried out by pre-training the large model, and the motion path of the tail end of the mechanical arm can be more accurately generated by utilizing the strong feature extraction capability of the large model.

Description

Technical Field

[0001] The present application relates to the field of robotic arm control technology, and in particular, to a robotic arm control method and device based on a pre-trained large model. Background Art

[0002] Traditional path planning algorithms (such as A and RRT) have difficulty in generating accurate paths in high-dimensional complex environments, especially in environments with dynamic obstacles. Summary of the invention

[0003] The problem that this application solves is that it is difficult to generate a high-precision path for a robotic arm.

[0004] To solve the above problems, the first aspect of the present application provides a robot arm control method based on a pre-trained large model, comprising:

[0005] Get the working information of the robot arm;

[0006] The work information is input into a large model to obtain a planned path; the large model is obtained by fine-tuning a pre-trained large model based on training data;

[0007] Control the robot arm to execute the planned path.

[0008] The second aspect of the present application provides a robotic arm control device based on a pre-trained large model, which includes:

[0009] An information acquisition module, which is used to obtain the working information of the robotic arm;

[0010] A path planning module, which is used to input the work information into a large model to obtain a planned path; the large model is obtained by fine-tuning a pre-trained large model based on training data;

[0011] A robot arm control module is used to control the robot arm to execute the planned path.

[0012] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0013] The memory is used to store programs;

[0014] The processor, coupled to the memory, is configured to execute the program to:

[0015] Get the working information of the robot arm;

[0016] The work information is input into a large model to obtain a planned path; the large model is obtained by fine-tuning a pre-trained large model based on training data;

[0017] Control the robot arm to execute the planned path.

[0018] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned method for controlling a robotic arm based on a pre-trained large model.

[0019] In this application, path planning is performed by pre-training a large model and utilizing the powerful feature extraction capabilities of the large model to more accurately generate the motion path of the end of the robotic arm.

[0020] In this application, in a complex environment, the model can comprehensively consider the location of obstacles, the coordinates of the starting point and the target point to generate the optimal path. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a robotic arm control method based on a pre-trained large model according to an embodiment of the present application;

[0022] Figure 2 is a structural block diagram of a robotic arm control device based on a pre-trained large model according to an embodiment of the present application;

[0023] Figure 3 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0025] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0026] In response to the above problems, the present application provides a new robotic arm control solution based on a pre-trained large model, which can perform path planning through the pre-trained large model and eliminate the problem that high-precision paths of the robotic arm are difficult to generate.

[0027] The present application embodiment provides a method for controlling a robotic arm based on a pre-trained large model. The specific scheme of the method is as follows: Figure 1 As shown, the method can be performed by a robot arm control device based on a pre-trained large model, and the robot arm control device based on a pre-trained large model can be integrated in electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, it is a flow chart of a robot arm control method based on a pre-trained large model according to an embodiment of the present application; wherein the robot arm control method based on the pre-trained large model includes:

[0028] S101, obtaining working information of the robot arm;

[0029] S102, inputting the work information into a large model to obtain a planned path; the large model is obtained by fine-tuning a pre-trained large model based on training data;

[0030] S103, controlling the robot arm to execute the planned path.

[0031] In this application, path planning is performed by pre-training a large model and utilizing the powerful feature extraction capabilities of the large model to more accurately generate the motion path of the end of the robotic arm.

[0032] In this application, in a complex environment, the model can comprehensively consider the location of obstacles, the coordinates of the starting point and the target point to generate the optimal path.

[0033] In one embodiment, the step S102, after inputting the work information into the large model to obtain the planned path, further includes:

[0034] Perform path optimization and smoothing on the planned path.

[0035] In one implementation, the step of performing path optimization and smoothing on the planned path includes:

[0036] Calculate the total length of the path point sequence as the path length loss;

[0037] Calculate the distance between the path point and the obstacle as the obstacle avoidance loss;

[0038] Calculate the curvature change of the path point sequence as the smoothness loss;

[0039] The planned path is optimized based on path length loss, obstacle avoidance loss and smoothness loss.

[0040] In this application, the path length calculation is: for the generated path point sequence, the Euclidean distance between adjacent path points is calculated. The distances between all adjacent path points are accumulated to obtain the total length of the path point sequence. The total path length is used as part of the path length loss.

[0041] In this application, the path length optimization goal is: the goal is to minimize the path length loss, that is, to generate the shortest possible path. The path point sequence is adjusted through an optimization algorithm (such as gradient descent) to gradually reduce the total path length. Ensure that the optimization of the path length does not affect the obstacle avoidance effect and smoothness of the path.

[0042] In this application, obstacle distance calculation: For each path point, calculate the distance between it and the nearest obstacle in the environment. Use distance field or nearest neighbor search algorithm to quickly calculate the distance between the path point and the obstacle. Use the distance between the path point and the obstacle as part of the obstacle avoidance loss.

[0043] In this application, the obstacle avoidance penalty mechanism: if the distance between the path point and the obstacle is less than the safety threshold, a higher obstacle avoidance penalty is imposed. Through the obstacle avoidance loss function, the path point is ensured to be away from the obstacle to avoid collision. The intensity of the obstacle avoidance penalty is dynamically adjusted to adapt to the distribution of obstacles in different environments.

[0044] In this application, curvature calculation: for a path point sequence, the curvature at each path point is calculated. The curvature reflects the degree of curvature of the path. The greater the curvature, the less smooth the path. The curvature change of the path point sequence is regarded as part of the smoothness loss.

[0045] Smoothness optimization goal: The goal is to minimize the smoothness loss, that is, to generate a path that is as smooth as possible. The path point sequence is adjusted through the optimization algorithm to gradually reduce the curvature change. Ensure the smoothness of the path and avoid severe jitter during the movement of the robot arm.

[0046] In this application, the path length loss, obstacle avoidance loss and smoothness loss are weighted and summed to obtain the overall loss function.

[0047] In this application, an optimization algorithm (such as gradient descent) is used to minimize the overall loss function; the path length, obstacle avoidance effect and smoothness are gradually optimized by iteratively adjusting the path point sequence. Ensure that the optimized path meets the task requirements while taking into account safety, efficiency and smoothness

[0048] In the present application, the optimized path point sequence is post-processed (such as B-spline curve fitting) to further improve the smoothness of the path; the post-processed path point sequence is output as the final planned path.

[0049] In this application, additional points are inserted into the path point sequence to ensure the accuracy of the calculation of the path length.

[0050] In this application, a distance field algorithm is used to quickly calculate the distance between the path point and the obstacle; for dynamic obstacles, their motion trajectories are predicted and the distance between the path point and the dynamic obstacle is calculated.

[0051] In one embodiment, after the step S103 of controlling the robot arm to execute the planned path, the step further includes:

[0052] Real-time monitoring of the motion status and working information of the robotic arm;

[0053] Determine whether there are obstacles on the planned path;

[0054] In the presence of obstacles, a new planning path is generated through the large model, and the robotic arm is controlled to execute the new planning path.

[0055] In the present application, the motion state of the robotic arm, including joint angles, end effector position, speed and acceleration, is acquired in real time through the sensors of the robotic arm (such as encoders and force sensors).

[0056] In this application, sensors (such as RGB-D cameras and lidar) are used to obtain the latest information of the working environment in real time, including the location, shape and movement trajectory of obstacles.

[0057] In this application, obstacle data in the environmental information is used to detect whether there are obstacles on the planned path. For static obstacles, the distance between the path point and the obstacle is calculated to determine whether a collision occurs. For dynamic obstacles, their movement trajectory is predicted to determine whether they conflict with the planned path.

[0058] In this application, collision detection is performed using geometric methods (such as bounding box detection) or deep learning models (such as object detection models). For each path point, the minimum distance between it and the obstacle is calculated to determine whether it is less than the safety threshold. If a collision risk is detected, the path point is marked as a dangerous point.

[0059] In this application, the latest environmental information (including obstacle locations) is input into the pre-trained large model. The model regenerates the path point sequence at the end of the robotic arm based on the updated environmental information. Ensure that the newly generated path avoids obstacles while meeting the task requirements.

[0060] In one embodiment, the step S101 of obtaining working information of the robot arm includes:

[0061] Acquire environmental data and planning data of the sensor; the planning data includes a starting position and a target position;

[0062] Based on the environmental data, a three-dimensional environmental model is generated; the three-dimensional environmental model includes the location of obstacles;

[0063] The starting position and target position are mapped to the three-dimensional environment model to obtain the working information of the robotic arm.

[0064] In this application, sensors such as RGB-D cameras and lidar are used to collect environmental data, including depth information, color information, and point cloud data.

[0065] In this application, the task planning data of the robotic arm is obtained, including the starting position (the initial coordinates of the end of the robotic arm) and the target position (the target coordinates of the end of the robotic arm); the starting position and the target position are encoded into a format suitable for model input (such as a three-dimensional coordinate vector).

[0066] In this application, the environmental data is preprocessed, including voxelization of point cloud data, normalization of image data, etc.

[0067] In this application, point cloud data or depth images are used to generate a three-dimensional environment model, the point cloud data is segmented and clustered, and obstacles in the environment are identified; the location, shape, and size information of the obstacles are encoded into the three-dimensional environment model.

[0068] In this application, point cloud data or depth images are used to generate a three-dimensional environment model, the point cloud data is segmented and clustered, and obstacles in the environment are identified; the location, shape, and size information of the obstacles are encoded into the three-dimensional environment model.

[0069] In this application, the coordinates of the starting position and the target position are mapped to the coordinate system of the three-dimensional environment model to ensure that the starting position and the target position and the obstacle information of the environment are in the same coordinate system; the mapped coordinates are verified to ensure their accuracy and rationality.

[0070] In this application, the mapped starting position, target position and three-dimensional environment model are combined to generate the working information of the robot arm. The working information includes the obstacle position, starting point coordinates, target point coordinates and task constraints in the environment model.

[0071] In one embodiment, the pre-trained large model is a large-scale language model based on the Transformer architecture.

[0072] Among them, it can be a deepseek large model or a ChatGPT large model.

[0073] In this application, the DeepSeek large model is used as a pre-trained model, and fine-tuning is performed to achieve the end-of-arm path planning, which can combine the powerful representation capabilities of deep learning and the domain knowledge of robotic arm motion control.

[0074] As a pre-trained model, the DeepSeek large model already has powerful feature extraction and sequence modeling capabilities. By fine-tuning the robot path planning task, the model can learn knowledge of specific tasks such as robot kinematics, environmental perception, and path optimization, thereby achieving efficient path planning.

[0075] In one embodiment, the fine-tuning process of the pre-trained large model includes:

[0076] Get a pre-trained large model;

[0077] Based on the task requirements of the pre-trained large model, the input layer and the output layer of the pre-trained large model are updated; the updated output layer outputs a sequence of path points;

[0078] Acquire a planning data set, wherein the planning data set includes sample work information and annotated paths;

[0079] Input the sample work information into the pre-trained large model to obtain the prediction path;

[0080] Calculate the overall loss based on the predicted path and the annotated path;

[0081] Iteratively pre-train the large model based on the overall loss until the loss converges.

[0082] In this application, select a pre-trained large model based on the Transformer architecture (such as GPT, BERT or other large-scale language models); load the weights and configuration files of the pre-trained model to ensure that the model can run normally.

[0083] In this application, according to the task requirements, the structure of the input layer is adjusted so that it can receive the working information of the robotic arm (such as environmental data, starting position, target position); the environmental data (such as point cloud, image) is encoded into a format suitable for input into the Transformer model (such as serialized data or feature vector).

[0084] In the present application, the structure of the output layer is adjusted so that it can output a sequence of path points, each of which contains three-dimensional coordinate information (such as x, y, and z coordinates), and the dimension of the output layer matches the length and coordinate dimension of the path point sequence.

[0085] In this application, task-specific layers (such as fully connected layers, convolutional layers, and intermediate modules) are added based on the pre-trained model to ensure that the structure of the model is suitable for the path planning task while retaining the powerful feature extraction capabilities of the pre-trained model.

[0086] In this application, a planning data set is obtained: sample data of the robot arm path planning is collected, including environmental data, starting position, target position, and annotated path. The sample data is annotated to generate an annotated path (i.e., the ideal motion trajectory of the end of the robot arm). The annotated path can be generated by a traditional path planning algorithm (such as A*, RRT) or obtained through manual teaching. The planning data set is divided into a training set, a validation set, and a test set to ensure a balanced distribution of the data set and avoid overfitting.

[0087] In one embodiment, the loss calculation includes path length loss, obstacle avoidance loss, and motion smoothness loss.

[0088] Path length loss: Minimize the difference in length between the predicted path and the labeled path.

[0089] Obstacle avoidance loss: penalizes collisions of the predicted path with obstacles.

[0090] Smoothness loss: Minimize the change in curvature of the predicted path.

[0091] In this application, the overall loss is calculated based on the predicted path and the labeled path:

[0092] The predicted path is compared with the marked path to calculate the path length loss; the obstacle avoidance loss is calculated based on the environmental data to ensure that the predicted path avoids obstacles; the smoothness loss of the predicted path is calculated to ensure the smoothness of the path.

[0093] In this application, the path length loss, obstacle avoidance loss and smoothness loss are weighted and summed to obtain the overall loss; the performance of the model is evaluated by the overall loss to guide model optimization.

[0094] In one embodiment, based on the task requirements of the pre-trained large model, the input layer and output layer of the pre-trained large model are updated, and an intermediate module is inserted between adjacent layers of Transformer structures to enhance the model's expressiveness.

[0095] In one embodiment, the structure and processing of the intermediate module include:

[0096] Divide the input feature map into blocks to obtain independent blocks;

[0097] For each independent block, obtain a first neighboring block and a second neighboring block with different spacings;

[0098] Generate a first feature block based on the independent block and the first neighborhood block;

[0099] Based on the independent block and the second neighborhood block, generating a second feature block;

[0100] Performing feature compression on the first feature block and the second feature block to obtain a compressed block;

[0101] Iterate over all independent blocks and generate output feature maps based on the resulting compressed blocks.

[0102] In the present application, the input feature map is divided into blocks, that is, the input feature map is divided into corresponding feature blocks through a checkerboard; wherein, the feature blocks can be at the pixel level (that is, each pixel is a feature block) or at other levels, and the specific division shall be based on the actual processing situation.

[0103] In this application, a sliding window or a fixed step size is used to divide the feature map into blocks of the same size.

[0104] It should be noted here that if the input feature map is a two-dimensional feature map, it is directly divided into checkerboard grids, and each grid is a feature map block; if the input feature map is a three-dimensional feature map, a face is selected for checkerboard division, and each grid is a strip grid with a lot of depth (the depth is the depth of the three-dimensional feature map), and the strip grid is a feature map block.

[0105] Preferably, in the present application, each feature block is 100-1000 pixels, so that more feature calculations can be performed between local areas on the basis of ensuring generation accuracy and reducing the amount of calculation.

[0106] In this application, a feature block is selected as an independent block, and the adjacent feature blocks on the upper side, lower side, left side, and right side of the independent block are the first neighborhood blocks; the feature blocks on the upper side, lower side, left side, and right side of the independent block that are one grid apart are the second neighborhood blocks. The first neighborhood blocks and the second neighborhood blocks have different spacings from the independent block.

[0107] In this application, the neighborhood information is extracted for each independent block to capture the local structure.

[0108] In the present application, the first feature block is generated to generate a local feature representation using an independent block and its first neighborhood block. Specifically, the independent block and the first neighborhood block are processed by a convolution layer and an attention layer to obtain a first feature block.

[0109] In this application, the specific structure and specific parameters of the convolutional layer and the attention layer can be obtained according to the training data, or determined according to the actual situation.

[0110] It should be noted that, in the present application, there are four first neighborhood blocks and multiple first feature blocks.

[0111] In the present application, the independent block and the first neighborhood block are processed by the convolution layer and the attention layer to obtain the first feature block. The specific process is: the independent block and the four neighborhood blocks are spliced ​​together to form a multi-channel input, and the convolution layer is used to extract features from the spliced ​​blocks; the self-attention mechanism or the channel attention mechanism is used to enhance important features, the attention weights are calculated, and the convolution layer output is weighted to enhance important features; the output of the attention layer is split into multiple feature blocks, each feature block corresponds to the processing results of the independent block and at least one neighborhood block.

[0112] In the present application, a second feature block is generated to generate a broader local feature representation using an independent block and its second neighborhood block. The specific generation process is the same as that of the first feature block, except that the parameters of the convolution layer and the attention layer are different.

[0113] In this application, the generated feature blocks are compressed into a more compact representation to reduce the amount of computation and retain key information. Pooling operations (such as maximum pooling or average pooling) or fully connected layers are used for feature compression.

[0114] In this way, through compression, the first feature blocks and the second feature blocks are compressed into a compressed block, which corresponds to the size and position of the independent block and is used to replace the independent block. All feature map blocks are replaced by the compressed block to obtain an output feature map.

[0115] In the present application, each feature map block of the input feature map is traversed by traversal to obtain a corresponding compressed block.

[0116] In this application, for feature blocks / independent blocks near the edge, their first and second neighboring blocks are incomplete. In this case, they are completed by copying the first and second neighboring blocks at relative positions. For example, if the first neighboring block on the upper side of the independent block does not exist, the first neighboring block on the lower side is copied and used as the block on the upper side.

[0117] In this application, completion is performed to greatly improve the processing accuracy of edge feature blocks.

[0118] In this application, the similarity relationship between local regions is captured through an intermediate module to enhance feature representation.

[0119] In this application, the generalization ability of the pre-trained model is used to reduce the computational complexity of path planning.

[0120] In this application, the real-time requirements are met through the efficient reasoning of the deep learning model, especially in a dynamic environment.

[0121] In the present application, a B-spline curve or a spline interpolation method is used to smooth the path to generate a smooth motion trajectory.

[0122] In this application, the smoothness and accuracy of the movement of the robotic arm are ensured, and the jitter and error of the robotic arm are reduced.

[0123] An embodiment of the present application provides a robotic arm control device based on a pre-trained large model, which is used to execute a robotic arm control method based on a pre-trained large model described in the above content of the present application. The robotic arm control device based on the pre-trained large model is described in detail below.

[0124] like Figure 2 As shown, the robotic arm control device based on the pre-trained large model includes:

[0125] An information acquisition module 101 is used to acquire working information of the robot arm;

[0126] A path planning module 102, which is used to input the work information into a large model to obtain a planned path; the large model is obtained by fine-tuning a pre-trained large model based on training data;

[0127] The robot arm control module 103 is used to control the robot arm to execute the planned path.

[0128] In one implementation, the path planning module 102 is further configured to:

[0129] Perform path optimization and smoothing on the planned path.

[0130] In one implementation, the path planning module 102 is further configured to:

[0131] Calculate the total length of the path point sequence as the path length loss; calculate the distance between the path point and the obstacle as the obstacle avoidance loss; calculate the curvature change of the path point sequence as the smoothness loss; optimize the planned path based on the path length loss, obstacle avoidance loss and smoothness loss.

[0132] In one embodiment, the robot arm control module 103 is further used to:

[0133] Monitor the motion state and working information of the robot arm in real time; determine whether there are obstacles on the planned path; if there are obstacles, generate a new planned path through the large model, and control the robot arm to execute the new planned path.

[0134] In one implementation, the information acquisition module 101 is further used to:

[0135] Acquire environmental data and planning data of the sensor; the planning data includes a starting position and a target position; generate a three-dimensional environmental model based on the environmental data; the three-dimensional environmental model includes obstacle positions; map the starting position and the target position to the three-dimensional environmental model to obtain the working information of the robotic arm.

[0136] In one embodiment, the pre-trained large model is a large-scale language model based on the Transformer architecture.

[0137] In one implementation, the path planning module 102 is further configured to:

[0138] Acquire a pre-trained large model; based on the task requirements of the pre-trained large model, update the input layer and output layer of the pre-trained large model; the updated output layer outputs a path point sequence; acquire a planning data set, the planning data set includes sample work information and annotated paths; input the sample work information into the pre-trained large model to obtain a predicted path; based on the predicted path and the annotated path, calculate the overall loss; iterate the pre-trained large model according to the overall loss until the loss converges.

[0139] The above-mentioned embodiments of the present application provide a robotic arm control device based on a pre-trained large model and a robotic arm control method based on a pre-trained large model provided in an embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0140] The above describes the internal functions and structure of a robotic arm control device based on a pre-trained large model, such as Figure 3 As shown, in practice, the robotic arm control device based on the pre-trained large model can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0141] The memory 301 may be configured to store programs.

[0142] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0143] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0144] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0145] Get the working information of the robot arm;

[0146] The work information is input into a large model to obtain a planned path; the large model is obtained by fine-tuning a pre-trained large model based on training data;

[0147] Control the robotic arm to execute the planned path.

[0148] In one implementation, the processor 303 is further configured to:

[0149] Perform path optimization and smoothing on the planned path.

[0150] In one implementation, the processor 303 is further configured to:

[0151] Calculate the total length of the path point sequence as the path length loss; calculate the distance between the path point and the obstacle as the obstacle avoidance loss; calculate the curvature change of the path point sequence as the smoothness loss; optimize the planned path based on the path length loss, obstacle avoidance loss and smoothness loss.

[0152] In one implementation, the processor 303 is further configured to:

[0153] Monitor the motion state and working information of the robot arm in real time; determine whether there are obstacles on the planned path; if there are obstacles, generate a new planned path through the large model, and control the robot arm to execute the new planned path.

[0154] In one implementation, the processor 303 is further configured to:

[0155] Acquire environmental data and planning data of the sensor; the planning data includes a starting position and a target position; generate a three-dimensional environmental model based on the environmental data; the three-dimensional environmental model includes obstacle positions; map the starting position and the target position to the three-dimensional environmental model to obtain the working information of the robotic arm.

[0156] In one embodiment, the pre-trained large model is a large-scale language model based on the Transformer architecture.

[0157] In one implementation, the processor 303 is further configured to:

[0158] Acquire a pre-trained large model; based on the task requirements of the pre-trained large model, update the input layer and output layer of the pre-trained large model; the updated output layer outputs a path point sequence; acquire a planning data set, the planning data set includes sample work information and annotated paths; input the sample work information into the pre-trained large model to obtain a predicted path; based on the predicted path and the annotated path, calculate the overall loss; iterate the pre-trained large model according to the overall loss until the loss converges.

[0159] In this application, Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0160] The electronic device provided in this embodiment is based on the same inventive concept as the robotic arm control method based on a pre-trained large model provided in an embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored therein.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CDROM, optical storage, etc.) that contain computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0167] The present application also provides a computer-readable storage medium corresponding to a robotic arm control method based on a pre-trained large model provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute a robotic arm control method based on a pre-trained large model provided in any of the aforementioned embodiments.

[0168] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CDROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0169] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and a robotic arm control method based on a pre-trained large model provided in an embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0170] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0171] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0172] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A robotic arm control method based on a pre-trained large model, characterized in that: include: Get the working information of the robot arm; Inputting the work information into the large model to obtain a planned path; The large model is obtained by fine-tuning the pre-trained large model based on the training data; Control the robotic arm to execute the planned path.

2. The robotic arm control method based on a pre-trained large model according to claim 1, characterized in that: After the work information is input into the large model to obtain the planned path, the method further includes: Perform path optimization and smoothing on the planned path.

3. The robotic arm control method based on a pre-trained large model according to claim 2, characterized in that: The step of performing path optimization and smoothing on the planned path includes: Calculate the total length of the path point sequence as the path length loss; Calculate the distance between the path point and the obstacle as the obstacle avoidance loss; Calculate the curvature change of the path point sequence as the smoothness loss; The planned path is optimized based on path length loss, obstacle avoidance loss and smoothness loss.

4. The robotic arm control method based on a pre-trained large model according to any one of claim 13, characterized in that: After the controlling the mechanical arm to execute the planned path, the method further includes: Real-time monitoring of the motion status and working information of the robotic arm; Determine whether there are obstacles on the planned path; In the presence of obstacles, a new planning path is generated through the large model, and the robotic arm is controlled to execute the new planning path.

5. The robotic arm control method based on a pre-trained large model according to any one of claim 13, characterized in that: The obtaining of the working information of the robot arm includes: Acquire environmental data and planning data of the sensor; the planning data includes a starting position and a target position; Based on the environmental data, a three-dimensional environmental model is generated; the three-dimensional environmental model includes the location of obstacles; The starting position and target position are mapped to the three-dimensional environment model to obtain the working information of the robotic arm.

6. The robotic arm control method based on a pre-trained large model according to any one of claim 13, characterized in that: The pre-trained large model is a large-scale language model based on the Transformer architecture.

7. The robotic arm control method based on a pre-trained large model according to any one of claim 13, characterized in that: The fine-tuning process of the pre-trained large model includes: Get a pre-trained large model; Based on the task requirements of the pre-trained large model, the input layer and the output layer of the pre-trained large model are updated; the updated output layer outputs a sequence of path points; Acquire a planning data set, wherein the planning data set includes sample work information and annotated paths; Input the sample work information into the pre-trained large model to obtain the prediction path; Calculate the overall loss based on the predicted path and the annotated path; Iteratively pre-train the large model based on the overall loss until the loss converges.

8. A robotic arm control device based on a pre-trained large model, characterized in that: include: An information acquisition module, which is used to obtain the working information of the robotic arm; A path planning module, which is used to input the work information into the large model to obtain a planned path; The large model is obtained by fine-tuning the pre-trained large model based on the training data; A robot arm control module is used to control the robot arm to execute the planned path.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Get the working information of the robot arm; The work information is input into a large model to obtain a planned path; the large model is obtained by fine-tuning a pre-trained large model based on training data; Control the robotic arm to execute the planned path.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement a robotic arm control method based on a pre-trained large model as described in any one of claim 17.

Images