Robotic arm control method, device, equipment and storage medium

Abstract

The present application discloses a robot arm control method, device, equipment and storage medium, which relates to the field of intelligent control technology. The present application discloses a robot arm control method, including: acquiring voice information and image information; analyzing the voice information and image information based on a large language model to obtain the robot arm's grasping task target information; obtaining the robot arm's grasping posture according to the grasping task target information; and controlling the robot arm to complete the grasping task based on the robot arm's grasping posture controller. The method analyzes the acquired external voice information and environmental images through a large language model, understands the matching targets in the voice information and environmental images, automatically generates a grasping task according to the position of the matching target in the image, makes full use of the large language model to process the task input information, identifies the task target and generates the grasping task information, improves the success rate of the grasping task, and broadens the task scenario.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and in particular to a robot arm control method, device, equipment and storage medium. Background Art

[0002] In recent years, the related technologies of large language models have received widespread attention and are an important breakthrough in the field of artificial intelligence. Large models can understand human language input and generate high-quality response content. This technology provides new possibilities for the mode of human-computer interaction. With the development of the field of large language models, multimodal large models have achieved new breakthroughs in task understanding. They can not only understand complex natural language instructions, but also process image information, analyze environmental scenes, estimate task goals, and further generate new sentences and even code information. Combined with multimodal large language models, robots can perform a series of complex and highly intelligent tasks, greatly improving the naturalness and fluency of human-computer interaction. Related technologies have received widespread attention and application in the field of embodied intelligence, but the current related research results are limited to laboratory scenarios, and are still far from being popularized in human life and commercialized in the market.

[0003] At present, the overall task execution process of the robot mainly includes two levels: the top-level high-level strategy formulation, and the bottom-level sensor perception and controller execution of the interaction with the environment. The relevant technology has always been a hot topic in academia. With the advent of ChatGPT, a new possibility has emerged in the interaction mode between the top-level and bottom-level paradigms of the robot. However, the control of the robotic arm grasping task is still not intelligent and flexible enough, and requires clear instructions to operate, resulting in the poor intelligence of the robotic arm grasping task and cannot be well applied in practice.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a robot arm control method, device, equipment and storage medium, aiming to solve the technical problems that the current robot arm grasping tasks require clear instructions and the robot arm grasping tasks have poor intelligence and cannot be well applied in practice.

[0006] To achieve the above objectives, the present application proposes a robot arm control method, the robot arm control method comprising:

[0007] Obtain voice information and image information;

[0008] Analyze the voice information and the image information based on a large language model to obtain the grasping task target information of the robot arm;

[0009] Obtaining the grasping posture of the robot arm according to the grasping task target information;

[0010] Based on the robotic arm grasping posture controller, the robotic arm is controlled to complete the grasping task.

[0011] In one embodiment, the analyzing the voice information and the image information based on the large language model to obtain the grasping task target information of the robot arm includes:

[0012] Recognize keyword information according to the voice information, extract color and depth information from the image information, and obtain supplementary constraints;

[0013] Matching the voice information and the image information based on the supplementary constraint to obtain a target object;

[0014] Mission target information is obtained based on the keyword information and the image information, and the mission target information includes a target object category, a name, and camera coordinate information.

[0015] In one embodiment, obtaining the grasping posture of the robot arm according to the grasping task target information includes:

[0016] Extracting a reference image of the target object according to the camera coordinate information of the grasping task target information;

[0017] The robot arm grasping posture is obtained according to the reference image and the camera coordinate information in the grasping task target information.

[0018] In one embodiment, controlling the robotic arm to complete the grasping task according to the grasping posture of the robotic arm includes:

[0019] Obtain the preset hand-eye calibration matrix and the preset inverse kinematics matrix of the robot arm;

[0020] Performing coordinate transformation on the manipulator grasping posture according to the preset hand-eye calibration matrix and the preset inverse kinematics matrix to obtain the manipulator base coordinates of the manipulator grasping posture;

[0021] The robotic arm is controlled to perform a grasping action based on the robotic arm base coordinates to complete a grasping task.

[0022] In one embodiment, the coordinate transformation of the robot grasping posture according to the preset hand-eye calibration matrix and the preset inverse kinematics matrix to obtain the robot base coordinates of the robot grasping posture includes:

[0023] Converting the robot arm grasping posture into robot arm space coordinates according to the preset hand-eye calibration matrix;

[0024] The robot arm space coordinates are converted into robot arm base coordinates in a robot arm base coordinate system according to the preset inverse kinematics matrix.

[0025] In one embodiment, controlling the robotic arm to perform a grasping action based on the robotic arm base coordinates includes:

[0026] Obtain the current state of the robot arm and obtain the expected state according to the base coordinates of the robot arm;

[0027] Generate a desired trajectory according to the current state, the desired state, and a robot arm dynamics model;

[0028] The robotic arm is controlled to perform a grasping action based on the expected trajectory.

[0029] In one embodiment, controlling the robotic arm to perform a grasping action based on the expected trajectory includes:

[0030] Fitting the time-varying output constraints based on the adaptive neural network, constructing the control rate based on the synovial surface;

[0031] The robot arm is controlled to perform a grasping action according to the time-varying output constraint, the control rate, and the desired trajectory.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a robot arm control device, the robot arm control device comprising:

[0033] An acquisition module, used for acquiring voice information and image information;

[0034] A model analysis module, used to analyze the voice information and the image information based on a large language model to obtain the grasping task target information of the robot arm;

[0035] A robotic arm control module, used to obtain the robotic arm grasping posture according to the grasping task target information;

[0036] The robotic arm control module is also used to control the robotic arm to complete the grasping task based on the robotic arm grasping posture controller.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a robotic arm control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the robotic arm control method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the robot arm control method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the robot arm control method described above are implemented.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] Through the large language model, the external voice information and environmental images obtained are analyzed, the matching targets in the voice information and environmental images are understood, and the grasping task is automatically generated according to the position of the matching target in the image. The large language model is fully utilized to process the task input information, identify the task target and generate the grasping task information, which improves the success rate of the grasping task and broadens the task scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a flow chart of a first embodiment of a robot arm control method of the present application;

[0045] Figure 2 This is a diagram showing the actual acquisition of voice information and image information by the robot arm provided in the first embodiment of the robot arm control method of the present application;

[0046] Figure 3 A schematic diagram of a robotic arm performing a grasping action based on a target task provided in Example 1 of the robotic arm control method of the present application;

[0047] Figure 4 A schematic diagram of the overall control logic architecture of a robotic arm provided in Embodiment 1 of the robotic arm control method of the present application;

[0048] Figure 5 A schematic diagram of a flow chart provided for the second embodiment of the robot arm control method of the present application;

[0049] Figure 6 A schematic diagram of changes in a robot arm input signal during a control process provided in Embodiment 2 of the robot arm control method of the present application;

[0050] Figure 7A schematic diagram of a comparison diagram of joint output errors provided in Example 2 of the robot arm control method of the present application;

[0051] Figure 8 A comparison diagram of whether to compensate for the hysteresis effect provided in the second embodiment of the robot arm control method of the present application;

[0052] Fig. 9 A schematic diagram showing the influence of different parameters η on the convergence time when the position error e11 is provided in the second embodiment of the robot arm control method of the present application;

[0053] Fig.10 A schematic diagram showing the influence of different parameters η on the convergence time when the position error e12 is provided in the second embodiment of the robot arm control method of the present application;

[0054] Fig.11 This is a schematic diagram of the module structure of the robot arm control device according to an embodiment of the present application;

[0055] Fig.12 Schematic diagram of the device structure of the hardware operating environment involved in the robot arm control method in the embodiment of the present application. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0057] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solutions of the embodiments of the present application are: obtaining voice information and image information; analyzing the voice information and image information based on a large language model to obtain the grasping task target information of the robotic arm; obtaining the grasping posture of the robotic arm according to the grasping task target information; and controlling the robotic arm to complete the grasping task based on the robotic arm grasping posture controller.

[0059] In this embodiment, for ease of description, the following description is made by taking the identification of the robot arm control device as the execution subject.

[0060] Due to the existing technology, in recent years, the related technologies of large language models have received widespread attention and are an important breakthrough in the field of artificial intelligence. Large models can understand human language input and generate high-quality response content. This technology provides new possibilities for the mode of human-computer interaction. With the development of the field of large language models, multimodal large models have achieved new breakthroughs in task understanding. They can not only understand complex natural language instructions, but also process image information, analyze environmental scenes, estimate task goals, and further generate new sentences and even code information. Combined with multimodal large language models, robots can perform a series of complex and highly intelligent tasks, greatly improving the naturalness and fluency of human-computer interaction. Related technologies have received widespread attention and application in the field of embodied intelligence, but the current related research results are limited to laboratory scenarios, and are still far from large-scale popularization in human life and market commercialization.

[0061] At present, the overall task execution process of the robot mainly includes two levels: the top-level high-level strategy formulation, and the bottom-level sensor perception and controller execution of the interaction with the environment. The relevant technology has always been a hot topic in academia. With the advent of ChatGPT, a new possibility has emerged in the interaction mode between the top-level and bottom-level paradigms of the robot. However, the control of the robotic arm grasping task is still not intelligent and flexible enough, and requires clear instructions to operate, resulting in the poor intelligence of the robotic arm grasping task and cannot be well applied in practice.

[0062] The present application provides a solution, which obtains external voice information and environmental images through large language model analysis, understands the matching targets in the voice information and environmental images, automatically generates a grasping task according to the position of the matching target in the image, makes full use of the large language model to process task input information, identifies the task target and generates grasping task information, improves the success rate of the grasping task, and broadens the task scenario.

[0063] It can be seen from the above embodiments that the present application discloses a robot arm control method, device, equipment and storage medium, which relates to the field of intelligent control technology, and discloses a robot arm control method, including: obtaining voice information and image information; analyzing the voice information and image information based on a large language model to obtain the robot arm's grasping task target information; obtaining the robot arm's grasping posture according to the grasping task target information; controlling the robot arm to complete the grasping task based on the robot arm's grasping posture controller. The method analyzes the acquired external voice information and environmental images through a large language model, understands the matching targets in the voice information and environmental images, and automatically generates a grasping task according to the position of the matching target in the image, makes full use of the large language model to process the task input information, identifies the task target to generate grasping task information, improves the success rate of the grasping task, and broadens the task scenario.

[0064] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a robotic arm control device, etc. The following takes the robotic arm control device as an example to illustrate this embodiment and the following embodiments.

[0065] Based on this, the present application embodiment provides a method for controlling a robotic arm, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the robot arm control method of the present application.

[0066] In this embodiment, the robot arm control method includes steps S10 to S40:

[0067] Step S10, acquiring voice information and image information.

[0068] It should be noted that the voice information can be the user's speech to the robotic arm, which captures external human voice in real time and uses the voice as input information.

[0069] It should be emphasized that the image information may be an image of the surrounding environment acquired in real time by the image acquisition device of the robot arm, and the environment image is used as the image information.

[0070] It should be further explained that the robotic arm captures the sound information of people around it in real time, and can also capture the surrounding environmental information in real time and synchronously. It can also trigger the image acquisition device after recognizing the voice information, and then collect the surrounding environmental image information.

[0071] Step S20, analyzing the voice information and the image information based on the large language model to obtain the grasping task target information of the robot arm.

[0072] It is understandable that large language models (LLMs) refer to deep learning models trained with large amounts of text data. These models usually have hundreds of millions or more parameters, can understand and generate human language, and perform tasks such as translation, summary generation, question answering, and dialogue. With the increase in computing resources and the advancement of algorithms, large language models have achieved remarkable achievements in the field of natural language processing (NLP).

[0073] It should be noted that in the overall task execution process of the robot, it mainly includes the top-level high-level strategy formulation, and the bottom-level sensor perception and controller execution of the interaction with the environment. The relevant technology has always been a hot topic in the academic community. With the advent of ChatGPT, a new possibility has emerged in the interaction mode between the top and bottom paradigms of the robot. At the level of task perception and planning, the world has launched a discussion on the application capabilities of large language models. Multimodal large language models can receive images as input information and have excellent performance in image understanding capabilities. Considering the task scenario of object grasping, guided by the task goal, using the task understanding ability and rich semantic knowledge of the large language model, a task-oriented grasping task prediction is constructed. With the help of the large language model, in some task scenarios, the robot can better simulate the real thinking mode and operation means of humans, greatly improving the executability of the task. Using large language models to optimize data analysis and decision-making processes has greatly expanded the task movement scenarios of drones, allowing drones to achieve autonomous information collection and control effects in disaster scenarios and scenarios with difficult communications.

[0074] It should be understood that voice information and image information can be the input of the large language model. The trained large language model can generate control tasks / control instructions that the robotic arm can understand for the voice information and image information, thereby controlling the robotic arm to perform the grasping task.

[0075] It should be noted that the target information of the grasping task may be information such as the position of the target object and the image of the target object.

[0076] In a feasible implementation, step S20 may include steps A21 to A23:

[0077] Step A21, recognizing keyword information according to the voice information, performing color extraction and depth information extraction on the image information, and obtaining supplementary constraints.

[0078] It is understandable that the keyword information may be information related to the crawling task in the voice information. For example, the voice information may be “help me organize the files”, wherein the keyword may be understood as files.

[0079] In a specific implementation, the supplementary constraint may be information obtained based on the image information to supplement the keyword information, and may be specification information of the object in the image, a coordinate origin, output information specifications, and the like.

[0080] In the specific implementation, the input signal of the grasping task is considered to be human voice input. First, the trigger keyword is set. When the keyword information is recognized, the operator's voice signal is converted into text information through the voice recognition module, which is stored and used as part of the text input information of the large model. The monitoring voice recognition module, when activated, triggers the visual device of the robot arm to shoot the task scene and extract the RGBD information of the shot picture. Finally, the shot picture is transmitted to the multimodal large model as the picture input of the large model.

[0081] It should be emphasized that voice information and image information are the input information of the large language model. In order to ensure the consistency of the large model output and obtain more stable grasping task performance, it is necessary to further supplement the prompt word input of the large model. Combined with the specific camera and robotic arm equipment of the system, it is necessary to supplement the image specification information, coordinate origin, and output information specifications in the prompt word.

[0082] Step A22, matching the voice information and the image information based on the supplementary constraint to obtain a target object.

[0083] It is understandable that the image information may include the category and coordinates of objects in the identification image.

[0084] It should be noted that matching the voice information and the image information based on the supplementary constraint may be matching objects in the image and keywords in the voice.

[0085] It should be noted that the large language model has been trained to be mature and can quickly extract information from images, extract keywords from text content, and match the two. This embodiment does not limit the training of the large language model and will not elaborate on it here.

[0086] It is understandable that the target object may be the object in the image that has the highest matching degree with the speech information.

[0087] Step A23, obtaining task target information based on the keyword information and the image information, wherein the task target information includes target object category, name and camera coordinate information.

[0088] In the specific implementation, the large model analyzes the voice information input by the operator based on the recognized text information and the supplementary constraint requirements, determines the task goal, and identifies the target object that best matches the voice input. After completing the target task judgment, the object is matched according to the image information provided by the visual module. If there is object information that meets the task requirements, the target bounding box is delineated, and the coordinates of the geometric center point of the object are calculated based on the bounding box and image information, and the obtained information is output. In the specific implementation, the output of the large model can be: [target object category, name: x-axis coordinate information, y-axis coordinate information].

[0089] In this implementation, by matching the voice information and image information of the input large language model to determine the grasping target of the robotic arm, it is possible to more flexibly identify user needs and determine task goals based on user needs, thereby flexibly executing grasping tasks in more scenarios.

[0090] The above are only feasible implementations of step S20 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S20.

[0091] Step S30, obtaining the grasping posture of the robot arm according to the grasping task target information.

[0092] It is understandable that the robot arm grasping posture can be a parameter that can be understood by the robot arm, and can be a quaternion array, which represents the specific posture of the end effector of the robot arm in space.

[0093] In a feasible implementation, step S30 may include steps A31-A32:

[0094] Step A31, extracting a reference image of the target object according to the camera coordinate information of the grasping task target information.

[0095] It is understandable that the reference image may be an image of the target object captured from the image information.

[0096] It should be noted that the image of the target object is captured separately to facilitate the robot arm to accurately identify it from the environmental image, rather than marking the target image from the image information, to avoid unclear marking and incorrect grasping by the robot arm due to the close distance of multiple objects.

[0097] Step A32, obtaining the robot arm grasping posture according to the reference image and the camera coordinate information in the grasping task target information.

[0098] It should be noted that the robot arm grasping posture obtained according to the reference image and the camera coordinate information in the grasping task target information can be based on the coordinate information of the target object returned by the large model, and based on the trained grasping neural network and the image information of the surrounding environment taken by the visual device, the optimal grasping posture of the object at the specified coordinate point is calculated, and a quaternion array is returned, which represents the specific posture of the robot arm end effector in space.

[0099] It should be further explained that, in the process of generating the grasping posture of the robot arm according to the position of the target object in the image information by the trained grasping neural network, the accuracy of the generated posture can be continuously optimized by updating the weights of the neural network, thereby improving the grasping success rate of the robot arm.

[0100] It should be emphasized that the design of the neural network weight update rate can refer to the following formula:

[0101]

[0102] Among them, г is a positive diagonal matrix, S is a radial basis function, Z is the network input, σ is the synergy gain coefficient, and the hysteresis parameter is set as:

[0103]

[0104] In this embodiment, a large language model is used to identify the grasping task that may be pointed to in the user's voice information, and the location of the target task is further analyzed through image information. The grasping posture of the robot arm is generated according to the target position based on a neural network, so as to accurately grasp the target object.

[0105] The above are only feasible implementations of step S30 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S30.

[0106] Step S40: controlling the robotic arm to complete the grasping task based on the robotic arm grasping posture controller.

[0107] It should be noted that after completing the large model task processing and the grasping neural network calculation, the robot arm movement process is triggered in the form of a topic. The robot arm obtains the coordinate information transmitted by the topic, combines the hand-eye calibration matrix and the robot arm inverse kinematics matrix, and converts the coordinates in the task plane space into the position information of the robot arm base coordinate system.

[0108] It should be emphasized that starting from the observation position, according to the coordinate information, the end mechanism of the robot arm first moves to the top of the grasping target, then moves closer to the target, and moves accurately to the grasping position, so that the end mechanism reaches the best grasping position. After reaching the grasping position, the end actuator closes to complete the object grasping, and the robot arm continues to move to the end position and opens the end gripper. After placing the object at the target location, the robot arm returns to the initial observation position and waits for the next execution topic command.

[0109] In the specific implementation, the actual display diagram of the robot arm obtaining voice information and image information can be referred to Figure 2 , the schematic diagram of the robot arm performing the grasping action based on the target task can be referred to Figure 3 .

[0110] It should be noted that the overall control logic architecture diagram of the robotic arm can be referenced Figure 4In the figure, the coordinate system transformation is performed according to the kinematic model of the robot arm to train the adaptive neural network controller; there are keywords in the voice information of the operator or the user, which are input into the large language model. At the same time, the keywords trigger the visual module, collect the environmental image, and pass the environmental image to the neural network of the grasping model. At the same time, the environmental image is also input into the large language model to supplement the user's voice information, and the grasping target based on the user's intention is obtained. Based on the grasping target, the robot arm grasping trajectory is generated by the adaptive neural network controller and the grasping model neural network, that is, the motion planning in the figure, and the grasping command is executed based on the motion planning.

[0111] This embodiment provides a robotic arm control method, which obtains external voice information and environmental images through large language model analysis, understands the matching targets in the voice information and environmental images, automatically generates a grasping task according to the position of the matching target in the image, and makes full use of the large language model to process task input information, identify task targets and generate grasping task information, thereby improving the success rate of grasping tasks and broadening task scenarios.

[0112] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 5 , step S40 also includes steps S41 to S43:

[0113] Step S41, obtaining a preset hand-eye calibration matrix and a preset inverse kinematics matrix of the robotic arm.

[0114] It is understandable that the robotic arm has three coordinate systems, namely, the coordinate system where the robotic arm image acquisition device is located, the coordinate system where the gripper at the end of the robotic arm is located, and the coordinate system where the robotic arm base is located.

[0115] It should be understood that the preset hand-eye calibration matrix can be a transformation matrix between the coordinate system where the image acquisition device is located and the coordinate system where the end gripper of the robotic arm is located; the inverse kinematics matrix can be a transformation matrix between the coordinate system where the end gripper of the robotic arm is located and the coordinate system where the base of the robotic arm is located.

[0116] Step S42, performing coordinate transformation on the robot arm grasping posture according to the preset hand-eye calibration matrix and the preset inverse kinematics matrix to obtain the robot arm base coordinates of the robot arm grasping posture.

[0117] It can be understood that the robot base coordinates of the robot grasping posture refer to the posture of the robot grasping posture relative to the robot base coordinate system.

[0118] In a feasible implementation, step S42 may include steps A421-A422:

[0119] Step A421, converting the robot arm grasping posture into robot arm space coordinates according to the preset hand-eye calibration matrix.

[0120] It should be understood that the robot arm spatial coordinates may be the coordinates at the end of the robot arm.

[0121] It is understandable that hand-eye calibration is a key technology in robot vision. It is used to determine the relative position relationship between the camera (eye) and the robot end effector (hand). By presetting the hand-eye calibration matrix, the points in the camera coordinate system can be converted to the robot base coordinate system.

[0122] It should be noted that the robot arm also needs to know how to adjust its posture to correctly grasp the object. Therefore, you may also need to calculate the posture (rotation) of the target object in the robot arm base coordinate system, which can be done through the hand-eye calibration matrix. According to the calculated position and posture of the target object in the robot arm base coordinate system, send corresponding instructions to the robot arm controller to move the robot arm to the specified position and adjust to the correct posture to complete the grasping action.

[0123] Step A422: convert the robot space coordinates into robot base coordinates in the robot base coordinate system according to the preset inverse kinematics matrix.

[0124] It can be understood that converting the robot space coordinates into the robot base coordinates in the robot base coordinate system according to the preset inverse kinematics matrix may be converting the coordinates of the midpoint of the robot space coordinates into the robot base coordinate system.

[0125] It should be noted that, through two coordinate transformations, the coordinates of the target object in the image are converted to spatial coordinates in the robot arm base coordinate system, and the control trajectory is generated based on the spatial coordinates and the grasping action is performed.

[0126] In this embodiment, the position of the target object in the image is converted into the spatial coordinates of the robot arm base coordinate system through two spatial transformations. Based on the spatial coordinate system, a grasping action trajectory that meets the grasping task is generated according to the neural network, thereby completing the grasping action of the target object.

[0127] The above are merely feasible implementations of step S42 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S42.

[0128] Step S43, controlling the robotic arm to perform a grasping action based on the robotic arm base coordinates to complete the grasping task.

[0129] It is understandable that a collision-free path to the target position is planned according to the robot base coordinates, and RRT (Rapidly-exploring Random Trees), A* algorithm or other path planning techniques may be used.

[0130] Furthermore, the path planning result is converted into a joint angle sequence or an end effector position sequence in a time series, that is, a specific motion trajectory is generated.

[0131] In a feasible implementation, step S43 may include steps A431 to A433:

[0132] Step A431, obtain the current state of the robot arm, and obtain the expected state according to the basic coordinates of the robot arm.

[0133] It should be noted that the expected state may be a state corresponding to the grasping action of the robot arm corresponding to the spatial coordinates of the robot arm base coordinate system converted according to the grasping posture of the robot arm.

[0134] It is understandable that the current state of the robot arm may be a state corresponding to the spatial coordinates of the current robot arm posture under the robot arm base coordinates.

[0135] Step A432, generating a desired trajectory according to the current state, the desired state and the robot arm dynamics model.

[0136] It is understandable that the parameters of the robot model and the moment of inertia of the robot can be obtained, and the robot dynamics model can be constructed according to the parameters of the robot model and the moment of inertia of the robot.

[0137] It should be noted that the system parameter matrix in the dynamic model is:

[0138]

[0139] in:

[0140]

[0141] Among them, C represents the Coriolis force and centrifugal force matrix, G represents the gravity vector and M represents the mass matrix. The three matrices or vectors together describe the forces and torques exerted on the robot during movement, which is crucial for achieving precise control and simulation.

[0142] Furthermore, the parameters of the simulated robot arm model are: the mass m1 = 2.00 kg and the length l1 = 0.35 m of the connecting rod 1, and the mass m1 = 0.85 kg and the length l1 = 0.31 m of the connecting rod 2. I1 and I2 correspond to the moments of inertia of the two rods, respectively.

[0143] Step A433, controlling the robotic arm to perform a grasping action based on the expected trajectory to complete the grasping task.

[0144] It should be noted that controlling the robotic arm to perform a grasping action based on the expected trajectory to complete the grasping task includes: fitting a time-varying output constraint according to an adaptive neural network, and constructing a control rate according to a synovial surface; and controlling the robotic arm to perform a grasping action according to the time-varying output constraint, the control rate, and the expected trajectory.

[0145] It should be noted that a set of kinematic and dynamic models of isomorphic manipulators and several Cartesian space trajectory models of expected regression are constructed, and each expected regression trajectory is defined as an execution state. According to the constructed kinematic model of the manipulator, the spatial trajectory coordinates in the Cartesian coordinate system are converted into coordinates in the joint space, and then the controller is designed. The controller form under the constraint condition is designed using an adaptive neural network algorithm.

[0146] It is understandable that in this embodiment, a mechanical arm dynamics model is constructed for the mechanical arm model. At the same time, when constructing the mechanical arm dynamics model, the repeatability of the grasping motion is also taken into account, and the mechanical arm dynamics model mathematics of the hysteresis effect is introduced.

[0147] It should be noted that the trajectory generated by the trained neural network is a motion trajectory generated based on the dynamic model that considers the hysteresis effect of the robot arm, and the control rate of the robot arm is adjusted by constructing a fast sliding membrane through the sliding membrane surface. At the same time, the time-varying output constraints and the control rate of the robot arm are also considered to generate the corresponding operation trajectory.

[0148] It should be emphasized that, considering the repeatability of the grasping motion, the mathematical description of the robot arm dynamics model with hysteresis effect is:

[0149]

[0150] in, They represent joint angle vector, velocity vector and acceleration vector respectively. They are the positive definite inertia matrix, the Coriolis force and centrifugal force matrix, and the gravity matrix respectively. is the input signal of the system, and the input signal is affected by the hysteresis effect. and To simplify the corresponding terms in the dynamic equation. Finally, regarding the output signal under hysteresis effect The mathematical expression is

[0151]

[0152] in, is a positive number and satisfies the condition , the hysteresis equation can be solved.

[0153]

[0154]

[0155] Then, the hysteresis characteristic can be rewritten as

[0156]

[0157] in , .

[0158] Further, according to the model information obtained in step 1, let , the kinetic model is:

[0159]

[0160] Define the position error and speed error as:

[0161]

[0162] Among them, the design of virtual control law for

[0163]

[0164] Where: K1 is the gain coefficient:

[0165]

[0166] The considered output error e1 will be subject to the time-varying output constraint, namely ,in .

[0167] Furthermore, the fast sliding film method is used to construct the control law, and the sliding film surface is constructed as

[0168]

[0169] Among them, the gain coefficient and positive definite matrix are as follows:

[0170]

[0171] Among them, p and q are positive odd numbers and satisfy the condition .

[0172] The final specific control law form is:

[0173]

[0174] in:

[0175]

[0176]

[0177] Jacobian matrix:

[0178]

[0179] The expected trajectory of the robot arm is

[0180]

[0181] In the specific implementation, Figure 6 It shows the changes of the input signal of the robot arm during the control process. Figure 7 This is a comparison chart of joint output errors. Compared with the controller designed based on quadratic Lyapunov function, the output performance of the robot arm is further improved by combining the obstacle Lyapunov function. Figure 8 This is a comparison chart of whether the hysteresis effect is compensated. Using a neural network to compensate for the hysteresis signal can further improve the stability of the system and improve the overall accuracy of the system. Fig. 9 and Fig.10 The influence of different parameters η on the system convergence time is shown. The specific parameter settings are shown in Table 1:

[0182] Table 1

[0183]

[0184] In this embodiment, the neural network for generating the grasping path is optimized and trained by taking into account the hysteresis effect and time-varying output constraints of the robotic arm, thereby obtaining a grasping path of the robotic arm that meets the requirements of the grasping task.

[0185] The above are only feasible implementations of step S43 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S43.

[0186] This embodiment provides a method for controlling a robotic arm. By adopting a neural network method for a controller with time-varying output constraints and hysteresis effects, an adaptive neural network is constructed to fit unknown constraints, so as to generate a robotic arm grasping path that meets the requirements of the grasping task. At the same time, on this basis, combined with the fast terminal sliding film technology, the response speed of the system output signal is improved, and the safety and stability of the system are guaranteed.

[0187] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the robot arm control method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0188] This application also provides a robot arm control device, please refer to Fig.11 , the robot arm control device comprises:

[0189] An acquisition module 10, used to acquire voice information and image information;

[0190] A model analysis module 20, used to analyze the voice information and the image information based on a large language model to obtain the grasping task target information of the robot arm;

[0191] A robot arm control module 30, used to obtain a robot arm grasping posture according to the grasping task target information;

[0192] The robotic arm control module 30 is further used to control the robotic arm to complete the grasping task based on the robotic arm grasping posture controller.

[0193] The robot arm control device provided by the present application adopts the robot arm control method in the above embodiment, which can solve the technical problem that the current robot arm grasping task requires clear instructions and the robot arm grasping task has poor intelligence and cannot be well applied in practice. Compared with the prior art, the beneficial effects of the robot arm control device provided by the present application are the same as the beneficial effects of the robot arm control method provided by the above embodiment, and the other technical features in the robot arm control device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0194] The model analysis module 20 is further used to identify keyword information according to the voice information, extract color and depth information from the image information, and obtain supplementary constraints;

[0195] Matching the voice information and the image information based on the supplementary constraint to obtain a target object;

[0196] Mission target information is obtained based on the keyword information and the image information, and the mission target information includes a target object category, a name, and camera coordinate information.

[0197] The robot arm control module 30 is further used to extract a reference image of the target object according to the camera coordinate information of the grasping task target information;

[0198] The robot arm grasping posture is obtained according to the reference image and the camera coordinate information in the grasping task target information.

[0199] The robot arm control module 30 is also used to obtain a preset hand-eye calibration matrix and a preset inverse kinematics matrix of the robot arm;

[0200] Performing coordinate transformation on the manipulator grasping posture according to the preset hand-eye calibration matrix and the preset inverse kinematics matrix to obtain the manipulator base coordinates of the manipulator grasping posture;

[0201] The robotic arm is controlled to perform a grasping action based on the robotic arm base coordinates to complete a grasping task.

[0202] The robotic arm control module 30 is further used to convert the robotic arm grasping posture into robotic arm space coordinates according to the preset hand-eye calibration matrix;

[0203] The robot arm space coordinates are converted into robot arm base coordinates in a robot arm base coordinate system according to the preset inverse kinematics matrix.

[0204] The robot control module 30 is also used to obtain the current state of the robot and obtain the desired state according to the robot base coordinates;

[0205] Generate a desired trajectory according to the current state, the desired state, and a robot arm dynamics model;

[0206] The robotic arm is controlled to perform a grasping action based on the expected trajectory to complete the grasping task.

[0207] The robot arm control module 30 is also used to fit the time-varying output constraints according to the adaptive neural network and construct the control rate according to the synovial surface;

[0208] The robot arm is controlled to perform a grasping action according to the time-varying output constraint, the control rate, and the desired trajectory.

[0209] The present application provides a robotic arm control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the robotic arm control method in the above-mentioned embodiment one.

[0210] Reference below Fig.12 , which shows a schematic diagram of the structure of a mechanical arm control device suitable for implementing the embodiment of the present application. The mechanical arm control device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The robot arm control device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0211] like Fig.12 As shown, the robot arm control device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the robot arm control device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the robot control device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a robot control device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0212] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0213] The robot arm control device provided by the present application adopts the robot arm control method in the above embodiment, which can solve the technical problem that the current robot arm grasping task requires clear instructions and the robot arm grasping task has poor intelligence and cannot be well applied in practice. Compared with the prior art, the beneficial effects of the robot arm control device provided by the present application are the same as the beneficial effects of the robot arm control method provided by the above embodiment, and the other technical features in the robot arm control device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0214] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0215] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0216] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the robot arm control method in the above-mentioned embodiment.

[0217] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0218] The computer-readable storage medium may be included in the robot arm control device; or may exist independently without being assembled into the robot arm control device.

[0219] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the robot arm control device, the robot arm control device: obtains voice information and image information; analyzes the voice information and the image information based on a large language model to obtain the robot arm's grasping task target information; obtains the robot arm's grasping posture according to the grasping task target information; and controls the robot arm to complete the grasping task based on the robot arm's grasping posture controller.

[0220] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0221] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0222] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0223] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned robot arm control method, and can solve the technical problem that the current robot arm grasping tasks require clear instructions, and the robot arm grasping tasks have poor intelligence and cannot be well applied in practice. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the robot arm control method provided by the above-mentioned embodiment, and will not be repeated here.

[0224] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned robot arm control method when executed by a processor.

[0225] The computer program product provided by this application can solve the technical problem that the current robot arm grasping task requires clear instructions and the robot arm grasping task has a poor intelligence level and cannot be well applied in practice. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the robot arm control method provided by the above embodiment, which will not be repeated here.

[0226] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A robot arm control method, characterized in that: The robot arm control method comprises: Obtain voice information and image information; Analyze the voice information and the image information based on a large language model to obtain the grasping task target information of the robot arm; The analyzing the voice information and the image information based on the large language model to obtain the grasping task target information of the robot arm includes: Recognize keyword information according to the voice information, extract color and depth information of the image information, and obtain supplementary constraints, wherein the supplementary constraints include specification information, coordinate origin, and output information specification of the target object image; Matching the voice information and the image information based on the supplementary constraint to obtain a target object; Obtaining task target information based on the keyword information and the image information, wherein the task target information includes a target object category, a name, and camera coordinate information; Obtaining the grasping posture of the robot arm according to the grasping task target information; Controlling the robotic arm to complete the grasping task based on the robotic arm grasping posture controller; The controlling the robotic arm to complete the grasping task based on the grasping posture of the robotic arm includes: Obtain the preset hand-eye calibration matrix and the preset inverse kinematics matrix of the robot arm; Performing coordinate transformation on the manipulator grasping posture according to the preset hand-eye calibration matrix and the preset inverse kinematics matrix to obtain the manipulator base coordinates of the manipulator grasping posture; Obtain the current state of the robot arm and obtain the expected state according to the base coordinates of the robot arm; Generate a desired trajectory according to the current state, the desired state, and a robot arm dynamics model; Fitting the time-varying output constraint according to the adaptive neural network, constructing a control rate according to the sliding membrane surface, wherein the control rate is used for hysteresis compensation in the process of controlling the robot arm; The robot arm is controlled to perform a grasping action according to the time-varying output constraint, the control rate, and the desired trajectory.

2. The robot arm control method according to claim 1, characterized in that: The obtaining of the robot arm grasping posture according to the grasping task target information includes: Extracting a reference image of the target object according to the camera coordinate information of the grasping task target information; The robot arm grasping posture is obtained according to the reference image and the camera coordinate information in the grasping task target information.

3. The robot arm control method according to claim 1, characterized in that: The step of performing coordinate transformation on the robot arm grasping posture according to the preset hand-eye calibration matrix and the preset inverse kinematics matrix to obtain the robot arm base coordinates of the robot arm grasping posture includes: Converting the robot arm grasping posture into robot arm space coordinates according to the preset hand-eye calibration matrix; The robot arm space coordinates are converted into robot arm base coordinates in a robot arm base coordinate system according to the preset inverse kinematics matrix.

4. A robot arm control device, characterized in that: The mechanical arm control device comprises: An acquisition module, used for acquiring voice information and image information; A model analysis module, used to analyze the voice information and the image information based on a large language model to obtain the grasping task target information of the robot arm; The model analysis module is further used to identify keyword information according to the voice information, perform color extraction and depth information extraction on the image information, and obtain supplementary constraints, wherein the supplementary constraints include specification information, coordinate origin, and output information specification of the target object image; match the voice information and the image information based on the supplementary constraints to obtain the target object; obtain task target information based on the keyword information and the image information, wherein the task target information includes the target object category, name, and camera coordinate information; A robotic arm control module, used to obtain the robotic arm grasping posture according to the grasping task target information; The robotic arm control module is further used to control the robotic arm to complete the grasping task based on the robotic arm grasping posture controller; The robot control module is also used to obtain a preset hand-eye calibration matrix and a preset inverse kinematics matrix of the robot; perform coordinate transformation on the robot grasping posture according to the preset hand-eye calibration matrix and the preset inverse kinematics matrix to obtain the robot base coordinates of the robot grasping posture; obtain the current state of the robot, and obtain the expected state according to the robot base coordinates; generate an expected trajectory according to the current state, the expected state and the robot dynamics model; fit the time-varying output constraints according to the adaptive neural network, and construct a control rate according to the synovial surface, wherein the control rate is used for hysteresis compensation in the robot control process; control the robot to perform a grasping action according to the time-varying output constraints, the control rate and the expected trajectory.

5. A robot arm control device, characterized in that: The device comprises: a memory, a processor, and a robot arm control program stored in the memory and executable on the processor, wherein the robot arm control program is configured to implement the robot arm control method according to any one of claims 1 to 3.

6. A storage medium, characterized in that: The storage medium stores a robot arm control program, and when the robot arm control program is executed by the processor, the robot arm control method according to any one of claims 1 to 3 is implemented.

Images