Mechanical arm control method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a mechanical arm control method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of obtaining simulation data of a mechanical arm simulation target task; acquiring first pose information of the mechanical arm at the (t-1) th moment of the target task, and performing spatio-temporal reasoning on the first pose information based on the simulation data to obtain second pose information of the mechanical arm at the t th moment of the target task; an image of the target point at the t-th moment is collected through a camera carried at the tail end of the mechanical arm; based on the image at the t-th moment, error prediction is conducted on the speed of the tail end of the mechanical arm, the speed error of the tail end of the mechanical arm at the t-th moment is obtained, the simulation speed of the tail end of the mechanical arm is corrected based on the speed error, and the estimated speed of the tail end of the mechanical arm at the t-th moment is obtained; and on the basis of the estimated speed, admittance inference is conducted on the first acceleration of the mechanical arm at the (t-1) th moment, the second acceleration of the mechanical arm at the t th moment is obtained, and the mechanical arm is driven to move on the basis of the second acceleration.

Description

Technical Field

[0001] The present application relates to the field of automated production technology, and in particular to a robot arm control method, device, electronic device and computer-readable storage medium. Background Art

[0002] Automated production refers to a production mode that uses automation technology and equipment to automatically complete production links during the production process. Among them, robotic arms, as key equipment for automated production, are widely used in industrial manufacturing, medical care, services, logistics and other fields. For example, when performing robotic arm shaft hole assembly tasks, multiple contact points and physical interactions are usually involved. In actual operations, due to the complexity of assembly tasks, robots must be able to handle dynamic external interference, errors and disturbances, as well as uncertainties. Among related technologies, motion planning-based methods have limitations when dealing with complex contact tasks, and it is difficult to adjust operating strategies in real time. Summary of the invention

[0003] The embodiments of the present application provide a robot arm control method, device, electronic device and computer-readable storage medium, which can achieve precise control and real-time adjustment of the robot arm movement through the combination of spatiotemporal reasoning and visual information feedback.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a method for controlling a robotic arm, the method comprising:

[0006] Acquire simulation data of the robot arm simulation target task, wherein the target task is used to control the end of the robot arm to move to a target point;

[0007] Acquire the first pose information of the manipulator at the time t-1 of the target task, and perform spatiotemporal reasoning on the first pose information based on the simulation data to obtain the second pose information of the manipulator at the time t of the target task;

[0008] Capturing the image of the target point at the time t by a camera carried by the end of the robotic arm;

[0009] Based on the image at the tth moment, an error prediction is performed on the speed of the end of the robotic arm to obtain a speed error of the end of the robotic arm at the tth moment, and a simulated speed of the end of the robotic arm is corrected based on the speed error to obtain an estimated speed of the end of the robotic arm at the tth moment, wherein the simulated speed is obtained by performing a derivative operation based on the second posture information;

[0010] Based on the estimated speed, perform admittance inference on the first acceleration of the robotic arm at the time t-1 to obtain the second acceleration of the robotic arm at the time t;

[0011] Based on the second acceleration, the robotic arm is driven to move.

[0012] The present application provides a robot arm control device, the device comprising:

[0013] A data acquisition module, used to obtain simulation data of the robot arm's simulation target task, wherein the target task is used to control the end of the robot arm to move to a target point;

[0014] A posture reasoning module is used to obtain the first posture information of the manipulator at the t-1th time of the target task, and perform spatiotemporal reasoning on the first posture information based on the simulation data to obtain the second posture information of the manipulator at the tth time of the target task;

[0015] A visual servo module, used for collecting the image of the target point at the tth moment through a camera carried by the end of the robotic arm;

[0016] The visual servo module is further used to perform an error prediction on the speed of the end of the robotic arm based on the image at the tth moment, obtain the speed error of the end of the robotic arm at the tth moment, and correct the simulated speed of the end of the robotic arm based on the speed error to obtain the estimated speed of the end of the robotic arm at the tth moment, wherein the simulated speed is obtained by performing a derivative operation based on the second posture information;

[0017] A robot driving module, configured to perform admittance inference on a first acceleration of the robot at the time t-1 based on the estimated speed, so as to obtain a second acceleration of the robot at the time t;

[0018] The robotic arm driving module is further used to drive the robotic arm to move based on the second acceleration.

[0019] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0020] A memory for storing computer executable instructions or computer programs;

[0021] The processor is used to implement the robot arm control method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0022] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the robotic arm control method provided in the embodiment of the present application when executed by a processor.

[0023] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the robot arm control method provided in the embodiment of the present application is implemented.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] Based on the simulation data, the first pose information of the robot arm at the t-1th moment is temporally and spatially inferred to realize the prediction of the pose information at the adjacent tth moment, making the motion control more forward-looking and continuous. At the same time, the camera at the end of the robot arm is used to collect the target point image to predict the speed error and correct the simulation speed to improve the motion accuracy and reliability. Finally, the acceleration of the robot arm motion is adjusted according to the corrected estimated speed to achieve precise control, efficiently and accurately complete the target task, and improve the overall performance of the robot arm and the quality of completion of the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a structural diagram of the robotic arm control system architecture provided by an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of the structure of an electronic device for controlling a robotic arm provided in an embodiment of the present application;

[0028] Figure 3 is a first flow chart of the robot arm control method provided in an embodiment of the present application;

[0029] Figure 4 is a second flow chart of the robot arm control method provided in an embodiment of the present application;

[0030] Figure 5 is a third flow chart of the robot arm control method provided in an embodiment of the present application;

[0031] Figure 6 is a fourth flow chart of the robot arm control method provided in an embodiment of the present application;

[0032] Figure 7 This is the fifth flow chart of the robot arm control method provided in the embodiment of the present application.

[0033] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0039] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) Robotic arm: A robotic arm is a mechanical device that can simulate the movements of human arms. It has the characteristics of high precision, high speed and high flexibility. The types of robotic arms include single robotic arms and dual robotic arms. Among them, the common single robotic arm is usually composed of a base, an arm and multiple joints. The arm is connected to the base through the joint, and the end is connected to the end effector. The dual robotic arm is composed of two bases, two arms and multiple joints. The two arms are respectively connected to their respective bases, and each end is connected to an end effector.

[0042] 2) Posture information: refers to the parameters that describe the position information and posture of an object in space. Position information refers to the specific position of an object in space, which can be represented by Cartesian coordinates. Posture information refers to the direction and orientation of an object, which can be described by a 3x3 rotation matrix to describe the rotation of an object relative to a reference coordinate system.

[0043] 3) Jacobian Matrix: It is a core concept in robotics and manipulator control. It describes the relationship between the velocity of the end effector of the manipulator and the velocity of each joint. Each element of the Jacobian matrix is ​​the partial derivative of the end effector velocity with respect to the velocity of each joint.

[0044] The embodiments of the present application provide a robot arm control method, device, electronic device, computer-readable storage medium and computer program product, which can achieve precise control and real-time adjustment of the robot arm movement through the combination of spatiotemporal reasoning and visual information feedback.

[0045] The robot arm control method provided in the embodiment of the present application can be implemented by a terminal or a server alone; it can also be implemented by a terminal and a server in collaboration. For example, the terminal alone undertakes the robot arm control method described below, or the terminal sends a robot arm control request to the server, and the server parses the received robot arm control request to obtain the simulation data carried by the robot arm control request, the image at the tth moment and the first posture information, so as to execute the robot arm control method.

[0046] The electronic device for controlling the robotic arm provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0047] See also Figure 1 , Figure 1It is a structural diagram of the robotic arm control system architecture provided in the embodiment of the present application. In the robotic arm control system 10 provided in the embodiment of the present application, in order to support a robotic arm control application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0048] Terminal 400 can be used to obtain a robot control request, which carries simulation data of the robot simulating a target task, first position information of the robot at the t-1th moment of the target task, and an image of the target point at the tth moment.

[0049] In some embodiments, a robotic arm control plug-in may be implanted in the client running in the terminal 400 to implement a robotic arm control method locally on the client. For example, the terminal 400 calls the robotic arm control plug-in to implement the robotic arm control method, parses the robotic arm control request to obtain simulation data of the robotic arm simulating the target task, the first posture information of the robotic arm at the t-1th moment of the target task, and the image of the target point at the tth moment. Based on the simulation data, the first posture information is temporally and spatially inferred to obtain the second posture information of the robotic arm at the tth moment of the target task; the speed of the end of the robotic arm is predicted through the image of the target point at the tth moment to obtain the speed error of the end of the robotic arm at the tth moment, and the simulated speed of the end of the robotic arm is corrected based on the speed error to obtain the estimated speed of the end of the robotic arm at the tth moment; based on the estimated speed, the first acceleration of the robotic arm at the t-1th moment is adjusted to obtain the second acceleration of the robotic arm at the tth moment; based on the second acceleration, the robotic arm is driven to move.

[0050] In some embodiments, after the terminal 400 obtains the robot arm control request, the robot arm control interface of the server 200 is called (which can be provided in the form of a cloud service). The server 200 implements the robot arm control method through the robot arm control plug-in, parses the robot arm control request, and obtains the simulation data of the robot arm simulating the target task, the first posture information of the robot arm at the t-1th time of the target task, and the image of the target point at the tth time. Based on the simulation data, the first posture information is temporally and spatially reasoned to obtain the second posture information of the robot arm at the tth time of the target task; the speed of the end of the robot arm is predicted through the image of the target point at the tth time to obtain the speed error of the end of the robot arm at the tth time, and the simulated speed of the end of the robot arm is corrected based on the speed error to obtain the estimated speed of the end of the robot arm at the tth time; based on the estimated speed, the first acceleration of the robot arm at the t-1th time is adjusted to obtain the second acceleration of the robot arm at the tth time, and the second acceleration is returned to the terminal 400, and the terminal 400 drives the robot arm to move based on the second acceleration.

[0051] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for controlling a robotic arm provided in an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1 The terminal 400 or the server 200 in the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .

[0052] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0053] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0054] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard disk drive, an optical disk drive, etc. The memory 550 may optionally include one or more storage devices that are physically located away from the processor 510.

[0055] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0056] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0057] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0058] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0059] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0060] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A robotic arm control device 555 stored in a memory 550 is shown, which may be software in the form of programs and plug-ins, including the following software modules: a data acquisition module 5551, a posture reasoning module 5552, a visual servo module 5553 and a robotic arm drive module 5554. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0061] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the robot arm control method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0062] As mentioned above, the electronic device for implementing the robot arm control method of the embodiment of the present application can be a terminal, a server, or a combination of the two, so the execution subject of each step will not be repeated below, see Figure 3 , Figure 3 is a first flow chart of the robot arm control method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0063] In step 101, simulation data of a robot arm simulating a target task is obtained.

[0064] Among them, the target task is used to control the end of the robotic arm to move to the target point. The robotic arm can be a single robotic arm or a dual robotic arm, which is not limited here. The simulation data can include the posture information, force and torque information of the robotic arm during the simulation of executing the target task.

[0065] As an example, a target task is simulated and executed through a remote operation device, and at least one piece of simulation data of the robot arm in the process of simulating the execution of the target task is collected, wherein the remote operation device can be a space mouse (Space-Mouse). For example, in the application scenario of performing a dual-robot arm shaft-hole assembly task, the target task is an assembly task, and the target point is the geometric center point of the object to be assembled. The operator uses the space mouse to simulate the execution of the dual-robot arm shaft-hole assembly task at least once to obtain at least one piece of simulation data of the robot arm simulating the execution of the assembly task.

[0066] The time information of the simulation data is normalized to obtain the normalized time information, i.e., the task phase h(t). Then, the simulation data of the task phase is decomposed by basis functions to approximately decompose the posture information Y at time t in the simulation data into Among them, φ represents the linear combination of basis functions, which is used to describe the basic shape of the motion, W is the basis function weight, which is used to describe the spatial distribution of the task, e represents the approximation error, and h(t) represents the phase of the task after normalization.

[0067] When E simulation data are collected, the pose information of the jth simulation data at any time can be represented as 1≤j≤E, where h represents the task phase, i.e., the completion progress of the target task. represents the phase speed, which defines the progress speed of the target task, w j The basis function weights representing the spatial dimension are used to describe the spatial variation of the target task.

[0068] In the embodiment of the present application, by collecting at least one piece of simulation data of the robotic arm in the process of simulating the execution of the target task, the motion trajectory and force feedback of the robotic arm are recorded, providing a basis for the imitation learning in the subsequent steps.

[0069] In step 102, the first posture information of the robot arm at the t-1th time of the target task is obtained, and based on the simulation data, the first posture information is subjected to spatiotemporal reasoning to obtain the second posture information of the robot arm at the tth time of the target task.

[0070] As an example, the multimodal data sensor mounted on the robot arm is used to obtain the multimodal data of the robot arm at the target task at the t-1th time, wherein the multimodal data at the t-1th time includes the posture information, force and torque information of the robot arm. Referring to the basis function decomposition method shown in step 101, the multimodal data at the t-1th time is fitted as X t-1|t-1 , Among them, h t-1 is the completion progress of the target task at time t-1, is the progress speed of the target task, and w is the basis function weight.

[0071] Based on the simulation data, the first pose information is subjected to spatiotemporal reasoning to obtain the second pose information of the robot at the tth moment of the target task, wherein the first pose information is the true pose information of the robot at the t-1th moment, and the second pose information is the pose information corresponding to the minimum motion cost of the robot at the tth moment.

[0072] In some embodiments, see Figure 4 , Figure 4 is a second flow chart of the robot arm control method provided in an embodiment of the present application, Figure 3 In step 102 shown, "based on the simulation data, perform spatiotemporal reasoning on the first pose information to obtain the second pose information of the robot arm at the tth moment of the target task" can be achieved by the following steps 1021 to 1023, which are described in detail below.

[0073] In step 1021, the initial posture information of the robot arm is obtained, and based on the initial posture information and the simulation data, the first posture information is subjected to spatiotemporal reasoning to obtain the posterior posture information of the robot arm at the tth moment.

[0074] As an example, the initial pose information of the robot arm can be represented as X 0 , Among them, the completion progress of the target task corresponding to the initial posture of the robot arm is 0. is the progress speed of the target task, and w is the basis function weight. Based on the initial pose information X 0and simulation data, perform time reasoning on the first pose information to obtain the prior pose information of the robot at the tth moment, perform spatial reasoning on the first pose information to obtain the prior pose information of the robot at the tth moment, perform time reasoning on the prior pose information to obtain the posterior pose information of the robot at the tth moment, wherein the prior pose information refers to the pose information obtained by prediction based only on engineering experience or historical data (for example, pose transformation matrix) in the absence of given simulation data, and the posterior pose information refers to the pose information obtained by prediction based on the prior pose information and the simulation data when the simulation data is given.

[0075] In some embodiments, see Figure 5 , Figure 5 is a third flow chart of the robot arm control method provided in an embodiment of the present application, Figure 4 The illustrated step 1021 can be implemented by following the steps 201 to 202, which are described in detail below.

[0076] In step 201, a posture transformation matrix is ​​obtained, and based on the posture transformation matrix, spatial reasoning is performed on the first posture information to obtain the prior posture information of the robot arm at the tth moment.

[0077] Among them, the posture transformation matrix is ​​used to describe the posture change of a uniformly moving robotic arm between two adjacent moments.

[0078] As an example, based on the posture change between two adjacent moments before the t-1th moment of the robot arm, the posture transformation matrix is ​​determined. Based on the posture transformation matrix, spatial reasoning is performed on the first posture information to obtain the prior posture information of the robot arm at the tth moment. The spatial reasoning formula (1.1) for the first posture information is as follows:

[0079] X t|t-1 =GX t-1|t-1 +η(0,Q t ) (1.1)

[0080] Among them, X t|t-1 is the prior position information of the robot arm at the tth moment, X t-1|t-1 is the first pose information, G is the pose transformation matrix, η(0,Q t ) is the process noise, which describes the random interference in the motion of the robot, such as external disturbances, sensor noise, and actuator errors. The process noise can be estimated using covariance.

[0081] In step 202, based on the initial posture and simulation data, time reasoning is performed on the prior posture information to obtain the posterior distribution of the posture information of the robot arm at the tth moment, and the mean of the posterior distribution is determined as the posterior posture information.

[0082] The posterior distribution represents the probability distribution of the posture information of the robot at the tth moment, given the simulation data at and before the tth moment and the initial posture information of the robot.

[0083] As an example, the observed distribution probability of the simulated data at the tth moment is determined given the prior pose information of the robot at the tth moment, and the predicted prior distribution probability of the prior pose information of the robot at the tth moment is determined given the simulation data at the t-1th moment and before the t-1th moment and the initial pose information of the robot. Temporal reasoning is performed based on the observed distribution probability and the predicted prior distribution probability to obtain the posterior distribution of the pose information of the robot at the tth moment. The posterior distribution of the pose information of the robot at the tth moment can be a Gaussian distribution. The mean of the posterior distribution is determined as the posterior pose information of the robot at the tth moment. The posterior pose information can be represented as The formula (1.2) for temporal reasoning based on observed distribution probability and predicted prior distribution probability is as follows:

[0084] p(X t|t-1 |Y 1:t ,X 0 )∝p(Y t |X t|t-1 )·p(X t|t-1 |Y 1:t-1 ,X 0 ) (1.2)

[0085] Among them, p(X t|t-1 |Y 1:t ,X 0 ) is the posterior distribution probability of the robot arm’s posture information at the tth moment, Y 1:t is the simulated data at and before time t, X 0 is the initial position information of the robot arm, X t|t-1 is the prior position information of the robot arm at the tth moment, p(Y t |X t|t-1 ) is the observed distribution probability, Y t The simulated data at time t, p(X t|t-1 |Y 1:t-1 ,X 0 ) is the predicted prior distribution probability, Y 1:t-1 is all the simulation data at and before the t-1th moment.

[0086] In an embodiment of the present application, spatial reasoning is performed on the first pose information through the pose transformation matrix to obtain the prior pose information of the robot arm at the tth moment, and then temporal reasoning is performed based on the initial pose and simulation data to obtain the posterior distribution of the pose information of the robot arm at the tth moment, and the mean of the posterior distribution is determined as the posterior pose information, thereby effectively integrating spatial and temporal information, improving the accuracy and reliability of pose estimation, and being able to better cope with random influences such as external disturbances, sensor noise, and actuator errors.

[0087] In step 1022, based on the posterior pose information, the cost of the simulation data is estimated to obtain the motion cost of the robot arm at the tth moment.

[0088] Among them, there is a transformation relationship between the motion cost of the robot arm at the t moment and the posture information of the robot arm at the t moment.

[0089] In some embodiments, Figure 4 Step 1022 shown can be implemented by the following steps: obtaining the minimum motion cost of the robot arm at the t-1th moment, and determining the product of the gradient value of the posterior pose information, the minimum motion cost at the t-1th moment and the adjustment coefficient, integrating the product to obtain the first weight of the pose information of the robot arm at the tth moment; determining the difference between the posterior pose information and the mean of the simulation data as the second weight of the derivative of the pose information of the robot arm at the tth moment; based on the first weight and the second weight, performing weighted summation on the pose information of the robot arm at the tth moment and the derivative of the pose information of the robot arm at the tth moment to obtain the motion cost of the robot arm at the tth moment.

[0090] As an example, based on formula (1.3), the gradient value of the posterior pose information, the product of the minimum motion cost at time t-1 and the adjustment coefficient are determined as follows:

[0091]

[0092] in, is the posterior pose information, is the gradient value of the posterior pose information, is the minimum motion cost at the t-1th moment, γ is the first adjustment coefficient, is the product of the gradient value of the posterior pose information, the minimum motion cost at the t-1th moment, and the adjustment coefficient. At the same time, Also the first weight Q 1 The first derivative of the product Perform an integral operation to obtain the first weight Q of the robot's posture information at time t 1 .

[0093] When E pieces of simulation data are collected, the mean of the simulation data is the mean of the posture information of the simulation data at the tth moment. Based on the difference between the posterior posture information and the mean of the simulation data, the formula (1.4) for determining the second weight of the derivative of the posture information of the manipulator at the tth moment is as follows:

[0094]

[0095] Among them, Q 2 is the second weight, is the first-order derivative of the posterior pose information, is the mean of the simulated data, E is the number of simulated data, It is the first-order derivative of the pose information of the j-th simulation data at the t-th time.

[0096] Based on the first weight Q 1 and the second weight Q 2 , the position information X of the robot arm at the tth moment t|t The derivative of the position information of the robot at the tth moment Perform weighted summation to obtain the motion cost of the robot at time t The motion cost of the robot at time t The determination formula (1.5) is as follows:

[0097]

[0098] in, is the motion cost of the robot at time t, X t|t is the position information of the robot arm at the tth moment, The derivative of the robot's posture information at time t, Q 1 is the first weight, Q 2 is the second weight, γ is the first adjustment coefficient, and α is the second adjustment coefficient.

[0099] In an embodiment of the present application, by comprehensively considering multiple factors such as the gradient of the posterior pose information, the minimum motion cost, and the mean of the simulation data, the first weight and the second weight of the pose information of the robot at the tth moment are accurately calculated, and then using these two weights, the pose information of the robot at the tth moment and its derivative are weighted summed to achieve a comprehensive and accurate evaluation of the motion cost of the robot.

[0100] In step 1023, in the transformation relationship, the posture information corresponding to the minimum motion cost at the t-th moment is determined as the second posture information.

[0101] As an example, based on the transformation relationship between the motion cost of the robot arm at the t moment and the posture information of the robot arm at the t moment, the posture information corresponding to the minimum motion cost at the t moment in the transformation relationship is determined as the second posture information.

[0102] In an embodiment of the present application, by using the posture information corresponding to the minimum motion cost at the tth moment as the second posture information, the transformation relationship between the motion cost and the posture information is effectively utilized, so that the robotic arm can accurately select the posture with the minimum motion cost as the target posture from many possible postures, thereby reducing the energy consumption of the robotic arm movement.

[0103] In step 103, an image of the target point at time t is captured by a camera carried by the end of the robotic arm.

[0104] As an example, a camera is mounted at the end of the robotic arm, the camera may be a depth camera, and the image may be a depth image of the target point at the tth moment. For example, in an application scenario where a dual-robotic-arm shaft-hole assembly task is performed, the target point is the geometric center point of the object to be assembled, and the image may be a depth image of the object to be assembled.

[0105] In step 104, based on the image at the tth moment, the speed of the end of the robot arm is predicted with error to obtain the speed error of the end of the robot arm at the tth moment, and the simulated speed of the end of the robot arm is corrected based on the speed error to obtain the estimated speed of the end of the robot arm at the tth moment.

[0106] The simulation speed is obtained by performing derivative calculation based on the second posture information.

[0107] As an example, the derivative operation on the second posture information may be a first-order derivative operation on the second posture information. The simulation speed may be characterized as the first-order derivative of the second posture information. By using formula (1.6), the simulation speed of the end of the manipulator is corrected based on the speed error to determine the estimated speed of the end of the manipulator at the tth moment. Formula (1.6) is as follows:

[0108]

[0109] in, is the estimated speed of the end of the robot at the tth moment, X t|t is the second posture information of the end of the robot at time t, is the simulated velocity of the end of the robot at the tth moment, Speed ​​error, λ is the third adjustment coefficient.

[0110] In some embodiments, see Figure 6 , Figure 6 is a fourth flow chart of the robot arm control method provided in an embodiment of the present application, Figure 3In step 104 shown, "based on the image at the tth moment, the error prediction of the speed of the end of the robot arm is performed to obtain the speed error of the end of the robot arm at the tth moment" can be achieved by the following steps 1041 to 1043, which are explained in detail below.

[0111] In step 1041 , the difference between the two-dimensional coordinates of the target point in the image at time t and the expected two-dimensional coordinates is determined as the image error.

[0112] As an example, get the camera intrinsic parameter matrix K, Among them, f x and f y is the focal length of the camera, c x and c y is the coordinate of the principal point of the camera. At the same time, from the image captured by the camera, determine the three-dimensional coordinate S of the target point in the camera's coordinate system, S = [x, y, z] T , project the three-dimensional coordinates of the target point to obtain the two-dimensional coordinates p of the target point in the camera projection coordinate system d , p d =[u,v] T ,in, The simulation data includes the simulated two-dimensional coordinates of the target point of the robot arm during the simulation of executing the target task. The simulated two-dimensional coordinates of the target point in the simulation data at the tth moment are determined as the expected two-dimensional coordinates. The expected two-dimensional coordinates can be represented as p c , the two-dimensional coordinates of the target point p d and the desired two-dimensional coordinate p c The difference dp between them is determined as the image error, and the image error can be represented by e.

[0113] In step 1042, the instantaneous velocity of the end of the robot arm is obtained, and a first transformation matrix between the image error and the instantaneous velocity of the end of the robot arm is determined.

[0114] As an example, the instantaneous speed of the end of the robotic arm is obtained through the multimodal data sensor mounted on the robotic arm. The instantaneous speed can be represented as v. The relationship between the image error e and the instantaneous speed v of the end of the robotic arm can be represented as e=J e v, where J e is a first transformation matrix, which is a Jacobian matrix used to describe the transformation relationship between the image error and the velocity of the end of the robot arm.

[0115] In some embodiments, see Figure 7 , Figure 7 is a fifth flow chart of the robot arm control method provided in an embodiment of the present application, Figure 6The shown step 1042 can also be implemented through the following steps 301 to 302, which are specifically described below.

[0116] In step 301, the projection relationship between the two-dimensional coordinates and the three-dimensional coordinates of the target point is determined as the second transformation matrix between the image error and the instantaneous change amount of the three-dimensional coordinates.

[0117] Among them, the three-dimensional coordinates of the target point are the three-dimensional coordinates of the target point in the coordinate system of the camera. The simulation data includes the simulated three-dimensional coordinates of the target point during the simulation of the manipulator executing the target task. The instantaneous change amount of the three-dimensional coordinates is the difference between the three-dimensional coordinates of the target point and the simulated three-dimensional coordinates. The instantaneous change amount of the three-dimensional coordinates can be characterized as dS.

[0118] As an example, the difference dp between the two-dimensional coordinates of the target point and the expected two-dimensional coordinates is determined as the image error e. The transformation relationship among the difference dp between the two-dimensional coordinates of the target point and the expected two-dimensional coordinates, the image error e, and the instantaneous change amount dS of the three-dimensional coordinates can be characterized as e = dp = J p dS, where J p is the second transformation matrix. The second transformation matrix is the Jacobian matrix describing the projection relationship between the two-dimensional coordinates of the target point and the three-dimensional coordinates of the target point. The projection relationship is obtained by performing partial derivative operations on the three-dimensional coordinates of the target point based on the two-dimensional coordinates of the target point. The projection relationship is: The second transformation matrix J p can be characterized as

[0119] In step 302, the third transformation matrix between the instantaneous velocity of the end of the manipulator and the instantaneous change amount of the three-dimensional coordinates is obtained, and the third transformation matrix is fused into the second transformation matrix to obtain the first transformation matrix.

[0120] As an example, the transformation relationship between the instantaneous velocity v of the end of the manipulator and the instantaneous change amount dS of the three-dimensional coordinates can be characterized as dS = J x v, where J x is the third transformation matrix. The third transformation matrix is the Jacobian matrix describing the transformation relationship between the instantaneous change amount of the three-dimensional coordinates and the instantaneous velocity of the end of the manipulator.

[0121] For the transformation relationship e = J p dS between the image error e and the instantaneous change amount dS of the three-dimensional coordinates, and the transformation relationship dS = J x v between the instantaneous velocity v of the end of the manipulator and the instantaneous change amount dS of the three-dimensional coordinates, a chain derivation is performed to obtain the change relationship e = J p J x v between the image error e and the instantaneous velocity v of the end of the manipulator, where Jp is the second transformation matrix, J x is the third transformation matrix, the first transformation matrix J between the image error and the instantaneous velocity of the end of the robot arm e It can be represented as J e =J p J x , that is, by fusing the third transformation matrix into the second transformation matrix, the first transformation matrix can be obtained.

[0122] In an embodiment of the present application, by establishing a second transformation matrix between the image error and the instantaneous change in the three-dimensional coordinates of the target point, and a third transformation matrix between the instantaneous velocity of the end of the robot arm and the instantaneous change in the three-dimensional coordinates, an accurate mapping from image error to the velocity of the end of the robot arm is achieved.

[0123] In step 1043, based on the first transformation matrix, the image error is subjected to speed transformation processing to obtain a speed error.

[0124] As an example, the pseudo-inverse matrix of the first transformation matrix is ​​determined, and based on formula (1.7), the product of the third adjustment coefficient, the pseudo-inverse matrix of the first transformation matrix and the image error is determined as the speed error. Formula (1.7) is as follows:

[0125]

[0126] in, is the speed error, is the pseudo inverse matrix of the first transformation matrix, e is the image error, and λ is the third adjustment coefficient.

[0127] In an embodiment of the present application, by accurately calculating the image error between the two-dimensional coordinates of the target point in the image and the expected two-dimensional coordinates, and using the first transformation matrix to associate the image error with the instantaneous speed of the end of the robotic arm, the robotic arm can adjust its movement speed in real time according to the image error, thereby improving the accuracy and adaptability of the operation, ensuring that the target position can be accurately reached when performing tasks, and enhancing the robotic arm's operational capabilities and stability in complex environments.

[0128] In step 105, based on the estimated speed, the first acceleration of the robot arm at time t-1 is inferred by admittance to obtain the second acceleration of the robot arm at time t.

[0129] As an example, the multimodal data sensor mounted on the robot arm is used to determine the posture information of the robot arm at the time t-1. The posture information of the robot arm at the time t-1 can be represented as X act , perform the second-order derivative operation on the position information of the robot at the time t-1, and obtain the first acceleration of the robot at the time t-1 Based on estimated speed The first acceleration of the robot arm at time t-1 Perform admittance inference to obtain the second acceleration of the robot at the tth moment.

[0130] In some embodiments, Figure 3 The shown step 105 can also be implemented by the following steps: integrating the estimated speed to obtain estimated posture information; determining a first difference between the estimated speed and the speed of the robot at the t-1th moment, and determining a second difference between the estimated posture information and the posture information of the robot at the t-1th moment; based on the damping matrix and stiffness matrix of the robot, weightedly summing the first difference and the second difference to obtain the acceleration error of the robot; based on the acceleration error, correcting the first acceleration to obtain the second acceleration.

[0131] As an example, for estimating the speed Perform integral operation to obtain the estimated pose information, which can be represented as The multimodal data sensor mounted on the robot arm is used to determine the posture information of the robot arm at the time t-1. The posture information of the robot arm at the time t-1 can be represented as X act , perform the first-order derivative operation on the position information of the robot arm at the t-1th time, and obtain the speed of the robot arm at the t-1th time Estimated speed and the speed of the robot at time t-1 The first difference between Estimated pose information and the position information X of the robot arm at the t-1th moment act The second difference between

[0132] Through the admittance inference formula (1.8), based on the damping matrix and stiffness matrix of the manipulator, the first difference and the second difference are weighted summed to obtain the acceleration error of the manipulator, and based on the acceleration error, the first acceleration is corrected to obtain the second acceleration. Formula (1.8) is as follows:

[0133]

[0134] in, is the second acceleration, is the first acceleration, is the first difference, is the second difference, M is the mass matrix, D is the damping matrix, K is the stiffness matrix, and f ext For external force.

[0135] In step 106 , the robot arm is driven to move based on the second acceleration.

[0136] As an example, the second acceleration is sent to the servo motors of the joints of the robotic arm, and the position control information is generated by the controller of the servo motor to drive the robotic arm to move.

[0137] Below, an exemplary application of the embodiment of the present application in an actual application scenario of a robot performing a dual-arm shaft hole assembly task will be described.

[0138] When robots perform dual-arm shaft-hole assembly tasks, multiple contact points and physical interactions are usually involved. In actual operation, due to the complexity of assembly tasks, robots must be able to handle dynamic external interference, errors and disturbances, as well as uncertainties. However, the motion planning-based methods in related technologies have limitations in dealing with complex contact tasks, especially when forces and environmental factors change, and it is difficult to adjust the operation strategy in real time.

[0139] In order to solve the above problems, the embodiment of the present application proposes a method for controlling a robotic arm, which learns the operation strategy in the robot assembly task through spatiotemporal reasoning, and the image feedback provided by the vision-guided Bayesian interactive primitive module and the force feedback collected by the sensor in real time, so that the robot can perform precise alignment and real-time adjustment when performing the assembly task, which enables the robot to have high precision and adaptability in complex environments and multi-point contact situations. The impedance controller is designed to make the robot respond compliantly to external force feedback, and the visual-guided Bayesian interactive home module is combined to achieve dual optimization of force and position information during the assembly process, especially in the case of multi-point contact and uneven force distribution, to ensure the flexibility, safety and assembly success rate of the robot's action.

[0140] The embodiment of the present application proposes a robotic arm control method which mainly includes four parts: data acquisition, model building, state inference and control drive, which will be explained one by one below.

[0141] 1. Data collection.

[0142] The operator obtains the teaching data of the assembly task (equivalent to the simulation data mentioned above) by using a remote operation device (e.g., SpaceMouse), including the position, posture, force and torque information of the robot in the assembly task. During the data collection process, the robot's motion trajectory and force feedback are recorded for subsequent imitation learning.

[0143] 2. Build a model.

[0144] Based on the en-Bayesian Interaction Primitives (enBIP) algorithm, the robot learns the operation strategy by performing spatiotemporal inference on multimodal teaching data, inferring the various stages of the assembly task based on the teaching data, and generating appropriate control signals. During the learning process, special attention is paid to timing control to ensure that the robot can identify external interference and make self-adjustments during task execution.

[0145] By converting the teaching data into a time-invariant representation, that is, using basis function decomposition technology, the teaching data is converted into task phase (a measure of relative time) rather than absolute time measurement, so that the model can represent teaching data of different lengths.

[0146] Any state of the teaching data (equivalent to the posture information above) can be approximately decomposed into: Among them, φ represents the linear combination of basis functions, which is used to describe the basic shape of the motion, W is the basis function weight to be learned, which is used to describe the spatial distribution of the task, e represents the approximation error, and h(t) represents the phase or time of the task after normalization.

[0147] In order to combine temporal and spatial reasoning, the state Y of the j-th teaching data (equivalent to the j-th trajectory above) is j The dimensions are designed to be: Among them, h represents the phase of the task, that is, the time progress of the current task, represents the phase velocity, which defines the time progress speed of the task, w j The basis function weights representing the spatial dimension are used to describe the spatial variation of the task.

[0148] 3. State inference.

[0149] 1. Bayesian interaction primitives and cost solver

[0150] The embodiment of the present application combines the Bayesian interaction primitive with the cost solver, and recursively updates the spatiotemporal joint reasoning through a recursive Bayesian update process. The robot dynamically adjusts its operating strategy according to the real-time sensor data and the spatiotemporal state of the current task.

[0151] The initial state of the robot (equivalent to the initial position information of the robot arm above) is defined as in, represents the phase velocity, which defines the time progress rate of the task, Right now Normal distribution μ l Normal distribution The mean of Normal distribution The variance of , w represents the basis function weight of the spatial dimension, K is the number of Gaussian components, α k is the kth solution of the EM algorithm based on the Gaussian mixture model parameter estimation, μ k For normal distribution N(μ k ,∑ k ),∑ k Normal distribution N(μ k ,∑ k ) is the covariance matrix of .

[0152] For each subsequent time step t, before combining the current observation data, only the state X at time t-1 is used. t-1|t-1 , the prior state X of the robot at time t t|t-1 The recursive formula for prediction is shown in formula (1.1) above, where X t|t-1 is the prior state of the robot at time t (or predicted state, equivalent to the prior position information of the robot arm at time t above), X t-1|t-1 is the estimated state of the robot at time t-1 (equivalent to the first position signal above), G here uses the state transfer matrix of the constant speed assumption (equivalent to the position transformation matrix above), η(0,Q t ) is the process noise. Process noise describes the random effects that are not modeled in the model. It is usually a combination of external disturbances, sensor noise, actuator errors and other factors. The process noise matrix can be estimated by conducting multiple experiments on the system, recording the difference between the output and the prediction, and using the covariance.

[0153] Through the constructed state vector X, the spatiotemporal reasoning formula (see formula 1.2 shown above) can jointly estimate the time and space states of the task and dynamically update the control signal of the robot. In the Bayesian interaction primitive framework, the probability formula is used to describe the process of spatiotemporal reasoning based on multimodal data (such as force, position, speed, etc.) during the execution of the task. The spatiotemporal reasoning formula reflects the inference of the current state of the robot based on current and past observations, which is divided into three main parts: the posterior probability p(X) t|t-1 |Y 1:t ,X 0 ) means that given all previous observations Y 1:t and the initial state X 0 Under the condition of t|t-1 The probability distribution of (equivalent to the posterior distribution of the position information of the robot arm at the tth moment above); the observation probability p(Y t |X t|t-1 ) means that given the current state X t|t-1In the case of, the observed value Y t The probability, which follows a Gaussian distribution and is mainly obtained from the linear expression of the state dimension solved by the teaching data (equivalent to the observed distribution probability in the above text); the prediction prior p(X t|t-1 |Y 1:t-1 ,X 0 ) follows a Gaussian distribution, indicating the prior probability of the current state X 1:t-1 given all the observed data X 0 at the previous moment t - 1 and the initial state X t|t-1 . This probability is obtained from the state recurrence formula at the previous moment (equivalent to the prediction prior distribution probability in the above text).

[0154] According to the spatio - temporal inference formula, determine the mean of the normal distribution that p(X t|t-1}Y 1:t ,X 0 ) follows, and determine the mean as the posterior state at the t - th moment (equivalent to the posterior pose information in the above text).

[0155] In the process of recursive update, the embodiment of the present application constructs a state priority solver based on the formula (1.5) shown above, and solves the optimal variable value X corresponding to the minimum cost value at the moment t by minimizing the cost value t|t (equivalent to the second pose information in the above text), where α and γ are adjustment coefficients, Q 1 is the weight coefficient of the state X t|t (equivalent to the first weight in the above text), Q 2 is the weight matrix related to speed (equivalent to the second weight in the above text). The calculation formula of the weight coefficient Q t|t of the state X 1 can be seen in the formula (1.3) shown above, and the calculation formula of the weight matrix Q 2 related to speed can be seen in the formula (1.4) shown above. In order to enable the control algorithm to adapt and adjust according to environmental changes, Q 1 needs to be dynamically adjusted by real - time learning, so the product of the gradient of X t|t-1 and the minimum cost at the moment t - 1 is introduced.

[0156] 2. Visual error calculation

[0157] To further optimize the accuracy of state inference, the embodiment of the present application introduces camera vision image information and establishes the following non - linear model of camera speed. The specific steps are as follows:

[0158] 1) Build a perspective projection model

[0159] Let the object point X in the camera coordinate system be [x, y, z] T (equivalent to the three-dimensional coordinates of the target point above) is projected onto the image plane by the camera, and the image coordinates p = [u, v] are obtained. T (equivalent to the two-dimensional coordinates of the target point above). The intrinsic parameter matrix of the camera can be expressed as: Among them, f x and f y is the focal length of the camera, c x and c y is the principal point coordinate of the camera.

[0160] For an object point X = [x, y, z] T , its projection in the camera coordinate system p = [u, v] T Satisfies the following equation:

[0161] 2) Establish a model of the relationship between motion and image coordinates

[0162] The change dX of the three-dimensional coordinates of the object will cause the change dp of the two-dimensional coordinates. The relationship between dX and dp can be obtained by differentiation. The image coordinate p = [u, v] T For the object position X = [x, y, z] T The partial derivative of is expressed as: The changes in the image coordinates depend on the position of the object in the camera coordinate system and the intrinsic parameters of the camera.

[0163] 3) Build a nonlinear model of visual following

[0164] The motion of the robot end effector (equivalent to the end of the robot arm mentioned above) can be expressed as a 6-degree-of-freedom velocity vector v = [v x ,v y ,v z ,ω x ,ω y ,ω z ] T , where [v x ,v y ,v z ] is the translation velocity, [ω x ,ω y ,ω z ] is the angular velocity. The change in the 3D position of the object dX is related to the velocity of the robot end effector according to the camera motion (rotation and translation of the camera).

[0165] Assuming that the movement of the robot end effector affects the position of the object through the camera's transformation matrix, the change in the object's spatial position X can be expressed as: dX = J x v, where J xis the Jacobian matrix of the object's motion relative to the robot's end effector (equivalent to the three transformation matrices above)

[0166] Derivative of image error with respect to object motion: dp = J p dX, where J p The Jacobian matrix of the image coordinate p to the object position X (equivalent to the second transformation matrix above),

[0167] According to the chain rule, the relationship between the image error e and the robot end effector velocity v can be expressed as: e = J e v, J e =J p J x , where J e It is the total Jacobian matrix (equivalent to the first transformation matrix above), combining the camera motion (given by the camera extrinsic and intrinsic parameters) with the object's spatial motion.

[0168] Referring to formula (1.7) shown above, the error e in the image space can be converted into the velocity command error of the robot end effector through the total Jacobian matrix: in, is the pseudo-inverse of the total Jacobian matrix, and λ is the gain parameter (equivalent to the third adjustment coefficient above).

[0169] 3) Posterior estimation state determination

[0170] Referring to formula (1.6) shown above, based on the optimal variable value X corresponding to the minimum cost value at the current moment t|t , combined with visual feedback information, the posterior estimation state at the current moment is obtained

[0171] The posterior state estimates the pose Substituting into the admittance inference controller shown in formula (1.8) above, we get the target acceleration Among them, X act represents the actual terminal state information at time t-1, obtained through the encoder, f ext Obtained through the end six-dimensional force sensor.

[0172] By combining the admittance inference controller with the visual Bayesian spatiotemporal inference module, the embodiment of the present application can adjust and infer the contact force in real time while maintaining visual guidance, thereby enabling the robot to dynamically adjust the operation strategy by integrating visual and force feedback information to achieve efficient and stable dual-arm assembly operations.

[0173] 4. Control drive.

[0174] The calculated target acceleration information is sent to the servo motors of each joint, and the position control information is generated through the motor controller (for example, PID controller), driving the double arms to move the clamped assembly parts to the assembly position. In the coarse positioning process, the visually guided Bayesian interaction primitives play a major role, facilitating the rapid movement of the robotic arm to the specified axis hole alignment position. In the fine positioning process, a force-controlled admittance inference controller is mainly designed to control the uniform distribution of contact force of assembly parts during the assembly process and optimize the assembly quality.

[0175] Compared with the related art, the robot arm control method provided in the embodiment of the present application has significant advantages and positive effects in the following aspects:

[0176] 1) Improving the accuracy and success rate of assembly tasks: Robot shaft-hole assembly in related technologies usually relies on predetermined trajectories or simple motion planning, and cannot cope with dynamic changes or external disturbances in tasks. The robotic arm control method provided in the embodiment of the present application combines vision-guided Bayesian interactive primitive control with contact force-based imitation learning, enabling the robot to make real-time adjustments during the assembly process, ensuring higher assembly accuracy and success rate, especially in multi-point contact and complex environments, the adaptability of assembly is significantly enhanced.

[0177] 2) Adaptive response to external interference: Related technologies are usually difficult to effectively deal with the force accumulation problem caused by assembly misalignment or external interference. The robot arm control method provided in the embodiment of the present application designs a visually guided Bayesian interaction primitive assisted admittance inference and force feedback mechanism, so that the robot can perceive and respond to changes in contact force in real time during the assembly process, avoiding assembly failures caused by excessive or unbalanced forces. At the same time, the imitation learning strategy is combined with the Bayesian interaction primitive (enBIP) algorithm, so that the system can learn the best operation mode to deal with various disturbances, ensuring the robustness of the assembly process.

[0178] 3) Inference optimization combining efficient imitation learning with visual force control: The robot arm control method provided in the embodiment of the present application utilizes the Bayesian Interaction Primitive (enBIP) algorithm and combines human demonstration data for imitation learning, and can learn assembly strategies from a small number of demonstrations. This learning method can not only handle uncertainties in dynamic environments, but also automatically generate optimal operating strategies for different working conditions. Unlike the assembly method of related technologies that relies on single sensor feedback, the robot arm control method provided in the embodiment of the present application combines the vision-guided Bayesian Interaction Primitive (IBVS) and force feedback. The vision-guided Bayesian Interaction Primitive module guides the robot arms to accurately align with the target position by acquiring feature point information in the image in real time, avoiding the assembly object from deviating from the field of view. Especially in dynamic environments or multi-point contact assembly tasks, the vision-guided Bayesian Interaction Primitive control greatly improves the stability and accuracy of the assembly. Compared with related technologies, the robot arm control method provided in the embodiment of the present application greatly reduces the dependence on complex motion planning, reduces the time for task setting and debugging, and does not require repeated manual adjustments.

[0179] 4) Improve the safety and flexibility of operation: The admittance inference controller assisted by the vision-guided Bayesian interactive primitives designed in the robotic arm control method provided in the embodiment of the present application enables the robot to perform assembly operations in a flexible manner, reducing equipment damage or assembly failures caused by rigid control. Compared with the rigid control method of the related technology, it can better control the force applied during the assembly process, prevent the contact surface from being subjected to excessive force, and improve the safety of the system and the service life of the equipment.

[0180] 5) Adapt to a variety of complex operating environments: The robotic arm control method provided in the embodiment of the present application can automatically adjust the control parameters according to the changes in the real-time environment. The system can maintain the optimal operating state in different working environments, so that the robot can continue to work stably in complex, uncertain or frequently disturbed environments, greatly expanding the application scenarios of robotic assembly technology.

[0181] The following is a description of an exemplary structure of a robot arm control device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the robot control device 555 of the memory 550 may include:

[0182] The data acquisition module 5551 is used to obtain simulation data of the robot arm's simulation target task, wherein the target task is used to control the end of the robot arm to move to a target point.

[0183] The posture reasoning module 5552 is used to obtain the first posture information of the robot arm at the t-1th time of the target task, and based on the simulation data, perform spatiotemporal reasoning on the first posture information to obtain the second posture information of the robot arm at the tth time of the target task.

[0184] The visual servo module 5553 is used to collect the image of the target point at the tth moment through the camera carried by the end of the robotic arm;

[0185] The visual servo module 5553 is also used to perform an error prediction on the speed of the end of the robotic arm based on the image at the tth moment, obtain the speed error of the end of the robotic arm at the tth moment, and correct the simulated speed of the end of the robotic arm based on the speed error to obtain the estimated speed of the end of the robotic arm at the tth moment, wherein the simulated speed is obtained by performing a derivative operation based on the second posture information.

[0186] The robot driving module 5554 is used to perform admittance inference on the first acceleration of the robot at the time t-1 based on the estimated speed, so as to obtain the second acceleration of the robot at the time t.

[0187] The robotic arm driving module 5554 is further used to drive the robotic arm to move based on the second acceleration.

[0188] In some embodiments, the posture inference module 5552 is also used to obtain the initial posture information of the robotic arm, and based on the initial posture information and the simulation data, perform spatiotemporal reasoning on the first posture information to obtain the posterior posture information of the robotic arm at the tth moment; based on the posterior posture information, perform cost estimation on the simulation data to obtain the motion cost of the robotic arm at the tth moment, wherein there is a transformation relationship between the motion cost of the robotic arm at the tth moment and the posture information of the robotic arm at the tth moment; in the transformation relationship, the posture information corresponding to the minimum motion cost at the tth moment is determined as the second posture information.

[0189] In some embodiments, the posture reasoning module 5552 is also used to obtain a posture transformation matrix, and based on the posture transformation matrix, perform spatial reasoning on the first posture information to obtain the prior posture information of the robotic arm at the tth moment, wherein the posture transformation matrix is ​​used to describe the posture change of the robotic arm in uniform motion between two adjacent moments; based on the initial posture and the simulation data, perform temporal reasoning on the prior posture information to obtain the posterior distribution of the posture information of the robotic arm at the tth moment, and determine the mean of the posterior distribution as the posterior posture information.

[0190] In some embodiments, the posture inference module 5552 is also used to obtain the minimum motion cost of the robotic arm at the t-1th moment, and determine the product of the gradient value of the posterior posture information, the minimum motion cost at the t-1th moment and the adjustment coefficient, integrate the product to obtain the first weight of the posture information of the robotic arm at the tth moment; determine the difference between the posterior posture information and the mean of the simulation data as the second weight of the derivative of the posture information of the robotic arm at the tth moment; based on the first weight and the second weight, perform weighted summation of the posture information of the robotic arm at the tth moment and the derivative of the posture information of the robotic arm at the tth moment to obtain the motion cost of the robotic arm at the tth moment.

[0191] In some embodiments, the visual servo module 5553 is also used to determine the difference between the two-dimensional coordinates of the target point in the image and the expected two-dimensional coordinates as an image error; obtain the instantaneous velocity of the end of the robotic arm, and determine a first transformation matrix between the image error and the instantaneous velocity of the end of the robotic arm; based on the first transformation matrix, perform speed transformation processing on the image error to obtain the speed error.

[0192] In some embodiments, the visual servo module 5553 is also used to determine the projection relationship between the two-dimensional coordinates and the three-dimensional coordinates of the target point as a second transformation matrix between the image error and the instantaneous change of the three-dimensional coordinates, wherein the three-dimensional coordinates of the target point are the three-dimensional coordinates of the target point in the coordinate system of the camera; obtain a third transformation matrix between the instantaneous velocity of the end of the robotic arm and the instantaneous change of the three-dimensional coordinates, and fuse the third transformation matrix into the second transformation matrix to obtain the first transformation matrix.

[0193] In some embodiments, the robotic arm driving module 5554 is also used to perform an integration operation on the estimated speed to obtain estimated posture information; determine a first difference between the estimated speed and the speed of the robotic arm at the t-1th moment, and determine a second difference between the estimated posture information and the posture information of the robotic arm at the t-1th moment; based on the damping matrix and stiffness matrix of the robotic arm, perform a weighted summation of the first difference and the second difference to obtain an acceleration error of the robotic arm; based on the acceleration error, correct the first acceleration to obtain the second acceleration.

[0194] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the above-mentioned robot arm control method of the embodiment of the present application.

[0195] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the robot arm control method provided by the present application embodiment, for example, Figure 3 The robot arm control method is shown.

[0196] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0197] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0198] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0199] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0200] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A robot arm control method, characterized in that: The method comprises: Acquire simulation data of the robot arm simulation target task, wherein the target task is used to control the end of the robot arm to move to a target point; Acquire the first pose information of the manipulator at the time t-1 of the target task, and perform spatiotemporal reasoning on the first pose information based on the simulation data to obtain the second pose information of the manipulator at the time t of the target task; Capturing the image of the target point at the time t by a camera carried by the end of the robotic arm; Based on the image at the tth moment, an error prediction is performed on the speed of the end of the robotic arm to obtain a speed error of the end of the robotic arm at the tth moment, and a simulated speed of the end of the robotic arm is corrected based on the speed error to obtain an estimated speed of the end of the robotic arm at the tth moment, wherein the simulated speed is obtained by performing a derivative operation based on the second posture information; Based on the estimated speed, perform admittance inference on the first acceleration of the robotic arm at the time t-1 to obtain the second acceleration of the robotic arm at the time t; Based on the second acceleration, the robotic arm is driven to move.

2. The method according to claim 1, characterized in that The performing spatiotemporal reasoning on the first posture information based on the simulation data to obtain the second posture information of the robot arm at the tth time of the target task includes: Acquire initial posture information of the robotic arm, and perform spatiotemporal reasoning on the first posture information based on the initial posture information and the simulation data to obtain posterior posture information of the robotic arm at the time t; Based on the posterior pose information, cost estimation is performed on the simulation data to obtain the motion cost of the robotic arm at the t-th moment, wherein there is a transformation relationship between the motion cost of the robotic arm at the t-th moment and the pose information of the robotic arm at the t-th moment; In the transformation relationship, the posture information corresponding to the minimum motion cost at the t-th moment is determined as the second posture information.

3. The method according to claim 2, characterized in that The performing spatiotemporal reasoning on the first posture information based on the initial posture information and the simulation data to obtain the posterior posture information of the robot arm at the time t includes: Acquire a posture transformation matrix, and perform spatial reasoning on the first posture information based on the posture transformation matrix to obtain the prior posture information of the manipulator at the tth moment, wherein the posture transformation matrix is ​​used to describe the posture change of the manipulator in uniform motion between two adjacent moments; Based on the initial posture and the simulation data, time reasoning is performed on the prior posture information to obtain the posterior distribution of the posture information of the robot arm at the tth moment, and the mean of the posterior distribution is determined as the posterior posture information.

4. The method according to claim 2, characterized in that: The cost estimation of the simulation data based on the posterior pose information to obtain the motion cost of the robot arm at the time t includes: Obtaining the minimum motion cost of the manipulator at the t-1th moment, and determining the product of the gradient value of the posterior pose information, the minimum motion cost at the t-1th moment, and the adjustment coefficient, integrating the product to obtain a first weight of the pose information of the manipulator at the tth moment; Determine the difference between the posterior pose information and the mean of the simulation data as a second weight of the derivative of the pose information of the robotic arm at the time t; Based on the first weight and the second weight, a weighted sum is performed on the posture information of the robotic arm at the tth moment and the derivative of the posture information of the robotic arm at the tth moment to obtain the motion cost of the robotic arm at the tth moment.

5. The method according to claim 1, characterized in that The error prediction of the speed of the end of the robotic arm based on the image at the time t to obtain the speed error of the end of the robotic arm at the time t includes: Determine the difference between the two-dimensional coordinates of the target point in the image at the time t and the expected two-dimensional coordinates as an image error; Acquire the instantaneous velocity of the end of the robotic arm, and determine a first transformation matrix between the image error and the instantaneous velocity of the end of the robotic arm; Based on the first transformation matrix, the image error is subjected to speed transformation processing to obtain the speed error.

6. The method according to claim 5, characterized in that The determining of a first transformation matrix between the image error and the instantaneous velocity of the end of the robotic arm comprises: Determine the projection relationship between the two-dimensional coordinates and the three-dimensional coordinates of the target point as a second transformation matrix between the image error and the instantaneous change of the three-dimensional coordinates, wherein the three-dimensional coordinates of the target point are the three-dimensional coordinates of the target point in the coordinate system of the camera; A third transformation matrix between the instantaneous velocity of the end of the robot arm and the instantaneous change of the three-dimensional coordinates is obtained, and the third transformation matrix is ​​integrated into the second transformation matrix to obtain the first transformation matrix.

7. The method according to claim 1, characterized in that The method of performing admittance inference on the first acceleration of the robotic arm at the time t-1 based on the estimated speed to obtain the second acceleration of the robotic arm at the time t includes: Performing an integration operation on the estimated speed to obtain estimated posture information; Determine a first difference between the estimated speed and the speed of the robotic arm at the time t-1, and determine a second difference between the estimated posture information and the posture information of the robotic arm at the time t-1; Based on the damping matrix and the stiffness matrix of the mechanical arm, weighted summing the first difference and the second difference is performed to obtain the acceleration error of the mechanical arm; Based on the acceleration error, the first acceleration is corrected to obtain the second acceleration.

8. A robot arm control device, characterized in that: The device comprises: A data acquisition module, used to obtain simulation data of the robot arm's simulation target task, wherein the target task is used to control the end of the robot arm to move to a target point; A posture reasoning module is used to obtain the first posture information of the manipulator at the t-1th time of the target task, and perform spatiotemporal reasoning on the first posture information based on the simulation data to obtain the second posture information of the manipulator at the tth time of the target task; A visual servo module, used for collecting the image of the target point at the tth moment through a camera carried by the end of the robotic arm; The visual servo module is further used to perform an error prediction on the speed of the end of the robotic arm based on the image at the tth moment, obtain the speed error of the end of the robotic arm at the tth moment, and correct the simulated speed of the end of the robotic arm based on the speed error to obtain the estimated speed of the end of the robotic arm at the tth moment, wherein the simulated speed is obtained by performing a derivative operation based on the second posture information; A robot driving module, configured to perform admittance inference on a first acceleration of the robot at the time t-1 based on the estimated speed, so as to obtain a second acceleration of the robot at the time t; The robotic arm driving module is further used to drive the robotic arm to move based on the second acceleration.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the robot arm control method described in any one of claims 1 to 7 when executing the computer executable instructions or computer programs stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the robot arm control method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the robot arm control method according to any one of claims 1 to 7 is implemented.

Images