A model training method, device and robotic arm system

Abstract

The embodiment of the present application discloses a model training method, device and robotic arm system; the robotic arm system of the present application includes a slave robotic arm, a master robotic arm, an image acquisition device and a control device, the slave robotic arm includes M joint motors and an end for installing an actuator, the M joint motors make the end have m degrees of freedom, the actuator is used to operate the target object, the master robotic arm includes N joint motors and a handle, the N joint motors make the handle have m degrees of freedom, the handle is configured for the user to remotely operate the actuator, the image acquisition device acquires image information containing the target object, the control device is configured to obtain the angular positions of the M joint motors and the angular positions of the N joint motors, and input the image information, the angular positions of the M joint motors and the angular positions of the N joint motors into a neural network model to train the neural network model; through the above method, the training efficiency of the robotic arm system is high and the operability is strong.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arms, and in particular to a model training method, device and robotic arm system. Background Art

[0002] In recent years, with the technological breakthroughs of large models and AI, the application of general artificial intelligence (AGI) in the field of robots (or robotic arms) has become more and more extensive. Through a large number of model training for robots (or robotic arms), robots (or robotic arms) in real environments can be made intelligent, so that they can interact with the environment like humans and perform various tasks. Summary of the invention

[0003] An embodiment of the present application provides a robotic arm system, comprising: a slave robotic arm, comprising M joint motors and an end for mounting an actuator, the M joint motors enable the end to have m degrees of freedom, and the actuator is used to operate a target object; a master robotic arm, comprising N joint motors and a handle, the N joint motors enable the handle to have m degrees of freedom, and the handle is configured for a user to remotely operate the actuator; an image acquisition device, acquiring image information containing the target object; a control device, configured to obtain the angular positions of the M joint motors and the angular positions of the N joint motors, and input the image information, the angular positions of the M joint motors and the angular positions of the N joint motors into a neural network model to train the neural network model.

[0004] An embodiment of the present application provides a model training method applied to a robotic arm system, including: obtaining first posture data after the main robotic arm moves; obtaining second posture data after the slave robotic arm follows the movement of the main robotic arm and first image information of the target object; returning to execute the step of obtaining the first posture data after the main robotic arm moves until the target task is completed, and obtaining a training data set; using the training data set to train a neural network model to obtain a target neural network model of the slave robotic arm for the target task.

[0005] An embodiment of the present application provides a model training device, which has a display interface, and the display interface can display a first acquisition module, a second acquisition module, a loop module and a training module; the first acquisition module has at least one line of code, and is used to acquire the first posture data after the main robotic arm moves; the second acquisition module has at least one line of code, and is used to acquire the second posture data of the slave robotic arm after following the movement of the main robotic arm and the first image information of the target object; the loop module has at least one line of code, and is used to return to execute the step of acquiring the first posture data after the movement of the main robotic arm until the target task is completed, so as to obtain a training data set; the training module has at least one line of code, and is used to train the neural network model using the training data set to obtain a target neural network model of the slave robotic arm for the target task.

[0006] The beneficial effect of the present application is that the robotic arm system provided by the present application allows the user to remotely operate the slave robotic arm through the master robotic arm to operate the target object, and in this process, image information containing the target object is collected through the image acquisition device, and the image information, the angular position of the joint motor of the master robotic arm and the angular position of the joint motor of the slave robotic arm and other data are transmitted to the control device, and then these data are input into the neural network model for corresponding training, so as to obtain the neural network model desired by the user, with high training efficiency and strong operability. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 It is a structural schematic diagram of an embodiment of a robotic arm system provided in an embodiment of the present application;

[0009] Figure 2 It is a structural schematic diagram of an embodiment of a robotic arm system provided in an embodiment of the present application;

[0010] Figure 3 It is a structural schematic diagram of an embodiment of a robotic arm system provided in an embodiment of the present application;

[0011] Figure 4 It is a structural schematic diagram of an embodiment of a robotic arm system provided in an embodiment of the present application;

[0012] Figure 5 It is a flowchart of an embodiment of a model training method provided in an embodiment of the present application;

[0013] Figure 6 It is a flowchart of an embodiment of the master-slave control logic provided in an embodiment of the present application;

[0014] Figure 7 It is a flowchart of an embodiment of collecting training data corresponding to a target task provided by an embodiment of the present application;

[0015] Figure 8 It is a flowchart of an embodiment of training a neural network model provided by an embodiment of the present application;

[0016] Fig. 9 It is a flowchart of an embodiment of the correction process provided by the embodiment of the present application;

[0017] Fig.10It is a flowchart of an embodiment of a model training method provided in an embodiment of the present application;

[0018] Fig.11 This is a page diagram of an embodiment of a data collection page provided in an embodiment of the present application;

[0019] Fig.12 It is a flow chart of an embodiment of a control method of a robotic arm provided in an embodiment of the present application;

[0020] Fig.13 It is a structural schematic diagram of an embodiment of a model training device provided in an embodiment of the present application;

[0021] Fig.14 It is a structural schematic diagram of an embodiment of a model training device provided in an embodiment of the present application;

[0022] Fig.15 It is a structural schematic diagram of an embodiment of a model training device provided in an embodiment of the present application;

[0023] Fig.16 It is a structural schematic diagram of an embodiment of a model training device provided in an embodiment of the present application;

[0024] Fig.17 It is a structural schematic diagram of an embodiment of a model training device provided in an embodiment of the present application;

[0025] Fig.18 It is a structural schematic diagram of an embodiment of a model training device provided in an embodiment of the present application;

[0026] Fig.19 It is a structural schematic diagram of an embodiment of a control device for a robotic arm provided in an embodiment of the present application;

[0027] Fig. 20 It is a structural diagram of an embodiment of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] Combination Figure 1The robot system 100 may include a slave robot 10A, a master robot 10B, an image acquisition device 10C, and a control device 10D. The slave robot 10A and the master robot 10B may be the same or similar, and they may refer to traditional mechanical structures fixed at a certain station to perform certain functions, such as the slave robot 10A being a collaborative robot, or other emerging mechanical structures (or a part thereof), such as the slave robot 10A being a humanoid robot (or its arm).

[0030] As an example, the slave robot arm 10A may include M joint motors and an end for mounting an actuator, the M joint motors allowing the end to have m degrees of freedom, the actuator being used to manipulate a target object.

[0031] Among them, in this application, joint motors are generally used to enable the joints of robots (or robotic arms) to have the ability to move. Joint motors may include brushless DC motors, stepper motors, DC motors, servo motors, steering gears, etc., and the selection depends on application requirements, such as the required torque, speed, accuracy, and power efficiency. Further, the degree of freedom is the minimum number of coordinates required to determine the position of a system in space. Among them, m can be a natural number greater than zero, such as m is 4, 6 or other values. For example: the end of the robotic arm can move completely freely in the air, and at least six quantities (such as rectangular coordinates x, y, z and three angles θ, φ, ρ) are required to determine its position, that is, the end has 6 degrees of freedom in the air.

[0032] The actuator may be a gripper, a suction cup or other mechanism that can perform the target operation. The actuator may be driven or controlled by electric, pneumatic, hydraulic, etc.

[0033] The target object may be understood as an operation object under the target task performed by the robot system 100. For example, if the target task is folding clothes, the target object may include at least one piece of clothing to be folded, or if the target task is washing dishes, the target object may include at least one dish to be washed, and so on.

[0034] As an example, the master robot 10B may include N joint motors and a handle, wherein the N joint motors enable the handle to have m degrees of freedom, and the handle is configured to allow a user to remotely operate an actuator. The degrees of freedom of the handle and the degrees of freedom of the end may be equal, mainly to facilitate the master robot 10B to remotely operate the slave robot 10A.

[0035] The handle may include a handle body, a steering gear connected to the handle body, and a wrench connected to the steering gear, wherein the steering gear allows the wrench to rotate relative to the handle body.

[0036] Teleoperation can be understood as the process of controlling a device or system by remote means. This technology allows operators to perform various operations and controls through wireless communication or network connection without direct contact with the target device. Teleoperation is common in applications such as drones, remote-controlled vehicles, robots, and remote-operated devices. In this solution, teleoperation can be understood as the user operating the master robotic arm to achieve the target task of the remote-operated slave robotic arm.

[0037] As an example, the image acquisition device 10C can acquire image information containing the target object. The image information may be of various types, such as RGB images, depth images, infrared images, 3D point cloud information, or images of other color spaces or types, and the like.

[0038] As an example, the control device 10D is configured to obtain the angular positions of M joint motors and the angular positions of N joint motors, and input the image information, the angular positions of M joint motors and the angular positions of N joint motors into the neural network model to train the neural network model, see the relevant description below for details.

[0039] Among them, the control device 10D can be a server or a terminal or other device. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud storage, and cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0040] Among them, the neural network model can be understood as an action block algorithm model suitable for the field of robot (or robotic arm) technology. The neural network model can use Transformers (a deep learning model architecture) to implement the action block strategy and train it as a conditional variational autoencoder (CVAE) to capture the variability in human data. CVAE is a deep learning model that combines the concepts of variational autoencoder (VAE) and conditional generation. When generating data, CVAE not only relies on sampling points in the latent space, but also introduces an additional conditional input, so that samples that meet specific conditions can be generated. The basic structure of CVAE consists of two parts: an encoder and a decoder. The encoder compresses the input data into low-dimensional latent variables, while the decoder reconstructs or generates data based on these latent variables and conditional inputs.

[0041] In some embodiments, M can be equal to m, that is, the number of joint motors in the slave robot 10A is equal to the degree of freedom of the end. For example, the slave robot 10A can include 6 joint motors, so that the degree of freedom of the end is 6. N can be greater than m, that is, the number of joint motors in the master robot 10B can be greater than the degree of freedom of the handle. For example: N=m+1. Optionally, N can also be equal to m, that is, the number of joint motors in the master robot 10B can be equal to the degree of freedom of the handle.

[0042] In some embodiments, a sensor may be provided on the handle of the main robotic arm 10B, the sensor having a first level signal when the user holds the handle, and the sensor having a second level signal when the user does not hold the handle. The control device 10D may be configured to obtain the level signal of the sensor, and lock the main robotic arm when the level signal changes from the first level signal to the second level signal. In this way, the level signal output by the sensor can ensure that the main robotic arm 10B is locked when the user is separated from the main robotic arm 10B, thereby preventing the main robotic arm from falling or dropping.

[0043] In some embodiments, the robotic arm system 100 may further include a workbench 10E and a bracket 10F connected to the workbench, the master robotic arm 10B and the slave robotic arm 10A may be fixed on the workbench 10E, and the image acquisition device 10C may include a first camera C1 and a second camera C2, the first camera C1 is fixed on the bracket 10F and is located above the slave robotic arm 10A, and the second camera C2 is fixed on the end of the slave robotic arm 10A.

[0044] The first camera C1 can collect global image information of the robot system 100 when performing the target task, and the global image information can include not only the target object, but also the slave robot 10A, the master robot 10B and the actuator. The second camera C2 can collect local image information containing the target object, and the local image information can include the target object.

[0045] Taking the number of slave robot arms 10A and master robot arms 10B as 2 as an example, the overall architecture of the robot arm system 100 can be as follows: Figure 2 As shown, the user's left and right hands can be simulated in this way. The user remotely controls the master robot arm 10B with one hand or two hands, and the slave robot arm 10A follows the movement of the master robot arm 10B, so that the slave robot arm 10A operates the target object to complete the teaching process of the target task, thereby collecting the teaching data corresponding to the target task.

[0046] In some embodiments, if the operation of the target task requires changing the state of the actuator, the teaching data may also include the first state information of the handle and the second state information of the actuator, and the second state information changes when the first state information changes. It should be noted that, taking the actuator as a clamp as an example, changing the state of the actuator can be understood as changing the opening and closing degree of the clamp, taking the actuator as a suction cup as an example, changing the state of the actuator can be understood as making the suction cup have suction, and so on.

[0047] It should be noted that the robotic arm system 100 can be in a real environment, that is, the user can see and touch each device in the robotic arm system 100, for example, the slave robotic arm 10A, the main robotic arm 10B, the image acquisition device 10C, the control device 10D, the workbench 10E and the bracket 10F can be seen and touched by the user.

[0048] In the description of this application, it should be understood that terms such as "first" and "second" are only used to distinguish similar objects and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0049] Combination Figure 3 , the robot system 200 may include a control device 20A and a main robot 20B.

[0050] Among them, the relevant structure of the control device 20A is the same or similar to that of the control device 10D, and will not be described one by one here. Furthermore, the control device 20A includes a display interface A1, and the display interface A1 can display the virtual slave robot 20C and the virtual target object. An actuator is set at the end of the slave robot 20C, and the actuator has m degrees of freedom. The actuator is used to operate the target object. Among them, the virtual target object can be understood as the mapping of the operation object targeted by the robot system 200 to perform the target task in the virtual environment. For example, taking the target task of folding clothes as an example, the virtual target object can include at least one image of clothes that need to be folded mapped to the virtual environment, or, taking the target task of washing dishes as an example, the virtual target object can include at least one dish that needs to be washed in the virtual environment. The mapping, and so on. In the virtual environment displayed on the display interface A1, the actuator set at the end of the slave robot 20C can operate the virtual target object to perform various target tasks.

[0051] The main robot arm 20B includes N joint motors and a handle, the N joint motors enable the handle to have m degrees of freedom, and the handle is configured for a user to remotely operate an actuator. The relevant structure of the main robot arm 20B is the same or similar to that of the main robot arm 10B, and will not be described in detail here.

[0052] Furthermore, when the handle remotely operates the actuator, the control device 20A can obtain the posture data of the robotic arm, the angular positions of N joint motors and image information including the target object, and input them into the neural network to train the neural network model, as described in the following text for details.

[0053] Combination Figure 4 The robot arm system 300 may include a robot arm 30A, an image acquisition device 30B and a control device 30C.

[0054] Among them, the relevant structure of the robot arm 30A is the same as or similar to that of the slave robot arm 10A, and will not be repeated here.

[0055] The image acquisition device 30B is configured to acquire image information containing the target object, which can be specifically described above and will not be described in detail here.

[0056] Among them, the relevant structure of the control device 30C is the same or similar to that of the control device 10D, and will not be described one by one here. Further, the control device 30C is configured to obtain the operation instruction for the robot arm, and determine the task scene through the operation instruction, and select the target neural network model corresponding to the task scene from the preset neural network model set, so as to control the robot arm to operate the target object based on the target neural network model, as described in the following text.

[0057] Among them, the operation instruction can be understood as an instruction that triggers the robot arm to perform an operation on the target object, and the operation instruction can be generated through image information.

[0058] The robot arm system 300 may further include a voice collection device, which may include a microphone or other device capable of collecting audio or voice information. The voice collection device may be configured to collect the user's voice information, and after the user's voice information is collected by the voice collection device, a job instruction may be generated by the voice information.

[0059] In some embodiments, in combination Figure 1 and Figure 2 , the method of controlling the slave robot 10A, the master robot 10B and the image acquisition device 10C to perform model training is further described from the perspective of the control device 10D in the robot system 100. Accordingly, as Figure 5 As shown, the specific process of the model training method for the robotic arm system 100 is as follows:

[0060] 110. Obtain the first position data of the main robot arm after movement.

[0061] The master robot may include N joint motors and handles. Taking the target task as an example of teleoperating an actuator installed at the end of the slave robot to push the target object from point A to point B, the first position data may include the first angular position of the N joint motors of the master robot. There are many ways to obtain the first position data after the master robot moves, which can be as follows:

[0062] For example, the first position data of the main robot arm after movement can be directly read, or the first position data of the main robot arm after movement returned by the terminal or client corresponding to the robot arm system can be received, or the first position of the handle of the main robot arm in the Cartesian coordinate system sent by a third-party device can be received, and the first position can be inversely kinematically solved to obtain the first angular position of the N joint motors of the main machine, and the first angular position can be used as the first position data of the main robot arm after movement, and so on.

[0063] Among them, there are many ways for the third-party device to obtain the first position of the handle of the main robot arm in Cartesian coordinates. For example, the first angular position of the N joint motors of the main robot arm after movement can be obtained, and the first angular position can be kinematically solved to obtain the first position of the handle of the main robot arm in the Cartesian coordinate system.

[0064] Optionally, in some embodiments, taking the target task of remotely controlling an actuator installed at the end of a slave robot arm to grasp a target object through a handle of a master robot arm, the first position data may also include first state information of the handle. The first state information may be understood as information indicating the state of the handle, for example, the position of a wrench in the handle (i.e., the angular position of a servo directly connected to the wrench) or an electrical signal of a button, etc.

[0065] Among them, in this scheme, different data of the main robot arm can be collected as the first pose data according to the type of target task, so as to improve the accuracy of the training data set corresponding to the target task, and then improve the training accuracy of the neural network model.

[0066] In some embodiments, before obtaining the first position data after the movement of the master robot arm, the slave robot arm can also be controlled to align the posture with the master robot arm. There are many ways to align the posture, for example, initialize the slave robot arm to a preset posture, obtain the current posture data of the master robot arm, and compare the current posture data with the reference posture data corresponding to the preset posture. When the difference between the current posture data and the reference posture data does not exceed the preset difference threshold, the slave robot arm is controlled to align the posture with the master robot arm based on the current posture data.

[0067] The preset posture can be understood as a preset posture or position. There are many ways to initialize the slave robot to the preset posture, such as obtaining reference posture data corresponding to the preset posture, and generating a target motion instruction based on the reference posture data, and controlling the slave robot to move to the preset posture based on the target motion instruction.

[0068] The current posture data can be understood as the posture data of the main robot at the current moment (i.e. before the movement). The method of obtaining the current posture data can be similar to the method of obtaining the first posture data, as described above, and will not be repeated here.

[0069] After obtaining the current posture data of the master robot, the current posture data can be compared with the reference posture data corresponding to the preset posture. When the difference between the current posture data and the reference posture data does not exceed the preset difference threshold, the slave robot is controlled to align the posture with the master robot based on the current posture data.

[0070] Among them, there can be many ways to control the posture alignment of the slave robotic arm and the master robotic arm based on the reference posture data. For example, based on the reference posture data, a control instruction for the slave robotic arm can be generated, and based on the control instruction, the slave robotic arm can be controlled to move so that the posture of the slave robotic arm after movement is the same as that of the master robotic arm.

[0071] Among them, in this scheme, the posture alignment of the main robot arm and the slave robot arm can be controlled to ensure the accuracy of the collected training data, thereby improving the training accuracy of the neural network model corresponding to the target task.

[0072] In some embodiments, when the difference between the current posture data and the reference posture data exceeds a preset difference threshold, a posture abnormality prompt message can be output, and the slave robot arm can be controlled to stop following the movement of the master robot arm.

[0073] In some embodiments, after obtaining the first posture data after the movement of the main robotic arm, the historical posture data of the main robotic arm in the previous time step at the current moment can also be obtained, and the data difference between the first posture data and the historical posture data is calculated. When the data difference exceeds a preset difference threshold, a motion abnormality prompt message is output, and the slave robotic arm is controlled to stop following the movement of the main robotic arm.

[0074] Among them, the historical posture data can be understood as the posture data of the main robot arm at the previous time step at the current moment.

[0075] Among them, in this scheme, by calculating the data difference between the current posture data of the main robot arm and the historical posture data of the previous time step, it is possible to determine whether the main robot arm has motion abnormalities during the movement. When motion abnormalities occur (that is, the data difference exceeds the preset difference threshold), not only can the motion abnormality prompt information be output in time, but also the slave robot arm can be controlled to follow the movement of the main robot arm, thereby improving the safety of the robot system.

[0076] In some embodiments, the current loop values ​​of N joint motors can be obtained, and based on the current loop values, the N joint motors are controlled to perform gravity compensation, thereby improving the safety of the robotic arm system. Among them, the current loop value can be understood as the numerical value of the current loop. The so-called current loop, as the innermost control loop in the servo control system, can ensure that the current in the motor winding can quickly and accurately track the given reference value. The performance of the current loop directly affects the dynamic response and control accuracy of the entire servo system.

[0077] 120. Acquire second posture data of the slave robot arm after following the movement of the master robot arm and first image information of the target object.

[0078] The slave robot arm includes M joint motors and an end for mounting an actuator. The second posture data may include the second angular positions of the M joint motors, and may also include second state information of the actuator, and the second state information changes when the first state information changes. The type of the second state information may be related to the type of the actuator. For example, when the actuator is a clamp, the second state information may be the degree of opening and closing of the clamp, and when the actuator is a suction cup, the second state information may be the degree of vacuum of the suction cup, and so on.

[0079] The first image information can be understood as image information of at least one scale of the target object after the slave robot follows the movement of the master robot. Accordingly, the first image information can be global image information including the target object, the master robot and the slave robot, or local image information including only the target object.

[0080] There are many ways to obtain the second posture data of the slave robot following the master robot and the first image information of the target object, which can be specifically as follows:

[0081] For example, based on the first posture data, a control instruction for controlling the slave robotic arm can be generated. Based on the control instruction, the slave robotic arm is controlled to follow the main robotic arm to move, and the second angular position of the slave robotic arm and the second state information of the actuator are obtained after the slave robotic arm follows the movement of the main robotic arm. Of course, only the second angular position can be obtained to obtain the second posture data. Then, the image information of at least one scale of the target object after the slave robotic arm follows the movement of the main robotic arm can be collected by an image acquisition device to obtain the first image information.

[0082] The image acquisition device includes a first camera and a second camera, the first camera is fixed on a bracket of the robot system, and the second camera is fixed on the end of the slave robot. There are multiple ways to acquire image information of at least one scale of the target object after the slave robot follows the movement of the main robot by the image acquisition device. For example, the global image information of the target object after the slave robot follows the movement of the main robot is acquired by the first camera, and the global image information may include the slave robot and the target object, and the local image information of the target object after the slave robot follows the movement of the main robot is acquired by the second camera, and the local image information may include the target object, and the global image information and the local image information are used as the first image information.

[0083] 130. Return to the step of obtaining the first position data after the main robot arm moves until the target task is completed, and obtain the training data set.

[0084] Among them, the target task can be understood as a task that needs to be performed. There can be many types of target tasks, for example, it can include folding clothes, washing dishes, or other tasks that replace or assist human work, etc.

[0085] The condition for completing the target task may include completing the target task once, or completing the target task a preset number of times. The preset number of times may be set according to actual application, and may include an integer greater than 1, such as 20-50 times, and so on.

[0086] Among them, executing the target task requires the slave robot arm to follow the main robot arm to complete at least one movement. For example, taking the target task as washing dishes, the slave robot arm can follow the main robot arm to move the bowl to be washed to the cleaning area and perform at least one cleaning action, thereby cleaning the bowl to be washed. This process may include at least one movement action, that is, the slave robot arm needs to follow the main robot arm to move at least once.

[0087] Among them, the step of returning to the step of obtaining the first pose data after the main robot moves until the target task is completed, there are many ways to obtain the training data set, which can be as follows:

[0088] For example, you can return to the step of obtaining the first posture data after the movement of the main robot arm until the slave robot completes a target task. When the stop condition is to execute the target task once, you can stop obtaining the first posture data after the movement of the main robot arm, and stop obtaining the second posture data and first image information of the slave robot following the movement of the main robot arm, and use the at least one first posture data, at least one second posture data and at least one first image information obtained as the training data set corresponding to the target task; when the stop condition is to execute the target task a preset number of times, you can return to continue executing the step of obtaining the first posture data after the movement of the main robot arm until the target task is executed a preset number of times, and use the at least one first posture data, at least one second posture data and at least one first image information obtained as the training data set corresponding to the target task.

[0089] Among them, in this scheme, the master-slave control logic loop can be used to obtain the first posture data after the master robot moves, and the second posture data after the slave robot follows the master robot and the first image information of the target object, so as to improve the accuracy of the training data corresponding to the target task, and then improve the training accuracy of the neural network model. In the master-slave control logic, taking the control device as a computer device as an example, the master robot and the servo installed on the master robot establish communication with the computer device through the USB (a connection interface or method) serial port, the slave robot establishes communication with the computer device through TCP (a communication method), the actuator of the slave robot is connected to the slave robot through the 485 port, and indirectly establishes communication with the computer device through the TCP specific port, and the first camera (or the global camera set on the bracket) and the second camera (or the wrist camera set on the end of the slave robot) are connected to the computer device through USB3.0. On the basis of the above communication or connection, taking the actuator of the slave robot as a gripper as an example, the overall process of the master-slave control logic can be as follows Figure 6 As shown, the specific details can be as follows:

[0090] (1) The master robot needs to be placed close to the initialization state of the slave robot, and the slave robot moves to the initialization position. In order to avoid placing the master robot in an unsafe posture position, the posture difference between the master robot and the slave robot will be judged. When the posture difference is too large, an alarm message will be output and the program will be stopped;

[0091] (2) Control the slave robot to fine-tune the movement to the joint position where the master robot is placed, and perform master-slave alignment;

[0092] (3) Entering loop mode;

[0093] The first angular positions of the M joint motors of the main robot arm and the first state information of the handle are obtained as the first position data, and the current loop value can also be read for gravity compensation;

[0094] The first angular position of the main robot arm is sent by ServoJ to control the slave robot to follow the movement of the main robot arm and control the opening and closing degree of the gripper at the same time.

[0095] Among them, the slave robot is controlled by the master-slave control logic to execute the target task for a preset number of times, so as to collect the training data set for the target task. Taking the first posture data including the first angular position and the first state information, the first state information is the position of the wrench in the handle of the master robot, the second posture data including the second angular position and the second state information, the second state information is the opening and closing degree of the clamp of the slave robot, and the stop condition corresponding to the target task is 20-50 times of teaching demonstration (that is, executing the target task 20-50 times) as an example, the method of collecting the training data corresponding to the target task can be as follows Figure 7 As shown, the first angular position of the N joint motors of the main robot arm and the first state information of the handle are read to obtain the first posture data, and the first posture data is used as the action data. The first angular position of the main robot arm is sent by ServoJ to control the slave robot to follow the movement of the main robot arm, and the opening and closing degree of the gripper is controlled at the same time. The second angular position of the N joint motors and the second state information of the gripper after the slave robot follows the movement of the main robot arm are read to obtain the second posture data, and the first image information containing the target object is collected. The second posture data and the first image information are used as observation data, and the process of obtaining the action data and the observation data is repeated until the teaching demonstration of a single target task is completed to obtain a one-time operation data set of the target task, and the process of obtaining the action data and the observation data is continued until the teaching demonstration target task reaches 20-50 times, thereby obtaining 20-50 data sets of the target task, and the 20-50 data sets are used as the training data sets of the target task.

[0096] 140. The training data set is used to train the neural network model to obtain a target neural network model for the target task from the robotic arm.

[0097] The target neural network model can be understood as a trained neural network model that controls the slave robot to perform the target task. The network structure of the neural network model can be various, for example, it can include an ACT model (an imitation learning algorithm network / model) or other networks / models that can perform imitation learning, etc.

[0098] There are many ways to train a neural network model using a training data set, which can be as follows:

[0099] For example, a training data subset corresponding to a preset time step can be screened out in the training data set, and the training data subset includes a first pose data set, a second pose data set and an image information set. The preset time step includes at least one time step for performing a target task. Based on the image information set and the second pose data set, a neural network model corresponding to the target task is used to predict the pose data of the next time step of the robotic arm at each time step in the preset time step to obtain a predicted pose data set. Based on the predicted pose data set and the first pose data set, the neural network model is converged to obtain a target neural network model.

[0100] The preset time step includes at least one time step when executing the target task. It should be noted that the number of time steps required to execute the target task is that each movement of the robot arm corresponds to one time step.

[0101] Among them, according to the image information set and the second pose data set, a neural network model corresponding to the target task is used to predict the pose data of the slave robot at each time step in the preset time step. There are many ways to obtain the predicted pose data set. For example, each time step can be taken as the current time step, and the second pose data of the current time step can be filtered out from the second pose data set to obtain the target pose data set. At least one first image information corresponding to the current time step can be filtered out from the image information set to obtain the target image information set corresponding to the target pose data set. Based on the target image information set and the target pose data set, a neural network model is used to predict the pose data of the slave robot at the target time step to obtain the predicted pose data of the target time step. The target time step is the next time step of the current time step, and the predicted pose data of each target time step constitutes a predicted pose data set.

[0102] Among them, there are many ways to use a neural network model to predict the pose data of the slave robot at a target time step based on the target image information set and the target pose data set. For example, based on the target image information set and the target pose data set, a neural network model can be used to predict the pose data of at least one time step after the current time step to obtain predicted motion sequence data, and the predicted motion sequence data corresponding to a preset number of time steps before the current time step can be filtered out from the historical predicted motion sequence data set to obtain the historical predicted motion sequence data, and the predicted motion sequence data and the historical predicted motion sequence data can be fused to obtain the predicted pose data of the target time step.

[0103] Among them, the predicted motion sequence data can be understood as the predicted pose data of at least one time step after the current time step. For example, taking the current time step as t0 (t=0) as an example, the predicted three time steps after the current time step are predicted as an example, and the predicted motion sequence data can include the pose data predicted at t1 (t=1), the pose data predicted at t2 (t=2), and the pose data predicted at t3 (t=3). Based on the target image information set and the target pose data set, a neural network model is used to predict the pose data of at least one time step after the current time step. There are many ways to obtain the predicted motion sequence data. For example, a neural network model can be used to extract features of the target pose data in the target pose data set to obtain a first style variable feature, and multi-dimensional feature extraction can be performed on the target image information in the target image information set, and the extracted first image features of at least one dimension are fused to obtain a first target image feature. Based on the target pose data, the first style variable feature, and the first target image feature, the pose data of at least one time step after the current time step is predicted to obtain the predicted motion sequence data.

[0104] After predicting the pose data (predicted motion sequence data) of at least one time step after the current time step, and filtering out the historical predicted motion sequence data of a preset number of time steps between the current time step, the predicted motion sequence data and the historical predicted motion sequence data can be fused to obtain the predicted pose data of the target time step. There are many ways to fuse the predicted motion sequence data with the historical predicted motion sequence data. For example, the pose data of the target time step can be extracted from the predicted motion sequence data to obtain the current candidate pose data, the pose data of the target time step can be identified in the historical predicted motion sequence data to obtain the historical candidate pose data, and the current candidate pose data and the historical candidate pose data can be fused to obtain the predicted pose data of the target time step.

[0105] Among them, there are many ways to fuse the current candidate pose data with the historical candidate pose data. For example, you can obtain the fusion weight, and based on the fusion weight, weight the current candidate pose data and the historical candidate pose data respectively to obtain the weighted current candidate pose data and the weighted historical candidate pose data, and add the weighted current candidate pose data and the weighted historical candidate pose data to obtain the predicted pose data of the target time step.

[0106] Among them, taking the target time step as t3 (t=3), the predicted motion data sequence corresponding to t3 includes the predicted pose data corresponding to the three time steps of t3 (t=3), t4 (t=4) and t5 (t=5), the historical predicted motion sequence data sequence includes the predicted motion sequence data corresponding to the two time steps of t1 (t=1) and t2 (t=2), the predicted motion data sequence corresponding to t1 includes t1 (t=1), t2 (t=2) and t3 (t=3), and the predicted motion data sequence corresponding to t2 includes t2 (t=2), t3 (t=3) and t4 (t=4) as an example, the process of predicting the predicted pose data corresponding to t3 can include: extracting the predicted pose data corresponding to t3 from the predicted motion sequence data corresponding to t3 to obtain the current candidate pose data, extracting the predicted pose data corresponding to t3 from the predicted motion sequences corresponding to t1 and t2 respectively to obtain the historical candidate pose data, and weightedly fusing the current candidate pose data with the historical candidate pose data to obtain the predicted pose data corresponding to t3.

[0107] After using the neural network model to predict the posture data of the slave robot at the target time step and obtaining the predicted posture data of the target time step, the predicted posture data of each time step can be formed into a predicted posture data set.

[0108] After the neural network model corresponding to the target task is used to predict the posture data of the next time step of each time step in the preset time step from the robot arm to obtain the predicted posture data set, the neural network model can be converged based on the predicted posture data set and the first posture data set to obtain the target neural network model. There are many ways to converge the neural network model based on the predicted posture data set and the first posture data set. For example, the predicted posture data in the predicted posture data set can be compared with the first posture data of the same time step in the first posture data set to obtain the motion loss. Based on the motion loss, the network parameters of the neural network model are updated to obtain the updated neural network model, and the updated neural network model is used as the neural network model to return to the step of filtering out the training data subset corresponding to the preset time step in the training data set until the neural network model converges to obtain the target neural network model.

[0109] Among them, taking the neural network model as the ACT imitation learning network model, the target task as folding clothes, and the target object as clothes as an example, the process of training the neural network model can be as follows: Figure 8As shown, by inputting the image information of the current time step (3 target images containing the target object, where the first is a global image containing clothes and the slave robot arm, and the second and third are local images containing clothes and the actuator (e.g., gripper) installed at the end of the slave robot arm) and the first pose data corresponding to the current time step as observation data into the ACT imitation learning network model, the ACT imitation learning network model can output the predicted pose data of the next time step (target time step) of the current time step as action data, and return to the execution input process until the predicted pose data of each target time step corresponding to the target task is obtained, thereby obtaining a predicted pose data set, and converging the ACT imitation learning network model through the predicted pose data set and the first pose data, thereby obtaining the trained ACT imitation learning network model, and using the trained ACT imitation learning network model as the target neural network model corresponding to the target task.

[0110] Among them, in this application, the target neural network model corresponding to the target task can be obtained by training the ACT imitation learning network model. During the training process, the compound error (the accumulation of errors in multiple time steps causes the robot to deviate from the training distribution, making it difficult to complete the task) can be reduced, and it can also help solve time-related interference factors, such as pauses in the demonstration; the time-series overlapping action blocks can also be averaged through the time-series set method to make the action accurate and coherent.

[0111] In some embodiments, after the neural network model is trained with a training data set to obtain a target neural network model for a target task of a slave robotic arm, the target neural network model can be corrected. There can be many ways to correct the target neural network model. For example, the third angular position of the M joint motors of the slave robotic arm and the third image information of the target object can be obtained. Based on the third angular position and the third image information, the target neural network model is used to control the slave robotic arm to operate on the target object to perform the target task. When a correction signal for the slave robotic arm is obtained, the slave robotic arm is corrected to obtain a target training data set after correction. The target neural network model is updated based on the target training data set, and the updated neural network model is used as the target neural network model.

[0112] Among them, the method of obtaining the third position of the M joint motors and the third image information of the target object can be similar to the method of obtaining the second posture data of the slave robot following the movement of the main robot and the first image information of the target object. See the above for details and will not be repeated here.

[0113] After obtaining the third angular position and the third image information, the target neural network model can be used to control the slave robot to operate on the target object based on the third angular position and the third image information to perform the target task. There are many ways to use the target neural network model to control the slave robot to operate on the target object based on the third angular position and the third image information. For example, based on the third angular position and the third image information, the target neural network model can be used to predict the predicted posture data of the next time step corresponding to the time step at the current moment, obtain the current predicted posture data, and generate the current control instruction for the slave robot based on the current predicted posture data, control the movement of the slave robot according to the current control instruction, and return to the step of obtaining the third angular position of the M joint motors and the third image information of the target object to perform the target task.

[0114] In the process of controlling the slave robot to perform the target task, when a correction signal for the slave robot is obtained, the slave robot can be corrected to obtain a target training data set after correction. There are many ways to correct the slave robot. For example, when voice control information is detected, a correction signal for the slave robot is generated, and based on the voice control information, the angular position change of the M joint motors is determined, and the slave robot is corrected according to the angular position change to obtain a target training data set after correction.

[0115] Among them, the voice control information can be understood as including voice information for controlling the slave robotic arm, for example, it can include moving or exercising to the left / right / front / back or other positions such as 1CM distance, etc.

[0116] Among them, the angular position change can be understood as the angular position change corresponding to the M joint motors at the moment when the voice control information is detected. Based on the voice control information, there are many ways to determine the angular position change of the M joint motors. For example, the voice control information can be identified based on the correction signal to obtain the voice recognition content, and at least one keyword can be identified in the voice recognition content. Based on the keyword, the displacement change of the end of the robot arm is determined, and the current absolute coordinates of the end in Cartesian coordinates at the current time step are obtained. Based on the displacement change, the current absolute coordinates are adjusted to obtain the target absolute coordinates, and the angular position change of the M joint motors is calculated based on the target absolute coordinates and the current absolute coordinates.

[0117] Among them, there are many ways to identify voice control information based on the correction signal. For example, based on the correction signal, a voice recognition model can be used to extract features of the voice control signal to obtain voice features, and the voice features can be converted into text to obtain the text content of the voice control signal, and the text content can be used as the voice recognition content.

[0118] After the voice control information is recognized based on the correction signal, at least one keyword can be recognized in the recognized voice recognition content. Keywords can be understood as words related to controlling the slave robotic arm, for example, they can include the left slave robotic arm (or simply the left hand), the right slave robotic arm (or simply the right hand), left, right, front, back or other control-related words, etc. There are many ways to recognize at least one keyword in the voice recognition content. For example, the voice recognition content can be segmented to obtain at least one text word, the text words can be recognized to obtain the text type of each text word, and at least one keyword can be screened out from the text words based on the text type.

[0119] After at least one keyword is identified in the speech recognition content, the displacement change amount of the end of the slave robot arm can be determined based on the keyword. There are many ways to determine the displacement change amount of the end of the slave robot arm, for example, a first keyword is identified in the keyword, and a second keyword is searched in the keyword, the first keyword indicates the movement direction of the end, and the second keyword indicates the movement distance of the end in the movement direction. When the keyword includes the second keyword, the displacement change amount of the end is determined based on the movement direction and the movement distance. When the keyword does not include the second keyword, the movement distance corresponding to the movement direction is screened out in the preset displacement distance set, and the displacement change amount of the end is determined based on the movement direction and the movement distance.

[0120] Among them, the first keyword indicates the movement direction of the terminal, for example, it may include front, front, back, side and back, left, right, top, bottom, vertical direction, horizontal direction or other directions, etc.

[0121] The second keyword indicates the movement distance of the terminal in the movement direction, for example, it may include 1 cm, 2 cm, 3 cm or any other distance, and so on.

[0122] When the keyword includes the second keyword, it means that the voice control information includes a specific movement distance. In this case, the displacement change of the terminal can be determined based on the movement direction and the movement distance. There are many ways to determine the displacement change of the terminal. For example, the movement direction and the movement distance can be used as the displacement change of the terminal.

[0123] When the keyword does not include the second keyword, it means that the voice control information does not include a specific movement distance. In this case, the movement distance corresponding to the movement direction can be screened out from the preset displacement distance set. It should be noted that the movement distances corresponding to different movement directions in the preset displacement distance set can be the same or different.

[0124] After the movement distance corresponding to the movement direction is screened, the displacement change of the end can be determined based on the movement direction and the movement distance. The method of determining the displacement change of the end can be referred to above, and will not be described in detail here.

[0125] Among them, the current absolute coordinates can be understood as the absolute coordinates of the end at the current time step. Based on the displacement change, the current absolute coordinates are adjusted to obtain the target absolute coordinates. There are many ways to adjust the current absolute coordinates. For example, the coordinate value corresponding to the direction of movement can be identified in the current absolute coordinates, the displacement change is added to the coordinate value to obtain the target coordinate value, and the coordinate value in the current coordinates is replaced with the target coordinate value to obtain the target absolute coordinates.

[0126] After adjusting the current absolute coordinates, the angular position changes of the M joint motors can be calculated based on the adjusted target absolute coordinates and the current absolute coordinates. There are many ways to calculate the angular position changes of the M joint motors, for example, based on the current absolute coordinates, the current angular positions of the M joint motors are calculated by inverse solution, based on the target absolute coordinates, the candidate angular positions of the M joint motors are calculated by inverse solution, and the change between the current angular position and the candidate angular position is calculated to obtain the angular position changes of the M joint motors.

[0127] After determining the angular position changes of the M joint motors based on the voice control information, the slave robot arm can be corrected according to the angular position changes to obtain a target training data set after correction. There are many ways to correct the slave robot arm according to the angular position changes. For example, the predicted angular position of the M joint motors predicted by the target neural network at the current time step can be obtained, the predicted angular position is added to the angular position change to obtain the target angular position of the M joint motors, and the slave robot arm is controlled to operate the target object based on the target angular position, and the step of obtaining the third angular position of the M joint motors and the third image information of the target object is returned until the target task is completed to obtain the target training data set after correction.

[0128] Among them, the method of obtaining the predicted angular positions of the M joint motors predicted by the target neural network at the current time step can be similar to the method of using the neural network model corresponding to the target task to predict the posture data of the next time step from the robotic arm at each time step in the preset time step. See the above for details, and will not be repeated here.

[0129] There are many ways to control the slave robot to operate on the target object based on the target angular position. For example, a target control instruction can be generated based on the target angular position, and the slave robot can be controlled to operate on the target object based on the target control instruction.

[0130] After controlling the robot arm to operate the target object based on the target angular position, the step of obtaining the third angular position of the M joint motors and the third image information of the target object can be returned to until the target task is completed, thereby obtaining the target training data set after correction.

[0131] In some embodiments, when the movement of the master robotic arm is detected, a correction signal for the slave robotic arm is generated, and the fourth angular position of the N joint motors after the movement of the master robotic arm is obtained. Then, the fifth angular position of the N joint motors of the master robotic arm before the movement is obtained, and the difference between the fourth angular position and the fifth angular position is calculated to obtain the angular position change of the M joint motors. The slave robotic arm is corrected according to the angular position change to obtain the target training data set after correction.

[0132] Among them, the method of correcting the slave robot arm according to the change in angular position can be referred to as described above, and will not be repeated here.

[0133] After correcting the deviation of the slave robotic arm to obtain a corrected target training data set, the target neural network model can be updated based on the target training data set, and the updated neural network model can be used as the target neural network model.

[0134] Among them, the method of updating the target neural network model based on the target training data set can be similar to the method of training the neural network model using the training data set, as described above, and will not be repeated here.

[0135] The slave robot arm can be corrected by voice control or by the movement of the master robot arm. Taking the correction method as voice control correction and the target task as folding clothes as an example, the whole correction process can be as follows: Fig. 9 As shown, the content of the voice control information is identified to obtain the text content, and the keywords are matched in the text content. By adjusting the tool point, the displacement change of the end of the slave robot arm is calculated to obtain the new absolute coordinates (target absolute coordinates) of the Cartesian coordinate system of the end at the current time step. Then, the angular position change (delta J) of the M joint motors is calculated by inverse operation, and the predicted angular position (action J) and the angular position change (delta J) are added to obtain the target angular position. The target angular position is controlled to operate the slave robot on the target object (for example, clothes), thereby obtaining a target training data set after correction, and the target neural network model is updated (online / offline training) through the target training data set to obtain an updated neural network model, and the updated neural network model is used as the target neural network model corresponding to the target task.

[0136] Among them, in this scheme, while the slave robotic arm is operating on the target object, the slave robotic arm can be corrected, and the corrected data can be collected as the target training data set after correction. The trained neural network model is updated through the target training data set, thereby ensuring the real-time accuracy of the neural network model and further improving the accuracy and efficiency of model training.

[0137] In some embodiments, in order to prevent the slave robotic arm from falling or dropping, the slave robotic arm can also be protected. There can be multiple ways of protection. For example, the first angular position of the N joint motors of the master robotic arm can be extracted from the first posture data, and the movement speed of the actuator installed on the slave robotic arm in the Cartesian coordinate system can be determined based on the first angular position. When the speed component of the movement speed in the vertical direction exceeds a preset speed threshold, the slave robotic arm is locked, with fast response and high reliability in preventing falling.

[0138] Among them, based on the first angular position, there are many ways to determine the movement speed of the actuator installed on the slave robot arm in Cartesian coordinates. For example, the first angular positions of the N joint motors of the master robot arm are mapped to the angular positions of the M joint motors of the slave robot arm. The angular positions of the M joint motors can be converted into the positions of the end in the Cartesian coordinate system through the kinematic solution, thereby obtaining the movement speed of the actuator in the Cartesian system.

[0139] The vertical direction may be understood as the gravity direction of the actuator, or the direction perpendicular to the horizontal plane where the worktable of the robotic arm system is located.

[0140] In some embodiments, the method for protecting the slave robotic arm may also include: extracting the second angular position of the M joint motors of the slave robotic arm from the second posture data, and determining the current movement speed of the actuator installed on the slave robotic arm in the Cartesian coordinate system through kinematic solution based on the second angular position, and locking the slave robotic arm when the velocity component of the current movement speed in the vertical direction exceeds a preset speed threshold.

[0141] From the above, it can be seen that in the present application, after obtaining the first posture data after the movement of the main robotic arm, the second posture data after the slave robotic arm follows the movement of the main robotic arm and the first image information of the target object are obtained, and then, the step of obtaining the first posture data after the movement of the main robotic arm is returned to execute until the target task is completed, and a training data set is obtained. Then, the neural network model is trained using the training data set to obtain the target neural network model of the slave robotic arm for the target task; since this scheme can complete the target task by remotely controlling the slave robotic arm through the master robotic arm, the posture data of the master robotic arm and the slave robotic arm in the remote operation and the image information of the target object are directly collected, thereby realizing efficient and accurate collection of teaching data, and therefore, the efficiency of model training can be improved.

[0142] According to the method described in the above embodiment, combined with Figure 3 , the embodiment of the present application can also provide another model training method using the robotic arm system 200, such as Fig.10 As shown, the specific process of the model training method can be as follows:

[0143] 210. Display the data collection page.

[0144] The data collection page includes the slave robot and the target object in the virtual environment. The virtual environment can be understood as an operating environment or space generated and presented by virtual reality technology, which is different from the real environment. Fig.11 Figure 2 shows a schematic diagram of the data collection page. Fig.11 As shown, the data acquisition page can be understood as a display carrier of various virtual images in the virtual environment, such as a page displayed on a computer display screen, such as a display page provided by a virtual reality display device. The data acquisition page can display various types of slave manipulators (including slave manipulator 1, slave manipulator 2, slave manipulator 3, etc.) in the virtual environment, so that the user can select an adapted slave manipulator from them according to the requirements of the target task. The data acquisition page can also display various types of target objects (including target object 1, target object 2, target object 3, etc.) in the virtual environment, so that the user can select an adapted target object from them according to the requirements of the target task.

[0145] There are many ways to display the data acquisition page. For example, virtual reality technology can be used to map the slave robot of the robot system in the real environment to the virtual environment to generate a virtual slave robot. At the same time, the operation object (i.e., the target object) for the robot system 200 to perform the target task can also be mapped to the virtual environment to generate a virtual target object. Based on the virtual slave robot and the virtual target object, a data acquisition page is generated and displayed.

[0146] For ease of description, the slave robot in the virtual environment is referred to as the slave robot, and the target object in the virtual environment is referred to as the target object. Thus, the virtual environment at least includes the slave robot and the target object, and the slave robot can operate the target object under the control of the control instruction to perform a given target task.

[0147] It can be understood that the target object can be determined according to the target task actually performed by the robot system 200, and the target object may be different for different target tasks. For example, if the target task is to wash dishes, the target object may be a bowl or a plate; if the target task is to fold clothes, the target object may be clothes; if the target task is to peel an apple, the target object may be a fruit knife or an apple, and so on.

[0148] 220. In response to a control instruction for the slave robotic arm, control the slave robotic arm to operate on the target object in the virtual environment to perform a target task.

[0149] Among them, the control instruction can be understood as the interactive instruction set by the user in the real environment for the slave robot in the virtual environment, and the object of the control instruction is the slave robot in the virtual environment. The process of the slave robot operating the target object in the virtual environment can be understood as the process of the slave robot making a series of actions for the target object under the control of the control instruction. For any target task, the process of the slave robot executing the target task may include actions of multiple time steps. If each control instruction corresponds to controlling the action of the slave robot in one time step, multiple control instructions are required for the slave robot to complete the target task.

[0150] Among them, an actuator may be provided at the end of the slave robot arm, and the control instruction may include a motion instruction and an operation instruction, wherein the motion instruction controls the slave robot arm to move in the virtual environment, and the operation instruction controls the actuator to operate the target object in the virtual environment. As mentioned above, there may be many types of actuators, such as a gripper, a suction cup, or other mechanisms that can perform target operations.

[0151] In some embodiments, in response to the control instruction for the slave robotic arm, before controlling the slave robotic arm to operate the target object in the virtual environment, a control instruction for the slave robotic arm may be generated. There may be multiple ways to generate the control instruction for the slave robotic arm, for example, the control instruction for the slave robotic arm may be generated through the posture data of the master robotic arm.

[0152] The slave robot in the virtual environment is remotely operated by the master robot in the real environment. The master robot includes N joint motors and a handle. The N joint motors make the degree of freedom of the handle the same as the degree of freedom of the actuator. The handle is configured for the user to remotely operate the actuator. In the process of the master robot in the real environment remotely operating the slave robot in the virtual environment, the slave robot moves following the movement of the master robot. In the process of the user operating the handle of the master robot, the angles of the N joint motors change following the user's movements. At the same time, the user's movements of operating the handle and the angles of the N joint motors are automatically mapped to the virtual environment, so that the slave robot in the virtual environment can perform the same movements as the master robot, thereby realizing remote operation of the slave robot in the virtual environment.

[0153] The posture data of the master robot may include the angular positions of the N joint motors and the status information of the handles. There are many ways to generate control instructions for the slave robot using the posture data of the master robot. For example, the angular positions of the N joint motors and the status information of the handles may be obtained; motion instructions may be generated based on the angular positions, and operation instructions may be generated based on the status information, and the motion instructions and operation instructions may be used as control instructions for the slave robot.

[0154] The handle may include a handle body, a steering gear connected to the handle body, and a wrench connected to the steering gear, the steering gear allows the wrench to rotate relative to the handle body, and the state information includes the angular position of the steering gear. The user can operate the wrench to rotate the wrench relative to the handle body.

[0155] In some embodiments, current loop values ​​of N joint motors may be acquired, and based on the current loop values, the N joint motors may be controlled to perform gravity compensation.

[0156] In some embodiments, the control instructions may come from a variety of sources, including the posture data of the master robot in the real environment, the augmented posture data after data augmentation of the historical posture data of the master robot, the code language directly written by the developer based on the target task, the voice information corresponding to the user's real-time needs, etc. Different sources correspond to different control instruction generation methods. Specifically, the control instructions for the slave robot may also include generation using at least one of the following three methods:

[0157] (1) Generate control instructions through augmented pose data.

[0158] For example, the historical posture data of the master robot can be obtained, and the historical posture data includes the historical angular positions of the N joint motors of the master robot and the historical state information of the handle, and the historical angular positions are augmented to obtain the augmented angular positions, and the historical state information is augmented to obtain the augmented state information, thereby obtaining the augmented posture data. Based on the augmented angular positions, motion instructions can be generated to control the movement of the slave robot in the virtual environment; and based on the augmented state information, operation instructions can be generated to control the actuator to operate the target object in the virtual environment.

[0159] Among them, the historical posture data can be understood as the posture data of the main robot arm recorded in the process of the user operating the main robot arm to perform the target task in the real environment. The historical posture data may include the historical angular positions of the N joint motors of the main robot arm and the historical state information of the handle. Among them, the historical angular position can reflect the angle information, torque information, etc. corresponding to each time step of each joint that constitutes the main robot arm during the movement process, and the historical angular position can be used as the source of generating motion instructions; the historical state information can reflect the state information of the handle at each time step when the user uses the handle to operate the target object in the real environment or virtual environment, and the historical state information can be used as the source of generating operation instructions.

[0160] Among them, data augmentation is a technique that increases the number of original data sets by applying a series of transformations, aiming to improve the effect of model training. This method helps the model better adapt to sample changes and noise by introducing new data variants, thereby improving the generalization ability and robustness of the model and reducing the possibility of overfitting.

[0161] It should be noted that the number of historical pose data may be a natural number greater than or equal to 1, and the augmented pose data obtained after data augmentation based on the historical pose data may be a natural number greater than or equal to 1. For example, if the number of historical pose data is two, they are historical pose data A and historical pose data B. Historical pose data A includes historical angular position A and historical state information A, and historical pose data B includes historical angular position B and historical state information B. The augmented pose data obtained after data augmentation of historical pose data A may include augmented pose data A1, augmented pose data A2, ..., augmented position data An, and the augmented pose data obtained after data augmentation of historical pose data B may include augmented pose data B1, augmented pose data B2, ..., augmented pose data Bn, where n may be a natural number greater than 2. The augmented pose data A1 includes the augmented angular position A1 and the augmented state information A1, the augmented pose data A2 includes the augmented angular position A2 and the augmented state information A2, the augmented pose data B1 includes the augmented angular position B1 and the augmented state information B1, and the contents contained in the remaining augmented pose data are similar and will not be elaborated one by one.

[0162] In this scheme, historical pose data collected in the real environment is obtained, and data augmentation is performed based on the historical pose data to obtain augmented pose data. Control instructions for the slave robot in the virtual environment are then generated based on the augmented pose data, thereby reducing the data volume requirement for historical pose data in the real environment, making full use of limited historical pose data to generate new pose data, increasing the diversity of samples, and using the new pose data (augmented pose data) as the basis for generating control instructions.

[0163] (2) Generate control instructions through the code language written directly by developers based on the target task.

[0164] For example, a control code of a slave robot arm may be obtained, the control code including a motion control code and an operation control code. Based on the motion control code, a motion instruction may be generated to control the slave robot arm to move in a virtual environment; and based on the operation control code, an operation instruction may be generated to control the actuator to operate a target object in the virtual environment.

[0165] The control code can be understood as the program code written by the developer for the actions that the slave robot needs to perform according to the requirements of the target task. The control code is used to control the slave robot to perform the target task. The control code can be converted into control instructions for the slave robot in the virtual environment. Therefore, running the control code can realize the control of the slave robot in the virtual environment.

[0166] In this solution, the control code of the slave robot is obtained, and control instructions are generated based on the control code, including motion instructions and operation instructions. This process does not need to rely on the historical posture data of the master robot in the real environment, and does not require the user to teach based on the master robot in the real environment, which greatly reduces the dependence on the real environment in the process of determining the control instructions, and provides a more convenient and efficient way for the collection of subsequent training data.

[0167] (3) Generate control instructions through voice information corresponding to the user’s real-time needs.

[0168] For example, the user's voice information can be collected; the voice information is subjected to voice recognition to obtain at least one of motion information and operation information. The motion information can be understood as the motion control information for the slave robot in the virtual environment contained in the user's voice information, and the operation information can be understood as the operation control information for requiring the slave robot to operate the target object contained in the user's voice information. Accordingly, motion instructions are generated based on the motion information to control the movement of the slave robot in the virtual environment; and / or, operation instructions are generated based on the operation information to control the actuator to operate the target object in the virtual environment. The motion instructions and / or operation instructions can be used as control instructions.

[0169] Among them, voice information can be understood as the control instructions that the user requires the robot arm to complete the corresponding task through voice output. For example, the user's voice information can include "put the apple in the refrigerator" or "charge the mobile phone on the table, etc." There are many ways to collect the user's voice information. For example, the user's voice information can be collected through a voice collection device, or the user's voice information sent by the client or terminal can be received, or the control information of the target modality (such as video, text or other modalities other than voice modality) sent by the user can be received, and the control information is converted into voice information, etc.

[0170] In this application, the user's voice information is collected and then control instructions are generated based on the user's voice information. This process does not need to rely on the historical posture data of the main robot arm in the real environment, and the user does not need to teach based on the main robot arm in the real environment. This greatly reduces the dependence on the real environment in the process of determining the control instructions, and provides a more convenient and efficient idea for the collection of subsequent training data.

[0171] It can be understood that the control instruction of the slave robot arm can be determined by any one of the above four methods, or by at least two of the above four methods. If at least two of the above four methods are collected for determination, the reliability of generating the control instruction for controlling the slave robot arm can be increased.

[0172] 230. Collect the posture data of the slave robot in the virtual environment and the image information of the target object until the slave robot completes the target task to obtain a training data set.

[0173] Among them, the training data set is used to train the neural network model corresponding to the target task.

[0174] The posture data may include the angular positions of the M shutdown motors of the slave robot in the virtual environment, and may further include the state information of the actuator installed at the end of the slave robot in the virtual environment. The so-called state information can be referred to above, and will not be described here one by one.

[0175] The image information may include an image of at least one scale of the target object after the robotic arm moves in the virtual environment.

[0176] Among them, the slave robot needs to complete a preset number of step-by-step actions to complete the target task, and each step-by-step action corresponds to a time step. For example, taking the target task of instant noodles as an example, when the slave robot is executing the target task, the steps that need to be executed in chronological order may include: unpacking the instant noodle package, taking out the seasoning packet, tearing open the seasoning packet and pouring in the seasoning, adding hot water, covering the instant noodle bowl, etc. The step-by-step action can be understood as the above-mentioned execution steps. The time step corresponding to the step-by-step action can be understood as the time required to execute the step-by-step action. For example, unpacking the instant noodle package corresponds to a time step, taking out the seasoning packet corresponds to a time step, and the rest of the execution steps are similar, and each execution step corresponds to a time step.

[0177] There are many ways to collect the posture data of the robot arm in the virtual environment and the image information of the target object until the robot arm completes the target task to obtain the training data set, which can be as follows:

[0178] For example, the pose data of the robot arm and the image information of the target object at each time step in the virtual environment can be collected to obtain an initial training data set, and the data labels corresponding to the initial training data set can be obtained, and the data labels can be added to the initial training data set to obtain the training data set, which can be specifically as follows:

[0179] S1. Collect the pose data of the robot arm and the image information of the target object at each time step in the virtual environment to obtain the initial training data set.

[0180] Among them, the initial training data set can be understood as the data obtained after data collection of the process of the slave robot arm performing the target task in the virtual environment. The initial training data set may include the posture data of the slave robot arm in the virtual environment at each time step, and the image information of the target object at each time step. For example, if the slave robot arm needs to complete 5 step-by-step actions (respectively, step-by-step action 1, step-by-step action 2, step-by-step action 3, step-by-step action 4 and step-by-step action 5) to perform the target task, the time to complete the target task can be divided into 5 time steps (respectively, time step 1, time step 2, time step 3, time step 4 and time step 5), wherein the slave robot arm completes step-by-step action 1 in time step 1, and completes step-by-step action 2 in time step 2. Similarly, the corresponding step-by-step actions are completed in other time steps, which are not described one by one here. For any of the time steps, for the sake of convenience, taking time step 1 as an example, the model training device can collect the posture data generated by the slave robot arm completing step-by-step action 1 at time step 1, and the image information of the target object when the slave robot arm completes step-by-step action 1 at time step 1. After collecting the pose data and image information corresponding to each time step, the pose data and image information of all time steps are integrated to obtain the initial training data set.

[0181] Among them, the data acquisition page may include a first camera and a second camera in the virtual environment. The robotic arm system is also configured with a camera, and the camera may include a global camera and a local camera. The global camera can capture images of the slave robotic arm operating on the target object, and the global camera can be installed independently of the slave robotic arm; the local camera can capture state images of the target object under the operation of the slave robotic arm, and the local camera can be installed on the end effector of the slave robotic arm. Therefore, the first camera can be understood as the mapping of the global camera in the virtual environment, and the second camera can be understood as the mapping of the local camera in the virtual environment. It can be understood that the number of the first camera and the second camera can be determined according to the actual configuration of the robotic arm system, and the installation positions of the first camera and the second camera can also be determined according to the actual configuration of the robotic arm system.

[0182] There are many ways to collect the pose data of the robot arm and the image information of the target object at each time step in the virtual environment, which can be as follows:

[0183] For example, the pose data of the slave robot arm in each time step in the virtual environment is collected to obtain a pose data set; the first image information of the target object in the virtual environment at each time step is collected by a first camera to obtain a first image information set, and the first image information includes the target object and the slave robot arm; the second image information of the target object in the virtual environment at each time step is collected by a second camera to obtain a second image information set, and the second image information includes the target object; the pose data set, the first image information set and the second image information set are used as the initial training data set.

[0184] Among them, for the collection of posture data, as mentioned above, the slave robot needs to complete a preset number of step-by-step actions to complete the target task, and each step-by-step action corresponds to a time step, that is, the process of the slave robot completing the target task can be divided into step-by-step actions corresponding to multiple time steps. Accordingly, in the process of collecting the initial training data set, the posture data corresponding to each time step of the slave robot in the virtual environment can be collected in chronological order, and the posture data corresponding to each time step can be combined to obtain a posture data set.

[0185] As for the acquisition of the image information of the target object, since two cameras (the first camera and the second camera) are configured in the virtual environment, the first camera can be used to acquire the global image of the slave manipulator in the virtual environment operating the target object, and the second camera can be used to acquire the state change image of the target object in the virtual environment under the operation of the slave manipulator. In the process of acquiring images, the first camera and the second camera can also acquire the corresponding image information in steps according to the time steps included in the target task. Specifically, the first camera can be used to acquire the first image information of the target object in each time step in the virtual environment, and the first image information corresponding to each time step can be integrated to obtain the first image information set. The first image information includes the target object and the slave manipulator, so the first image information can be understood as the global image of the slave manipulator in the virtual environment operating the target object. While acquiring the first image information, the second camera can also be used to acquire the second image information of the target object in each time step in the virtual environment, and the second image information corresponding to each time step can be integrated to obtain the second image information set. The second image information includes the target object, so the second image information can be understood as the state change image of the target object in the virtual environment under the operation of the slave manipulator.

[0186] It is understood that there are many ways to integrate the above, for example, the first image information can be arranged and combined in chronological order to obtain the first image information set. Similarly, the second image information can be arranged and combined in chronological order to obtain the second image information set.

[0187] After collecting the pose data set, the first image information set and the second image information set generated by the robotic arm completing the target task in the virtual environment, the pose data set, the first image information set and the second image information set can be packaged and the packaged data set can be used as the initial training data set.

[0188] S2. Obtain data labels corresponding to the initial training data set, and add the data labels to the initial training data set to obtain the training data set.

[0189] The data label can be understood as the target that the slave robot arm in the real environment needs to achieve in the process of completing the target task. As a target reference object, the data label can be used to judge the accuracy and completion of the action of the slave robot arm in the real environment in the process of performing the target task. Each target task has a corresponding data label. By adding the data label to the initial training data set corresponding to the target task, a training data set for the target task can be obtained.

[0190] Since the source of the control instruction can include generation based on the posture data of the main robot arm (angular position of N joint motors and status information of the handle), data generation based on historical posture data after data augmentation, generation based on control code or generation based on voice information, etc. Therefore, the method of obtaining the data label corresponding to the initial training data set can specifically include: when the control instruction is generated based on the posture data of the main robot arm, the main arm posture data corresponding to the slave arm posture data in the posture data set is screened out in the main arm posture data to obtain the data label; when the control instruction is generated based on the historical posture data after data augmentation, the data label corresponding to the initial training data set is determined based on the augmented angular position and augmented status information; when the control instruction is generated based on the control code, the data label corresponding to the initial training data set is generated based on the control code; when the control instruction is generated based on voice information, the data label corresponding to the initial training data set is generated based on the voice information. It can be specifically as follows:

[0191] (1) When the control instruction is generated based on the posture data of the main robot arm, the target posture data belonging to the same time step as the posture data in the posture data set can be screened out from the posture data of the main robot arm, and the target posture data can be used as the data label.

[0192] Among them, the posture data of the master robot can be understood as the posture data generated by the user operating the master robot in the real environment in the process of completing the corresponding task. As mentioned above, since the slave robot can be remotely operated by the master robot in the real environment, it means that the posture data of the master robot can be converted into control instructions for the slave robot in the virtual environment in real time, so that the slave robot in the virtual environment can follow the master robot and make the same action as the master robot. Therefore, the posture data of the master robot can be used as the goal that the slave robot in the real environment needs to achieve in the process of completing the corresponding task, that is, the posture data of the master robot can be used as the source of data labels.

[0193] (2) When the control instruction generates augmented data based on the historical posture data, the augmented angular position and augmented state information of each time step can be used as the augmented posture data of each time step. The augmented posture data of the corresponding time step in the posture data set can be filtered out from the augmented posture data to obtain the data label of each time step.

[0194] (3) When the control instruction is generated based on the control code, the target angular position corresponding to the joint motor can be extracted from the control code, and the target state information corresponding to the actuator can be extracted from the control code, and the target angular position and the target state information can be used as data labels corresponding to the initial training data set.

[0195] Among them, the control code can be the program code written by the developer according to the requirements of the target task, for the actions that the slave robot needs to perform. This means that the control code contains the various requirements for the slave robot to perform the target task, that is, the control code can contain the various action indicators that the slave robot needs to complete, such as the angular position that each joint motor needs to reach in the process of the slave robot performing the target task, and the status information of each actuator, so that this information is used as the data label corresponding to the initial training data set.

[0196] (4) When the control instructions are generated based on voice information, the angular positions that each joint motor needs to reach and the status information of each actuator can be pre-set for each target task, and a preset action indicator data set for the target task can be obtained. When voice information is detected, the voice information can be recognized, and a preset action indicator data set corresponding to the voice information can be determined based on the recognition result, and the preset action indicator data set can be used as the data label corresponding to the initial training data set.

[0197] Furthermore, there are many ways to add data labels to the initial training data set. For example, the training data (pose data and image information) of each time step in the initial training data set can be associated with the data label of the corresponding time step to obtain the training data set.

[0198] In some embodiments, after collecting the posture data of the slave robot in the virtual environment and the image information of the target object until the slave robot completes the target task to obtain a training data set, the neural network model corresponding to the target task can be trained based on the training data set to obtain a target neural network model corresponding to the target task. There are many ways to train the neural network model corresponding to the target task based on the training data set. For example, based on the training data set, a neural network model can be used to predict the posture data of the next time step of each time step to obtain a predicted posture data set, and the target data label corresponding to the predicted posture data in the predicted posture data set is screened out in the training data set, and the posture loss corresponding to the predicted posture data is determined based on the target data label, and the neural network model is converged based on the posture loss to obtain the target neural network model.

[0199] The predicted pose data set may include the predicted pose data of the slave robot at each time step. In this solution, the initial training data set for the target task in the training data set may be input into the neural network model, and the neural network model may output the predicted pose data set for the target task. Since the initial training data set includes pose data of multiple time steps and image information of the target object, the pose data of the slave robot at the next time step of each time step may be predicted in sequence according to the order of the time steps, thereby obtaining the predicted pose data set.

[0200] Among them, there can be many ways to filter out the target data labels corresponding to the predicted pose data in the predicted pose data set in the training data set. For example, the data label of the time step corresponding to each step-by-step action can be filtered out in the training data set as the target data label of the time step, thereby obtaining the target data label corresponding to the predicted pose data of each time step.

[0201] After filtering out the target data labels corresponding to the predicted pose data, the pose loss corresponding to the predicted pose data can be determined based on the target data labels. There are many ways to determine the pose loss corresponding to the predicted pose data. For example, the difference between the predicted pose data and the corresponding target loss can be calculated to obtain the pose loss corresponding to the predicted pose data.

[0202] After determining the posture loss corresponding to the predicted posture data, the neural network model can be converged based on the posture loss to obtain the target neural network model. The method of converging the neural network model based on the posture loss can be referred to above, and will not be described here one by one.

[0203] In some embodiments, in order to improve the training accuracy of the neural network model, the neural network model can also be assisted in training through a training data set corresponding to the target task in a real environment. There can be many ways of auxiliary training. For example, the training data set after the target task is trained in a virtual environment can be used as the first training data set, and the neural network model is trained using the first training data set to obtain a candidate neural network model; the training data set after the target task is trained in a real environment is obtained as the second training data set; the candidate neural network model is trained based on the second training data set to obtain a target neural network model. Among them, the specific implementation method of obtaining the training data set after the target task is trained in a real environment is detailed in the relevant description above, and will not be repeated here.

[0204] Among them, the collection of the second training data set depends on the actual movement of the slave robotic arm in the real environment, and often requires more manpower and material resources in the collection process. By mapping the slave robotic arm in the real environment to the virtual environment, it is possible to simulate the movement of the slave robotic arm in the virtual environment and obtain the first training data set that can be used for model training. This can improve the flexibility and efficiency of the training data collection process.

[0205] In this solution, the neural network model can be preliminarily trained based on the first training data set to obtain a candidate neural network model. Compared with the initial neural network model, the model parameters of the candidate neural network model have been preliminarily optimized, and the prediction accuracy has been improved. After obtaining the candidate neural network model, the candidate neural network model can be further trained based on the second training data set until the candidate neural network model converges, and the target neural network model can be obtained. The process of training the candidate neural network model can be understood as the process of fine-tuning the model parameters of the candidate neural network model.

[0206] In the process of using the second training data set to assist model training, model training can be divided into two steps. The first step is to train the initial neural network model based on the first training data set to obtain a candidate neural network model; the second step is to train the candidate neural network model based on the second training data set to obtain a target neural network model. In this way, the training data set collected in the virtual environment (the first training data set) and the training data set collected in the real environment (the second training data set) can be fully utilized. In addition, by adopting a two-stage training method, a candidate neural network model with a certain prediction accuracy can be obtained after the first step of training, and the second training data set can be used in the second step to fine-tune the candidate neural network model to reduce the overfitting of the model and improve the prediction accuracy and robustness of the trained target neural network model.

[0207] In some embodiments, the method of assisting model training with the second training data set may include: using the training data set as the first training data set, obtaining the second training data set after the target task is trained in the real environment; fusing the first training data set and the second training data set to obtain the target training data set; using the target training data set to train the neural network model to obtain the target neural network model. Among them, the method of obtaining the second training data set after the target task is trained in the real environment can be referred to as described above, and will not be repeated here. Further, there are many ways to fuse the first training data set and the second training data set. For example, the first training data set and the second training data set can be directly put into the same data set, which is the target training data set, or the data corresponding to each step action in the first training data set and the second training data set can be reorganized and exchanged in units of time steps. In this process, a new training data set can be generated to further increase the diversity of the training data set. For the newly generated training data set, it can be put into the same data set together with the first training data set and the second training data set, and the data set is the target training data set. Further, after the first training data set and the second training data set are fused, the neural network model can be trained based on the fused target training data set to obtain the target neural network model. The method of training the neural network model based on the fused target training data set can be similar to the method of training the neural network model based on the training data, as described above, and will not be repeated here.

[0208] As can be seen from the above, the present application displays the slave robot and the target object by displaying a data collection page; then, in response to the control instructions for the slave robot, the slave robot is controlled to operate on the target object in a virtual environment to perform the target task; and then the posture data of the slave robot in the virtual environment and the image information of the target object are collected until the slave robot completes the target task to obtain a training data set, which is used to train the neural network model corresponding to the target task. Since the scheme can map the slave robot in the real environment to the virtual environment through 3D modeling and other methods, not only can the neural network model be trained, but also the slave robot and the target object in the virtual environment are conducive to reducing the training cost of the neural network model and improving the flexibility and versatility of the training.

[0209] According to the method described in the above embodiment, after training the neural network model to obtain the target neural network model for the target task, the target neural network model can also be used to control the robot arm to perform the target task. Figure 4 The embodiment of the present application may also provide a control method for a robot arm applied to the robot arm system 300, such as Fig.12As shown, the specific process of the control method of the robotic arm can be as follows:

[0210] 310. Obtain at least one modal information of the working area where the robot arm is located.

[0211] Among them, the working area allows the placement of at least one target object. The target object can be understood as an object placed in the working area to be operated by the robotic arm. For example, if an apple is placed in the working area, and the robotic arm can cut or peel the apple, then the apple is the target object. Furthermore, the modal information may include various types of images, videos, audios or texts, etc. Accordingly, the image information of the area where the robotic arm is located can be collected by an image acquisition device to obtain modal information under at least one image modality, the audio information of the area where the robotic arm is located can be collected by an audio acquisition device to obtain modal information under an audio modality, and various types of sensors can be used to collect information of the area where the robotic arm is located to obtain at least one modal information, and so on.

[0212] The method of triggering the acquisition of at least one modal information of the robot arm in the working area may include: acquiring an operation instruction for the robot arm, and in response to the operation instruction, acquiring at least one modal information of the area where the robot arm is located. The operation instruction may be understood as instruction information that triggers the robot arm to start working in the working area.

[0213] In some embodiments, the method of obtaining the operation instruction for the robot arm may include: displaying a task list, the task list includes at least one preset task, determining the task information in response to the selection operation on the task list, and generating the operation instruction based on the task information. The preset task may be a pre-set task that the robot arm can perform, for example, it may include "cooking task", "cleaning task", and "carrying task". Furthermore, there may be multiple ways to display the task list, for example, the robot arm startup page may be displayed, the startup page includes a task selection control, and in response to the trigger operation on the task selection control, the task selection interface is displayed, and the task selection interface includes the task list.

[0214] After the task list is displayed, the user can select the task list by clicking or selecting with the mouse, and then determine the task information in response to the selection operation on the task list. The task information can be understood as information containing the selected task, for example, it can include task identification, task attribute information, task start time, stop time, task preset result information or other task-related information.

[0215] In some embodiments, the method of obtaining the operation instruction for the robot arm may further include: receiving a control request for the robot arm, the control request carrying task information corresponding to the robot arm, and generating the operation instruction based on the task information. The control request may be understood as a request to control the robot arm.

[0216] Among them, there can be many ways to trigger a control request for the robotic arm in the robotic arm system. For example, corresponding trigger controls such as buttons can be set on the robotic arm system. For example, a button corresponding to the "cooking task", a button corresponding to the "cleaning task", and a button corresponding to the "carrying task" can be set. Then, when it is detected that the user presses the button corresponding to the "cleaning task", a corresponding robotic arm control request is generated, and the control request carries the task information corresponding to the cleaning task, and so on.

[0217] In some embodiments, the method of obtaining operating instructions for the robotic arm may also include: obtaining voice information for the robotic arm, performing voice recognition on the voice information to obtain task information, and generating operating instructions based on the task information.

[0218] Among them, an audio acquisition device can be installed in the robot arm system to collect voice information for the robot arm through the audio acquisition device. After obtaining the voice information for the robot arm, the voice information can be recognized by a voice recognition model or other methods to obtain task information.

[0219] 320. Use a task identification model to identify at least one modal information to determine a task scenario.

[0220] Among them, the task recognition model can understand and recognize the model of the task or scene corresponding to the target object, such as CNN, RNN, DNN, LSTM, transformer or other models that can perform task or scene recognition. Furthermore, the task scene can be a task or scene that requires operations on the target object. For example, taking the target object as a bowl, the task scene can include washing dishes, moving bowls or other tasks or scenes that operate on bowls.

[0221] There are many ways to use the task identification model to identify at least one modal information to determine the task scenario, which can be as follows:

[0222] For example, a task recognition model can be used to identify the object information of the target object in at least one modal information. The object information can be understood as information indicating the object type, object identification or other attributes of the target object. For example, taking the target object as a bowl, the object information can include the type of bowl, the identification of the bowl, the color of the bowl, the size of the bowl, the position of the bowl in the work area or other attribute information, etc. Specifically, the task recognition model can be used to recognize each modal information separately to obtain at least one initial object information of the target object, and the initial object information can be fused to obtain the object information of the target object; or, the task recognition model corresponding to each modal information can be selected from the preset task recognition model set, and the task recognition model can be used to perform object recognition on the corresponding modal information, so as to obtain the initial object information corresponding to each modal information, and the initial object information can be fused to obtain the object information of the target object, and so on.

[0223] (1) When there is only one target object, the object information can be used as the first object information, and task scenes can be screened out from a preset task scene set based on the first object information. There are multiple ways to screen out task scenes from a preset task scene set based on the first object information. For example, at least one candidate task scene corresponding to the first object information can be screened out from the preset task scene set, and the task scene corresponding to the target object can be screened out from the candidate task scenes based on the job instruction. Alternatively, the object identifier or type of the target object can be extracted from the first object information, and the task scene corresponding to the object identifier or type can be screened out from the preset task scene set as the task scene corresponding to the target object, and so on.

[0224] Among them, there may be multiple ways to filter out at least one candidate task scene corresponding to the first object information in the preset task scene set. For example, taking the first object information indicating that the target object is a bowl as an example, at least one candidate task scene related to the bowl can be filtered out in the preset task scene set. For example, it may include cleaning tasks (washing dishes), carrying tasks (placing / carrying / moving the bowl to a designated location), or placing a specific item in the bowl and other candidate task scenes.

[0225] After at least one candidate task scenario is screened out, the task scenario corresponding to the target object can be screened out from the candidate task scenarios according to the task instructions. There are many ways to screen out the task scenario corresponding to the target object from the candidate task scenarios according to the task instructions. For example, the job type of the job instruction is determined, and the task scenario corresponding to the job type is screened out from the candidate task scenarios. For example, if the target object is a bowl, if the job type is a cleaning job, washing dishes can be screened out as a task scenario from the candidate task scenarios. If the job type is a carrying job, carrying bowls can be screened out as a task scenario from the candidate task scenarios, and so on.

[0226] (2) When there are multiple target objects, the object information may be used as the second object information. Based on the second object information, the target objects are compared, and based on the comparison result and the operation instruction, the task scene is screened out from the preset task scene set.

[0227] Among them, there can be multiple ways to compare the target objects based on the second object information. For example, the current object information corresponding to each target object can be identified in the second object information, and the current object information can be compared to obtain a comparison result, which indicates whether the target objects are of the same type of object. For example, the target objects include large bowls, small bowls and plates. At this time, the large bowls and small bowls can belong to the same type of objects, but the large bowls and plates or the small bowls and plates do not belong to the same type of objects.

[0228] After the target object is compared based on the second object information, the task scene can be screened out from the preset task scene set based on the comparison result and the operation instruction. There are multiple ways to screen out the task scene from the preset task scene set based on the comparison result and the operation instruction. For example, when the target objects are objects of the same category, the task scene is screened out from the preset task scene set based on the second object information. When the target objects are objects of different categories, the candidate task scene corresponding to each target object is screened out from the preset task scene set based on the second object information, and the task scene is determined from the candidate task scene based on the operation instruction.

[0229] When the target objects are objects of the same category, a candidate task scene of any target object is screened out from a preset task scene set according to the second object information, and the candidate task scene is used as the task scene.

[0230] Among them, when the target objects are of different categories, the candidate task scenes corresponding to each target object can be screened out from the preset task scene set according to the second object information. When the operation instruction is generated based on the task information, the task scene is screened out from the candidate task scenes based on the task information. When the operation instruction is generated based on the object detection result of the operation area, the attribute information of the candidate task scene is obtained, and the candidate task scenes are sorted based on the attribute information to obtain the task scene.

[0231] 330. According to the task scenario, select the target neural network model from the preset neural network model set.

[0232] The preset neural network model set can be understood as a pre-set trained neural network model. The trained neural network model can be obtained by training using the above model training method.

[0233] Among them, according to the task scenario, there are many ways to filter out the target neural network model from the preset neural network model set, which can be as follows:

[0234] For example, the corresponding target task can be determined according to the task scenario, and the neural network model corresponding to the target task can be screened out from the preset neural network model set to obtain the target neural network model.

[0235] Among them, there are many ways to determine the corresponding target task according to the task scenario. For example, a task set corresponding to a preset neural network model set can be obtained, and the tasks corresponding to the task scenario can be screened out from the task set to obtain the target task.

[0236] The so-called target task can be understood as the task that the robot arm needs to perform in the task scenario. For example, if the task scenario is a dishwashing scenario, the target task corresponding to the task scenario can be dishwashing, or if the task scenario is a transporting scenario, the target task corresponding to the task scenario can be transporting the target object, and so on.

[0237] After the corresponding target task is determined, the neural network model corresponding to the target task can be screened out from the preset neural network model set. There are many ways to screen out the neural network model corresponding to the target task from the preset neural network model set. For example, the task identifier of the target task can be determined, and the neural network model corresponding to the task identifier can be screened out from the preset neural network model set to obtain the target neural network model, and so on.

[0238] In some embodiments, according to the task scenario, the method of filtering out the target neural network model from the preset neural network models may also include: determining the corresponding target task according to the task scenario, filtering out the neural network model corresponding to the target task from the preset neural network model set, obtaining the initial neural network model, obtaining the model configuration parameters corresponding to the target task, and configuring the initial neural network model based on the model configuration parameters, so as to obtain the target neural network model.

[0239] The initial neural network model can be understood as an untrained or pre-trained general neural network model for performing at least one task.

[0240] The model configuration parameters can be understood as the network parameters of the neural network model corresponding to the target task trained according to the model training method described above. There are many ways to configure the initial neural network model based on the model configuration parameters. For example, the network parameters in the initial neural network model can be replaced with the network parameters in the model configuration parameters to obtain the target neural network model.

[0241] 340. Based on the target neural network model, control the robotic arm to operate on at least one target object.

[0242] For example, the posture data of the robotic arm and the image information of at least one target object can be obtained. Based on the posture data and the image information, the target neural network model is used to predict the posture data of the robotic arm at the next time step at the current moment to obtain the predicted posture data. Based on the predicted posture data, a control instruction for the robotic arm is generated. According to the control instruction, the robotic arm is controlled to perform an operation on at least one target object, and the step of obtaining the posture data of the robotic arm and the image information of at least one target object is returned to execute until the preset stop condition is reached to execute the target task in the task scenario.

[0243] The robot arm may include M joint motors and an end for mounting an actuator, the M joints enable the end to have m degrees of freedom, and the actuator is used to operate a target object. There are many ways to obtain the posture data of the robot arm and the image information of at least one target object. For example, the angular positions of the M joint motors and the state information of the actuator can be read to obtain the posture data of the robot arm, and the image information of at least one target object can be collected by an image acquisition device.

[0244] After acquiring the posture data of the robot and the image information of at least one target object, the target neural network model can be used to predict the posture data of the robot at the next time step at the current moment based on the posture data and image information, thereby obtaining the predicted posture data. There are many ways to use the target neural network model to predict the posture data of the robot at the next time step at the current moment. For example, the next time step at the current moment can be used as the target time step, and based on the posture data and image information, the target neural network model can be used to predict the posture data of at least one time step after the current time step corresponding to the current moment to obtain the predicted motion sequence data, and the predicted motion sequence data corresponding to the preset number of time steps before the current time step can be screened out from the historical predicted motion sequence data to obtain the historical predicted motion sequence data, and the predicted motion sequence data and the historical motion sequence data can be fused to obtain the predicted posture data of the target time step.

[0245] Among them, there can be multiple ways to use the target neural network model to predict the pose data of at least one time step after the current time step corresponding to the current moment based on the pose data and image information. For example, the target neural network model can be used to extract features from the pose data to obtain second style variable features, perform multi-dimensional feature extraction on the image information to obtain second image features of at least one dimension, fuse the second image features to obtain second target image features, and predict the pose data of at least one time step after the current time step based on the pose data, the second style variable features and the second target image features, thereby obtaining predicted motion sequence data.

[0246] Among them, there are many ways to fuse the predicted motion sequence data and the historical motion sequence data. For example, the pose data of the target time step can be extracted from the predicted motion sequence data to obtain the first candidate pose data, and the pose data of the target time step can be extracted from the historical predicted motion sequence data to obtain the second candidate pose data. The first candidate pose data and the second candidate pose data can be fused to obtain the predicted pose data of the target time step.

[0247] Among them, there are many ways to fuse the first candidate pose data and the second candidate pose data. For example, the pose data fusion weight can be obtained, and based on the pose data fusion weight, the first candidate pose data and the second candidate pose data are weighted respectively to obtain the weighted first pose data and the weighted second pose data, and the weighted first pose data and the weighted second pose data are added together to obtain the predicted pose data of the target time step.

[0248] After the target neural network model is used to predict the posture data of the robot arm at the next time step at the current moment based on the posture data and image information, a control instruction for the robot arm can be generated based on the predicted posture data. The control instruction may include motion instructions and operation instructions. The motion instructions control the robot arm to move, and the operation instructions control the actuator to operate or work on the target object. There are many ways to generate control instructions for the robot arm based on the predicted posture data. For example, the predicted angular position of M joint motors and the predicted state information of the actuator can be extracted from the predicted posture data. Based on the predicted angular position and the predicted state information, motion instructions for the M joint motors and operation instructions for the actuator are generated respectively. The motion instructions and operation instructions are used as control instructions for the robot arm, thereby generating control instructions for the robot arm.

[0249] After generating a control instruction for the robot arm based on the predicted posture data, the robot arm can be controlled to operate on at least one target object according to the control instruction. There are many ways to control the robot arm to operate on at least one target object according to the control instruction, such as controlling the movement of M joint motors of the robot arm through motion instructions, and controlling the actuator to operate on at least one target object through operation instructions to operate on at least one target object.

[0250] After controlling the robotic arm to operate on at least one target object according to the control instructions, the process can return to the step of acquiring the posture data of the robotic arm and the image information of at least one target object until the preset stop condition is reached to complete the target task in the task scenario.

[0251] The preset stop condition can be understood as a preset condition for stopping the robot arm from operating at least one target object. There can be multiple preset stop conditions, for example, the target object reaches a preset state or a specified position, or the number of operations of the robot arm on at least one target object reaches a preset number, or other preset stop conditions, etc.

[0252] It should be noted that the robotic arm can be one of the master robotic arm or the slave robotic arm mentioned above.

[0253] From the above, it can be seen that after obtaining at least one modal information of the working area where the robotic arm is located, the present application uses a task identification model to identify at least one modal information to determine the task scenario, and then screens out the target neural network model from the preset neural network model set according to the task scenario, and controls the robotic arm to operate on at least one target object in the working area based on the target neural network model, so that the robotic arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation and thus improving the operating efficiency and reliability of the robotic arm.

[0254] An embodiment of the present application also provides a model training device (hereinafter referred to as the first model training device), which can be integrated into an electronic device, such as a server or a terminal. The terminal may include a tablet computer, a laptop computer and / or a personal computer, etc.

[0255] For example, Fig.13 As shown, the first model training device has a display interface, which can display the first acquisition module 410, the second acquisition module 420, the loop module 430 and the training module 440, which can be specifically as follows:

[0256] (1) A first acquisition module 410, having at least one line of code, and used to acquire the first position data after the main robot arm moves.

[0257] (2) A second acquisition module 420, having at least one line of code, and used to acquire second posture data of the slave robot arm after following the movement of the master robot arm and first image information of the target object.

[0258] (3) A loop module 430 has at least one line of code and is used to return to the step of obtaining the first position data after the main robot arm moves until the target task is completed, thereby obtaining a training data set.

[0259] (4) A training module 440 having at least one line of code and used to train the neural network model using a training data set to obtain a target neural network model for a target task from the robotic arm.

[0260] In some embodiments, the first model training device may further include a posture alignment module 450, such as Fig.14 The posture alignment module 450 has at least one line of code and is used to control the posture alignment of the slave robot arm and the master robot arm.

[0261] For example, the posture alignment module 450 can be specifically used to initialize the slave robot arm to a preset posture, obtain the current posture data of the master robot arm, and compare the current posture data with the reference posture data corresponding to the preset posture. When the difference between the current posture data and the reference posture data does not exceed the preset difference threshold, the slave robot arm is controlled to perform posture alignment with the master robot arm based on the current posture data.

[0262] In some embodiments, the first model training device may further include a slave mechanical arm locking module 460, such as Fig.15 The slave mechanical arm locking module 460 has at least one line of code and is used to lock the slave mechanical arm under certain circumstances.

[0263] For example, the slave robot arm locking module 460 can be specifically used to extract the first angular position of the N joint motors of the master robot arm in the first posture data, and determine the movement speed of the actuator installed on the slave robot arm in the Cartesian coordinate system based on the first angular position. When the speed component of the movement speed in the vertical direction exceeds a preset speed threshold, the slave robot arm is locked.

[0264] In some embodiments, the first model training device may further include a correction module 470, such as Fig.16 The deviation correction module 470 has at least one line of code and is used to correct the deviation of the slave robot arm to obtain a target training data set after correction, and update the target neural network model based on the target training data set.

[0265] For example, the correction module 470 can be specifically used to obtain the third angular position of M joint motors and the third image information of the target object, and based on the third angular position and the third image information, use the target neural network model to control the slave robot arm to operate on the target object to perform the target task. When the correction signal for the slave robot arm is obtained, the slave robot arm is corrected to obtain a target training data set after correction, and the target neural network model is updated based on the target training data set, and the updated neural network model is used as the target neural network model.

[0266] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0267] From the above, it can be seen that, in the present application, after the first acquisition module 410 acquires the first posture data after the movement of the main robotic arm, the second acquisition module 420 acquires the second posture data after the slave robotic arm follows the movement of the main robotic arm and the first image information of the target object, and then, the loop module 430 returns to execute the step of acquiring the first posture data after the movement of the main robotic arm until the target task is completed, and a training data set is obtained, and then, the training module 440 uses the training data set to train the neural network model to obtain the target neural network model of the slave robotic arm for the target task; since this scheme can complete the target task by remotely controlling the slave robotic arm through the master robotic arm, directly collect the posture data of the master robotic arm and the slave robotic arm in the remote operation and the image information of the target object, thereby realizing efficient and accurate collection of teaching data, and therefore, the efficiency of model training can be improved.

[0268] An embodiment of the present application also provides a model training device (hereinafter referred to as the second model training device), which can be integrated into an electronic device, such as a server or a terminal. The terminal may include a tablet computer, a laptop computer and / or a personal computer, etc.

[0269] For example, Fig.17 As shown, the second model training device has a display interface, which can display a virtual slave robot arm and a virtual target object, and can display a control module 510 and a collection module 520, which can be specifically as follows:

[0270] (1) A control module 510 , having at least one line of code, and configured to respond to a control instruction for the slave robot arm and control the slave robot arm to operate on a target object in a virtual environment to perform a target task.

[0271] (2) An acquisition module 520 having at least one line of code and used to acquire posture data of the slave robot in the virtual environment and image information of the target object until the slave robot completes the target task, so as to obtain a training data set, which is used to train a neural network model corresponding to the target task.

[0272] In some embodiments, the second model training device may further include a training module 530, such as Fig.18 The training module 530 has at least one line of code and is used to train the neural network model corresponding to the target task through the training data to obtain the target neural network model.

[0273] For example, the training module 530 can be specifically used to predict the posture data of the next time step of each time step based on the training data set, using a neural network model to obtain a predicted posture data set, screen out target data labels corresponding to the predicted posture data in the predicted posture data set in the training data set, determine the posture loss corresponding to the predicted posture data based on the target data label, and converge the neural network model based on the posture loss to obtain a target neural network model.

[0274] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0275] As can be seen from the above, the present application displays a virtual slave robot and a virtual target object through a display interface, and displays a control module 510 and an acquisition module 520, wherein the control module 510 controls the slave robot to operate on the target object in a virtual environment in response to a control instruction for the slave robot to perform a target task; the acquisition module 520 acquires the posture data of the slave robot in the virtual environment and the image information of the target object until the slave robot completes the target task, so as to obtain a training data set, which is used to train a neural network model corresponding to the target task. Since the scheme can map the slave robot in the real environment to the virtual environment through 3D modeling and other methods, it can not only train the neural network model, but also the slave robot and the target object in the virtual environment, which is conducive to reducing the training cost of the neural network model and improving the flexibility and versatility of the training.

[0276] An embodiment of the present application also provides a control device for a robotic arm, which can be integrated into an electronic device, such as a server or a terminal, and the terminal may include a tablet computer, a laptop computer and / or a personal computer, etc.

[0277] For example, Fig.19 As shown, the control device of the robot arm has a display interface, which can display the acquisition module 610, the identification module 620, the screening module 630 and the control module 640, which can be specifically as follows:

[0278] (1) An acquisition module 610 , which has at least one line of code and is used to acquire at least one modal information of a working area where the robot arm is located, and the working area allows at least one target object to be placed.

[0279] (2) Identification module 620, having at least one line of code, and configured to identify at least one modal information using a task identification model to determine a task scenario.

[0280] (3) A screening module 630 , which has at least one line of code and is used to screen out a target neural network model from a preset set of neural network models according to a task scenario.

[0281] (4) A control module 640 having at least one line of code and used to control the robot arm to operate on at least one target object based on the target neural network model.

[0282] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0283] From the above, it can be seen that in this application, after the acquisition module 610 acquires at least one modal information of the working area where the robot arm is located, the identification module 620 uses the task identification model to identify at least one modal information to determine the task scenario, and then the screening module 630 filters out the target neural network model from the preset neural network model set according to the task scenario, and the control module 640 controls the robot arm to operate on at least one target object in the working area based on the target neural network model; so that the robot arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robot arm.

[0284] The present application also provides an electronic device, such as Fig. 20 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0285] The electronic device may include components such as a processor 710 with one or more processing cores, a memory 720 with one or more computer-readable storage media, a power supply 730, and an input unit 740. Those skilled in the art will appreciate that Fig. 20 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0286] The processor 710 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device, and performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 720, and calling data stored in the memory 720.

[0287] The memory 720 can be used to store software programs and modules, and the processor 710 executes various functional applications and data processing by running the software programs and modules stored in the memory 720. The memory 720 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc.

[0288] The power supply 730 can be logically connected to the processor 710 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system.

[0289] The input unit 740 may be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0290] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 710 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 720 according to the following instructions, and the processor 710 will run the application programs stored in the memory 720, thereby realizing various functions, as follows:

[0291] The first pose data after the main robot arm moves is obtained, the second pose data after the slave robot arm follows the movement of the main robot arm and the first image information of the target object are obtained, and the step of obtaining the first pose data after the main robot arm moves is returned to execute until the target task is completed, and a training data set is obtained. The training data set is used to train the neural network model to obtain a target neural network model of the slave robot arm for the target task.

[0292] Alternatively, a data acquisition page is displayed, which includes a slave robotic arm and a target object in a virtual environment. In response to a control instruction for the slave robotic arm, the slave robotic arm is controlled to operate on the target object in the virtual environment to perform a target task, and posture data of the slave robotic arm and image information of the target object in the virtual environment are collected until the slave robotic arm completes the target task to obtain a training data set, which is used to train a neural network model corresponding to the target task.

[0293] Alternatively, obtain at least one modal information of the working area where the robotic arm is located, the working area allows at least one target object to be placed, and use a task recognition model to recognize the at least one modal information to determine the task scenario; according to the task scenario, filter out a target neural network model from a preset neural network model set, and control the robotic arm to operate on at least one target object based on the target neural network model.

[0294] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0295] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0296] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions, which can be loaded by a processor to execute the steps in any of the model training generation or robotic arm control methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0297] The first pose data after the main robot arm moves is obtained, the second pose data after the slave robot arm follows the movement of the main robot arm and the first image information of the target object are obtained, and the step of obtaining the first pose data after the main robot arm moves is returned to execute until the target task is completed, and a training data set is obtained. The training data set is used to train the neural network model to obtain a target neural network model of the slave robot arm for the target task.

[0298] Alternatively, a data acquisition page is displayed, which includes a slave robotic arm and a target object in a virtual environment. In response to a control instruction for the slave robotic arm, the slave robotic arm is controlled to operate on the target object in the virtual environment to perform a target task, and posture data of the slave robotic arm and image information of the target object in the virtual environment are collected until the slave robotic arm completes the target task to obtain a training data set, which is used to train a neural network model corresponding to the target task.

[0299] Alternatively, obtain at least one modal information of the working area where the robotic arm is located, the working area allows at least one target object to be placed, and use a task recognition model to recognize the at least one modal information to determine the task scenario; according to the task scenario, filter out a target neural network model from a preset neural network model set, and control the robotic arm to operate on at least one target object based on the target neural network model.

[0300] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0301] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0302] Since the instructions stored in the computer-readable storage medium can execute the steps in any model training method or robotic arm control method provided in the embodiments of the present application, the beneficial effects that can be achieved by any model training method or robotic arm control method provided in the embodiments of the present application can be achieved. Please see the previous embodiments for details and will not be repeated here.

[0303] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method provided in various optional implementations of the above-mentioned application and the model training aspect of the robotic arm system or the control aspect of the robotic arm.

[0304] The above is a detailed introduction to a model training method, device and robotic arm system provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A model training method for a robotic arm system, characterized in that: The robotic arm system includes a master robotic arm and a slave robotic arm, wherein the master robotic arm includes N joint motors and a handle, and the slave robotic arm includes M joint motors and an end for installing an actuator, and the master robotic arm remotely controls the slave robotic arm to perform a target task. The model training method includes: Acquire first position data of the main robot arm after movement, wherein the first position data includes first angular positions of the N joint motors and first state information of the handle; Acquire second posture data of the slave robot arm after following the movement of the master robot arm and first image information of the target object, wherein the second posture data includes second angular positions of the M joint motors and second state information of the actuator, and the second state information changes when the first state information changes; Returning to the step of obtaining the first position data after the movement of the main robot arm until the target task is completed, a training data set is obtained; Filtering out a training data subset corresponding to a preset time step from the training data set, the training data subset comprising a first pose data set, a second pose data set and an image information set, the preset time step comprising at least one time step when performing the target task; Taking each time step as the current time step, filtering out the second pose data of the current time step from the second pose data set, and obtaining a target pose data set; Filtering out at least one first image information corresponding to the current time step from the image information set to obtain a target image information set corresponding to the target pose data set; Based on the target image information set and the target pose data set, a neural network model is used to predict the pose data of the slave robot at a target time step to obtain the predicted pose data of the target time step, where the target time step is the next time step of the current time step; The predicted pose data of each target time step constitute the predicted pose data set; Based on the predicted posture data set and the first posture data set, the neural network model is converged to obtain a target neural network model of the slave robot arm for the target task.

2. The model training method according to claim 1, characterized in that: The method of using a neural network model to predict the pose data of the slave robot at a target time step based on the target image information set and the target pose data set to obtain the predicted pose data of the target time step includes: Based on the target image information set and the target posture data set, using the neural network model to predict the posture data of at least one time step after the current time step to obtain predicted motion sequence data; Filtering out predicted motion sequence data corresponding to a preset number of time steps before the current time step from the historical predicted motion sequence data set to obtain the historical predicted motion sequence data; The predicted motion sequence data and the historical predicted motion sequence data are fused to obtain the predicted pose data of the target time step.

3. The model training method according to claim 2, characterized in that: The step of fusing the predicted motion sequence data with the historical predicted motion sequence data to obtain the predicted pose data of the target time step includes: Extracting the pose data of the target time step from the predicted motion sequence data to obtain current candidate pose data; Identifying the pose data of the target time step in the historical predicted motion sequence data to obtain historical candidate pose data; The current candidate pose data and the historical candidate pose data are fused to obtain the predicted pose data of the target time step.

4. The model training method according to claim 1, characterized in that: Before obtaining the first position data after the main robot arm moves, the method further includes: Initializing the slave robotic arm to a preset posture; Acquire current posture data of the master robot arm, and compare the current posture data with reference posture data corresponding to the preset posture; When the difference between the current posture data and the reference posture data does not exceed a preset difference threshold, the slave robotic arm is controlled to perform posture alignment with the master robotic arm based on the current posture data.

5. The model training method according to claim 1, characterized in that: Also includes: Extracting the first angular positions of the N joint motors of the main robot arm from the first pose data; Based on the first angular position, determining a movement speed of an actuator mounted on the slave robotic arm in a Cartesian coordinate system; When the speed component of the movement speed in the vertical direction exceeds a preset speed threshold, the slave robot arm is locked.

6. The model training method according to claim 1, characterized in that: After obtaining the first position data of the main robot arm after the movement, the method further includes: Obtaining historical position data of the main robot arm at a previous time step at the current moment; Calculating a data difference between the first posture data and the historical posture data; When the data difference exceeds a preset difference threshold, a motion abnormality prompt message is output, and the slave robotic arm is controlled to stop following the movement of the master robotic arm.

7. The model training method according to claim 1, characterized in that: After converging the neural network model based on the predicted pose data set and the first pose data set to obtain the target neural network model of the slave robot arm for the target task, the method further includes: Acquire the third positions of the M joint motors and the third image information of the target object; Based on the third angular position and the third image information, using the target neural network model to control the slave robot arm to operate on the target object to perform the target task; When a correction signal for the slave robot arm is obtained, the slave robot arm is corrected to obtain a target training data set after correction; The target neural network model is updated based on the target training data set, and the updated neural network model is used as the target neural network model.

8. The model training method according to claim 7, characterized in that: When the correction signal for the slave robot arm is obtained, the slave robot arm is corrected to obtain a corrected target training data set, including: When voice control information is detected, generating a correction signal for the slave robot arm; Based on the voice control information, determining the angular position change of the M joint motors; The slave robot arm is rectified according to the angular position change to obtain a target training data set after correction.

9. The model training method according to claim 8, characterized in that: The determining, based on the voice control information, the angular position changes of the M joint motors comprises: Recognize the voice control information based on the correction signal to obtain voice recognition content; Recognize at least one keyword in the speech recognition content, and determine the displacement change of the end of the slave robot arm based on the keyword; Acquire the current absolute coordinates of the terminal in the Cartesian coordinate system at the current time step, and adjust the current absolute coordinates based on the displacement change to obtain the target absolute coordinates; Based on the target absolute coordinates and the current absolute coordinates, the angular position changes of the M joint motors are calculated.

10. The model training method according to claim 9, characterized in that: The step of determining the displacement change of the end of the slave robot arm based on the keyword includes: Identifying a first keyword from the keywords, and searching for a second keyword from the keywords, wherein the first keyword indicates a moving direction of the terminal, and the second keyword indicates a moving distance of the terminal in the moving direction; When the keyword includes the second keyword, determining the displacement change amount of the end based on the movement direction and the movement distance; When the keyword does not include the second keyword, the movement distance corresponding to the movement direction is screened out from a preset displacement distance set, and the displacement change amount of the terminal is determined based on the movement direction and the movement distance.

11. The model training method according to claim 8, characterized in that: The step of correcting the slave robot arm according to the angular position change to obtain a corrected target training data set includes: Obtaining the predicted angular positions of the M joint motors predicted by the target neural network model at the current time step; Adding the predicted angular position to the angular position variation to obtain the target angular position of the M joint motors, and controlling the slave robot arm to operate the target object based on the target angular position; Return to the step of obtaining the third triangle positions of the M joint motors and the third image information of the target object until the target task is completed to obtain the target training data set after correction.

12. The model training method according to claim 7, characterized in that: When the correction signal for the slave robot arm is obtained, the slave robot arm is corrected to obtain a corrected target training data set, including: When the movement of the master robot arm is detected, a correction signal for the slave robot arm is generated, and the fourth angular position of the N joint motors after the movement of the master robot arm is obtained; Acquire the fifth angular position of the N joint motors before the main robot arm moves, and calculate the difference between the fourth angular position and the fifth angular position to obtain the angular position change of the M joint motors; The slave robot arm is rectified according to the angular position change to obtain a target training data set after correction.

13. A robotic arm system, characterized in that: include: A slave robot arm includes M joint motors and an end for mounting an actuator, wherein the M joint motors allow the end to have m degrees of freedom, and the actuator is used to manipulate a target object; A main robot arm, comprising N joint motors and a handle, wherein the N joint motors enable the handle to have m degrees of freedom, and the handle is configured to allow a user to remotely operate the actuator; An image acquisition device, which acquires image information containing the target object; A control device configured to execute the model training method described in any one of claims 1-12.

14. The robotic arm system according to claim 13, characterized in that: The M is equal to the m, and the N is greater than the m.

15. The robotic arm system according to claim 13, characterized in that: The handle is provided with a sensor, the sensor having a first level signal when the user holds the handle, and the sensor having a second level signal different from the first level signal when the user does not hold the handle; The control device is configured to obtain a level signal from the sensor, and lock the main robot arm when the level signal changes from the first level signal to the second level signal.

16. The robotic arm system according to claim 13, characterized in that: The robotic arm system also includes a workbench and a bracket connected to the workbench, the slave robotic arm is fixed on the workbench, the image acquisition device includes a first camera and a second camera, the first camera is fixed on the bracket and is located above the slave robotic arm, and the second camera is fixed on the end.

Images