A control method for a robotic arm and a robotic arm system

Abstract

The embodiment of the present application discloses a control method for a robotic arm and a robotic arm system; after obtaining at least one modal information of the working area where the robotic arm is located, the present application uses a task recognition model to identify at least one modal information to determine the task scenario, and then screens out a target neural network model from a preset neural network model set according to the task scenario, and controls the robotic arm to operate on at least one target object in the working area based on the target neural network model, so that the robotic arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robotic arm.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arms, and in particular to a robotic arm control method and a robotic arm system. Background Art

[0002] In recent years, with the technological breakthroughs of large models and AI, the application of general artificial intelligence (AGI) in the field of robots (or robotic arms) has become more and more extensive. Through a large number of model training for robots (or robotic arms), robots (or robotic arms) in real environments can be made intelligent, so that they can interact with the environment like humans and perform various tasks. Summary of the invention

[0003] The present application provides a method for controlling a robotic arm, including:

[0004] Acquire at least one modal information of a working area where the robot arm is located, wherein the working area allows at least one target object to be placed;

[0005] Using a task identification model to identify the at least one modal information to determine a task scenario;

[0006] According to the task scenario, a target neural network model is selected from a preset neural network model set;

[0007] Based on the target neural network model, the robotic arm is controlled to perform operation on the at least one target object.

[0008] The present application provides a robotic arm system using a robotic arm control method, including:

[0009] A robotic arm, comprising M joint motors and an end for mounting an actuator, wherein the M joint motors enable the end to have m degrees of freedom, and the actuator is used to manipulate a target object;

[0010] An image acquisition device configured to acquire image information containing a target object;

[0011] The control device is configured to obtain operating instructions for the robotic arm, determine the task scenario through the operating instructions, select a target neural network model corresponding to the task scenario from a preset neural network model set, and control the robotic arm to operate on the target object based on the target neural network model.

[0012] The beneficial effect of the present application is: after obtaining at least one modal information of the working area where the robotic arm is located, the present application uses a task recognition model to identify at least one modal information to determine the task scenario, and then screens out a target neural network model from a preset neural network model set according to the task scenario, and controls the robotic arm to operate on at least one target object in the working area based on the target neural network model, so that the robotic arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 It is a structural schematic diagram of an embodiment of a robotic arm system provided in an embodiment of the present application;

[0015] Figure 2 It is a flow chart of an embodiment of a method for controlling a robotic arm provided in an embodiment of the present application;

[0016] Figure 3 It is a structural schematic diagram of an embodiment of a control device for a robotic arm provided in an embodiment of the present application;

[0017] Figure 4 It is a structural schematic diagram of an embodiment of a control device for a robotic arm provided in an embodiment of the present application;

[0018] Figure 5 It is a structural diagram of an embodiment of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] Combination Figure 1, the robot system 100 may include a robot 110, an image acquisition device 120 and a control device 130. The robot 110 described in the present application may refer to a traditional mechanical structure fixed at a certain station to perform certain functions, such as a collaborative robot, or other emerging mechanical structures (or a part thereof), such as a humanoid robot (or its arm).

[0021] The robot arm 110 may include M joint motors and an end for mounting an actuator, wherein the M joint motors enable the end to have m degrees of freedom, and the actuator is used to operate a target object.

[0022] Among them, in this application, joint motors are generally used to enable the joints of robots (or robotic arms) to have the ability to move. Joint motors may include brushless DC motors, stepper motors, DC motors, servo motors, steering gears, etc., and the selection depends on application requirements, such as the required torque, speed, accuracy, and power efficiency. Further, the degree of freedom is the minimum number of coordinates required to determine the position of a system in space. Among them, m can be a natural number greater than zero, such as m is 4, 6 or other values. For example: the end of the robotic arm can move completely freely in the air, and at least six quantities (such as rectangular coordinates x, y, z and three angles θ, φ, ρ) are required to determine its position, that is, the end has 6 degrees of freedom in the air.

[0023] The actuator may be a gripper, a suction cup or other mechanism that can perform the target operation. The actuator may be driven or controlled by electric, pneumatic, hydraulic, etc.

[0024] The target object may be understood as an operation object under the target task performed by the robot system 100. For example, if the target task is folding clothes, the target object may include at least one piece of clothing to be folded, or if the target task is washing dishes, the target object may include at least one dish to be washed, and so on.

[0025] The image acquisition device 120 may acquire image information containing the target object. The image information may be of various types, such as RGB images, depth images, infrared images, 3D point cloud information, or images of other color spaces or types, and the like.

[0026] Among them, the control device 130 can be a server or a terminal or other device. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud storage, and cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0027] Furthermore, the control device 130 is configured to obtain operating instructions for the robotic arm, determine the task scenario through the operating instructions, and filter out a target neural network model corresponding to the task scenario from a preset neural network model set, so as to control the robotic arm to perform operations on the target object based on the target neural network model, as described in the following text for details.

[0028] Among them, the operation instruction can be understood as an instruction that triggers the robot arm to perform an operation on the target object, and the operation instruction can be generated through image information.

[0029] In some embodiments, the robot arm system 100 may further include a voice collection device, which may include a microphone or other device that can collect audio or voice information. The voice collection device may be configured to collect the user's voice information, and after the user's voice information is collected by the voice collection device, a job instruction may be generated through the voice information.

[0030] In some embodiments, in combination Figure 1 , the embodiment of the present application can provide a control method for a robot arm applied to a robot arm system 100, such as Figure 2 As shown, the specific process of the robot arm control method can be as follows:

[0031] 210. Obtain at least one modal information of the working area where the robot arm is located.

[0032] Among them, the working area allows the placement of at least one target object. The target object can be understood as an object placed in the working area to be operated by the robotic arm. For example, if an apple is placed in the working area, and the robotic arm can cut or peel the apple, then the apple is the target object. Furthermore, the modal information may include various types of images, videos, audios or texts, etc. Accordingly, the image information of the area where the robotic arm is located can be collected by an image acquisition device to obtain modal information under at least one image modality, the audio information of the area where the robotic arm is located can be collected by an audio acquisition device to obtain modal information under an audio modality, and various types of sensors can be used to collect information of the area where the robotic arm is located to obtain at least one modal information, and so on.

[0033] The method of triggering the acquisition of at least one modal information of the robot arm in the working area may include: acquiring an operation instruction for the robot arm, and in response to the operation instruction, acquiring at least one modal information of the area where the robot arm is located. The operation instruction may be understood as instruction information that triggers the robot arm to start working in the working area.

[0034] In some embodiments, the method of obtaining the operation instruction for the robot arm may include: displaying a task list, the task list includes at least one preset task, determining the task information in response to the selection operation on the task list, and generating the operation instruction based on the task information. The preset task may be a pre-set task that the robot arm can perform, for example, it may include "cooking task", "cleaning task", and "carrying task". Furthermore, there may be multiple ways to display the task list, for example, the robot arm startup page may be displayed, the startup page includes a task selection control, and in response to the trigger operation on the task selection control, the task selection interface is displayed, and the task selection interface includes the task list.

[0035] After the task list is displayed, the user can select the task list by clicking or selecting with the mouse, and then determine the task information in response to the selection operation on the task list. The task information can be understood as information containing the selected task, for example, it can include task identification, task attribute information, task start time, stop time, task preset result information or other task-related information.

[0036] In some embodiments, the method of obtaining the operation instruction for the robot arm may further include: receiving a control request for the robot arm, the control request carrying task information corresponding to the robot arm, and generating the operation instruction based on the task information. The control request may be understood as a request to control the robot arm.

[0037] Among them, there can be many ways to trigger a control request for the robotic arm in the robotic arm system. For example, corresponding trigger controls such as buttons can be set on the robotic arm system. For example, a button corresponding to the "cooking task", a button corresponding to the "cleaning task", and a button corresponding to the "carrying task" can be set. Then, when it is detected that the user presses the button corresponding to the "cleaning task", a corresponding robotic arm control request is generated, and the control request carries the task information corresponding to the cleaning task, and so on.

[0038] In some embodiments, the method of obtaining operating instructions for the robotic arm may also include: obtaining voice information for the robotic arm, performing voice recognition on the voice information to obtain task information, and generating operating instructions based on the task information.

[0039] Among them, an audio acquisition device can be installed in the robot arm system to collect voice information for the robot arm through the audio acquisition device. After obtaining the voice information for the robot arm, the voice information can be recognized by a voice recognition model or other methods to obtain task information.

[0040] 220. Use a task identification model to identify at least one modal information to determine a task scenario.

[0041] Among them, the task recognition model can understand and recognize the model of the task or scene corresponding to the target object, such as CNN, RNN, DNN, LSTM, transformer or other models that can perform task or scene recognition. Furthermore, the task scene can be a task or scene that requires operations on the target object. For example, taking the target object as a bowl, the task scene can include washing dishes, moving bowls or other tasks or scenes that operate on bowls.

[0042] There are many ways to use the task identification model to identify at least one modal information to determine the task scenario, which can be as follows:

[0043] For example, a task recognition model can be used to identify the object information of the target object in at least one modal information. The object information can be understood as information indicating the object type, object identification or other attributes of the target object. For example, taking the target object as a bowl, the object information can include the type of bowl, the identification of the bowl, the color of the bowl, the size of the bowl, the position of the bowl in the work area or other attribute information, etc. Specifically, the task recognition model can be used to recognize each modal information separately to obtain at least one initial object information of the target object, and the initial object information can be fused to obtain the object information of the target object; or, the task recognition model corresponding to each modal information can be selected from the preset task recognition model set, and the task recognition model can be used to perform object recognition on the corresponding modal information, so as to obtain the initial object information corresponding to each modal information, and the initial object information can be fused to obtain the object information of the target object, and so on.

[0044] (1) When there is only one target object, the object information can be used as the first object information, and task scenes can be screened out from a preset task scene set based on the first object information. There are multiple ways to screen out task scenes from a preset task scene set based on the first object information. For example, at least one candidate task scene corresponding to the first object information can be screened out from the preset task scene set, and the task scene corresponding to the target object can be screened out from the candidate task scenes based on the job instruction. Alternatively, the object identifier or type of the target object can be extracted from the first object information, and the task scene corresponding to the object identifier or type can be screened out from the preset task scene set as the task scene corresponding to the target object, and so on.

[0045] Among them, there may be multiple ways to filter out at least one candidate task scene corresponding to the first object information in the preset task scene set. For example, taking the first object information indicating that the target object is a bowl as an example, at least one candidate task scene related to the bowl can be filtered out in the preset task scene set. For example, it may include cleaning tasks (washing dishes), carrying tasks (placing / carrying / moving the bowl to a designated location), or placing a specific item in the bowl and other candidate task scenes.

[0046] After at least one candidate task scenario is screened out, the task scenario corresponding to the target object can be screened out from the candidate task scenarios according to the task instructions. There are many ways to screen out the task scenario corresponding to the target object from the candidate task scenarios according to the task instructions. For example, the job type of the job instruction is determined, and the task scenario corresponding to the job type is screened out from the candidate task scenarios. For example, if the target object is a bowl, if the job type is a cleaning job, washing dishes can be screened out as a task scenario from the candidate task scenarios. If the job type is a carrying job, carrying bowls can be screened out as a task scenario from the candidate task scenarios, and so on.

[0047] (2) When there are multiple target objects, the object information may be used as the second object information. Based on the second object information, the target objects are compared, and based on the comparison result and the operation instruction, the task scene is screened out from the preset task scene set.

[0048] Among them, there can be multiple ways to compare the target objects based on the second object information. For example, the current object information corresponding to each target object can be identified in the second object information, and the current object information can be compared to obtain a comparison result, which indicates whether the target objects are of the same type of object. For example, the target objects include large bowls, small bowls and plates. At this time, the large bowls and small bowls can belong to the same type of objects, but the large bowls and plates or the small bowls and plates do not belong to the same type of objects.

[0049] After the target object is compared based on the second object information, the task scene can be screened out from the preset task scene set based on the comparison result and the operation instruction. There are multiple ways to screen out the task scene from the preset task scene set based on the comparison result and the operation instruction. For example, when the target objects are objects of the same category, the task scene is screened out from the preset task scene set based on the second object information. When the target objects are objects of different categories, the candidate task scene corresponding to each target object is screened out from the preset task scene set based on the second object information, and the task scene is determined from the candidate task scene based on the operation instruction.

[0050] When the target objects are objects of the same category, a candidate task scene of any target object is screened out from a preset task scene set according to the second object information, and the candidate task scene is used as the task scene.

[0051] Among them, when the target objects are of different categories, the candidate task scenes corresponding to each target object can be screened out from the preset task scene set according to the second object information. When the operation instruction is generated based on the task information, the task scene is screened out from the candidate task scenes based on the task information. When the operation instruction is generated based on the object detection result of the operation area, the attribute information of the candidate task scene is obtained, and the candidate task scenes are sorted based on the attribute information to obtain the task scene.

[0052] Among them, there can be many ways to filter out task scenarios from candidate task scenarios based on task information. For example, the task scenario identifier or type can be identified in the task information, and the candidate task scenarios corresponding to the task scenario identifier or type can be filtered out from the candidate task scenarios to obtain the task scenario. Alternatively, the execution priority of the candidate task scenarios can be determined based on the task information, and the candidate task scenarios with the highest execution priority can be filtered out from the candidate task scenarios as the task scenarios. Alternatively, the operation order of the robot arm for each target object can be determined based on the task information, and the candidate task scenarios corresponding to the target object can be sorted based on the operation order. Based on the sorting result, the task scenarios can be filtered out from the candidate task scenarios, and so on.

[0053] The execution priority can be understood as the priority of the target task in the execution task scenario. Based on the task information, there are many ways to determine the execution priority of the candidate task scenario. For example, taking the candidate task scenarios including washing vegetables, cutting vegetables and plating as an example, when the task information indicates that vegetables need to be washed, the execution priority of washing vegetables can be set to the highest, and when the task information indicates that vegetables need to be cut, the execution priority of cutting vegetables can be set to the highest, and so on.

[0054] Among them, based on the task information, there can be many ways to determine the working order of the robot arm for the target objects. For example, taking the target objects including objects A, B and C as an example, the task information instructs the robot arm to work on object A first, then object B, and finally object C. At this time, the working order can include objects ABC.

[0055] After determining the operation sequence of the robot arm for the target object, the candidate task scenes corresponding to the target object can be sorted based on the operation sequence, and the task scenes can be screened out from the candidate task scenes based on the sorting result. For example, if the operation sequence is object ABC, the sorting result can include the candidate task scene corresponding to object A - the candidate task scene corresponding to object B - the candidate task scene corresponding to object C. At this time, the candidate task scene corresponding to object A can be used as the task scene that needs to be executed currently.

[0056] Among them, the attribute information of the candidate task scenario can be understood as information indicating at least one attribute of the candidate task scenario, for example, it can include the identification, type, execution condition, execution authority, execution priority or other attribute-related information of the candidate task scenario. Based on the attribute information, there can be multiple processes for sorting the candidate task scenarios, for example, the execution priority can be extracted from the attribute information, the candidate task scenarios can be sorted based on the execution priority, and the task scenarios can be screened out from the candidate task scenarios based on the sorting result, or the execution condition of the candidate scenario task can be determined from the attribute information, and the execution priority of the candidate scenario task can be determined based on the execution condition, the candidate task scenarios can be sorted based on the execution priority, and the task scenarios can be screened out from the candidate task scenarios based on the sorting result, and so on.

[0057] Among them, there are many ways to determine the execution priority of candidate task scenarios based on the execution conditions. For example, taking the execution condition of candidate task scenario A (cooking) as starting execution after candidate task scenario B (washing vegetables) is completed, at this time, it can be determined that the execution priority of candidate task scenario B is greater than that of candidate task scenario A, and so on, thereby determining the execution priority of each candidate task scenario, and so on.

[0058] 230. According to the task scenario, select the target neural network model from the preset neural network model set.

[0059] The preset neural network model set can be understood as a pre-set trained neural network model. The trained neural network model can be obtained by training using the above model training method.

[0060] Among them, according to the task scenario, there are many ways to filter out the target neural network model from the preset neural network model set, which can be as follows:

[0061] For example, the corresponding target task can be determined according to the task scenario, and the neural network model corresponding to the target task can be screened out from the preset neural network model set to obtain the target neural network model.

[0062] Among them, there are many ways to determine the corresponding target task according to the task scenario. For example, a task set corresponding to a preset neural network model set can be obtained, and the tasks corresponding to the task scenario can be screened out from the task set to obtain the target task.

[0063] The so-called target task can be understood as the task that the robot arm needs to perform in the task scenario. For example, if the task scenario is a dishwashing scenario, the target task corresponding to the task scenario can be dishwashing, or if the task scenario is a transporting scenario, the target task corresponding to the task scenario can be transporting the target object, and so on.

[0064] After the corresponding target task is determined, the neural network model corresponding to the target task can be screened out from the preset neural network model set. There are many ways to screen out the neural network model corresponding to the target task from the preset neural network model set. For example, the task identifier of the target task can be determined, and the neural network model corresponding to the task identifier can be screened out from the preset neural network model set to obtain the target neural network model, and so on.

[0065] In some embodiments, according to the task scenario, the method of filtering out the target neural network model from the preset neural network models may also include: determining the corresponding target task according to the task scenario, filtering out the neural network model corresponding to the target task from the preset neural network model set, obtaining the initial neural network model, obtaining the model configuration parameters corresponding to the target task, and configuring the initial neural network model based on the model configuration parameters, so as to obtain the target neural network model.

[0066] The initial neural network model can be understood as an untrained or pre-trained general neural network model for performing at least one task.

[0067] The model configuration parameters can be understood as the network parameters of the neural network model corresponding to the target task obtained after training the neural network model. There are many ways to configure the initial neural network model based on the model configuration parameters. For example, the network parameters in the initial neural network model can be replaced with the network parameters in the model configuration parameters to obtain the target neural network model.

[0068] In some embodiments, before selecting the target neural network model from the preset neural network models according to the task scenario, the neural network model can also be trained to obtain the target neural network. There are many ways to train the neural network model. For example, the master robot can remotely operate the slave robot in the real environment to perform the target task to train the neural network model corresponding to the target task to obtain the target neural network model. Alternatively, the slave robot in the virtual environment can perform the target task to train the neural network model corresponding to the target task to obtain the target neural network model. The specific method is as follows:

[0069] (1) The master robot remotely controls the slave robot in the real environment to perform the target task in order to train the neural network model corresponding to the target task.

[0070] For example, after obtaining the first pose data after the main robot arm moves, the second pose data after the slave robot arm follows the movement of the main robot arm and the first image information of the target object can be obtained, and then, the step of obtaining the first pose data after the main robot arm moves is returned to execute until the target task is completed, and a training data set is obtained. Then, the training data set is used to train the neural network model to obtain the target neural network model of the slave robot arm for the target task.

[0071] Among them, there are many ways to obtain the first posture data of the main robot arm after the movement. For example, the first posture data of the main robot arm after the movement can be directly read, or the first posture data of the main robot arm after the movement can be received from the terminal or client corresponding to the robot arm system, or the first position of the handle of the main robot arm in the Cartesian coordinate system sent by a third-party device can be received, and the first position can be inversely kinematically solved to obtain the first angular position of the N joint motors of the main machine, and the first angular position can be used as the first posture data of the main robot arm after the movement, and so on.

[0072] After obtaining the first position data after the main robot moves, the second position data of the slave robot in the real environment after following the main robot and the first image information of the target object can be obtained. There are many ways to obtain the second position data and the first image information of the target object. For example, based on the first position data, a control instruction to control the slave robot in the real environment can be generated. Based on the control instruction, the slave robot in the real environment is controlled to follow the main robot to move, and the second angular position of the slave robot after following the main robot and the second state information of the actuator are obtained. Of course, only the second angular position can be obtained to obtain the second position data. Then, the image acquisition device can be used to acquire at least one scale of the target object after the slave robot follows the main robot to obtain the first image information.

[0073] After acquiring the second posture data of the slave manipulator in the real environment and the first image information of the target object, the step of acquiring the first posture data after the movement of the master manipulator can be returned to execute until the target task is completed, and the second training data set is obtained. For example, the step of acquiring the first posture data after the movement of the master manipulator can be returned to execute until the slave manipulator completes a target task. When the stop condition is to execute a target task once, at this time, the acquisition of the first posture data after the movement of the master manipulator can be stopped, and the acquisition of the second posture data and the first image information of the slave manipulator following the movement of the master manipulator in the real environment can be stopped, and the acquired at least one first posture data, at least one second posture data and at least one first image information can be used as the training data set corresponding to the target task; when the stop condition is to execute the target task for a preset number of times, at this time, the acquisition of the first posture data after the movement of the master manipulator can be returned to execute until the target task is executed for a preset number of times, and the acquired at least one first posture data, at least one second posture data and at least one first image information can be used as the training data set corresponding to the target task.

[0074] After obtaining the training data set corresponding to the target task, the neural network model corresponding to the target task can be trained to obtain the trained target neural network model. For example, in the training data set, the neural network model is used to predict the posture data of the next time step of each time step to obtain a predicted posture data set, and the target data label corresponding to the predicted posture data in the predicted posture data set is screened out in the training data set, and the posture loss corresponding to the predicted posture data is determined based on the target data label, and the neural network model is converged based on the posture loss to obtain the target neural network model.

[0075] The predicted pose data set may include the predicted pose data of the slave robot at each time step. In this solution, the initial training data set for the target task in the training data set may be input into the neural network model, and the neural network model may output the predicted pose data set for the target task. Since the initial training data set includes pose data of multiple time steps and image information of the target object, the pose data of the slave robot at the next time step of each time step may be predicted in sequence according to the order of the time steps, thereby obtaining the predicted pose data set.

[0076] Among them, there can be many ways to filter out the target data labels corresponding to the predicted pose data in the predicted pose data set in the training data set. For example, the data label of the time step corresponding to each step-by-step action can be filtered out in the training data set as the target data label of the time step, thereby obtaining the target data label corresponding to the predicted pose data of each time step.

[0077] After filtering out the target data labels corresponding to the predicted pose data, the pose loss corresponding to the predicted pose data can be determined based on the target data labels. There are many ways to determine the pose loss corresponding to the predicted pose data. For example, the difference between the predicted pose data and the corresponding target loss can be calculated to obtain the pose loss corresponding to the predicted pose data.

[0078] After determining the pose loss corresponding to the predicted pose data, the neural network model can be converged based on the pose loss to obtain the target neural network model. There are many ways to converge the neural network model based on the pose loss. For example, a gradient descent algorithm can be used to update the network parameters of the neural network model based on the pose loss to obtain an updated neural network model, and the updated neural network model is used as the neural network model, and the step of determining the pose loss corresponding to the predicted pose data based on the target data label is returned until the neural network model converges to obtain the target neural network model. Alternatively, a momentum algorithm can be used to update the network parameters of the neural network model to obtain an updated neural network model, and the updated neural network model is used as the neural network model, and the step of determining the pose loss corresponding to the predicted pose data based on the target data label is returned until the neural network model converges to obtain the target neural network model, and so on.

[0079] (2) The target task is performed by the slave robot arm in the virtual environment to train the neural network model corresponding to the target task.

[0080] For example, a data collection page can be displayed, which includes a slave robotic arm and a target object in a virtual environment; in response to a control instruction for the slave robotic arm, the slave robotic arm is controlled to operate on the target object in the virtual environment to perform a target task; then the posture data of the slave robotic arm in the virtual environment and the image information of the target object are collected until the slave robotic arm completes the target task to obtain a training data set, which is used to train a neural network model corresponding to the target task.

[0081] There are many ways to display the data acquisition page. For example, virtual reality technology can be used to map the slave robot of the robot system in the real environment to the virtual environment to generate a virtual slave robot. At the same time, the operation object (i.e., the target object) for the robot system 100 to perform the target task can also be mapped to the virtual environment to generate a virtual target object. Based on the virtual slave robot and the virtual target object, a data acquisition page is generated and displayed.

[0082] Among them, the control instruction can be understood as the interactive instruction set by the user in the real environment for the slave robot in the virtual environment, and the object of the control instruction is the slave robot in the virtual environment. The process of the slave robot operating the target object in the virtual environment can be understood as the process of the slave robot making a series of actions for the target object under the control of the control instruction. An actuator can be set at the end of the slave robot, and the control instruction can include motion instructions and operation instructions. The motion instruction controls the movement of the slave robot in the virtual environment, and the operation instruction controls the actuator to operate the target object in the virtual environment.

[0083] After responding to the control instruction for the slave robot arm and controlling the slave robot arm to operate on the target object in the virtual environment, the posture data of the slave robot arm in the virtual environment and the image information of the target object can be collected until the slave robot arm completes the target task to obtain a training data set. For example, the posture data of the slave robot arm in the virtual environment at each time step and the image information of the target object are collected to obtain an initial training data set, and the data label corresponding to the initial training data set is obtained, and the data label is added to the initial training data set to obtain a training data set.

[0084] After obtaining the training data set, the neural network model corresponding to the target task can be trained based on the training data set, thereby obtaining the target neural network model corresponding to the target task. The process of training the neural network model based on the training data set can be seen above, and will not be repeated here.

[0085] 240. Based on the target neural network model, control the robotic arm to operate on at least one target object.

[0086] For example, the posture data of the robotic arm and the image information of at least one target object can be obtained. Based on the posture data and the image information, the target neural network model is used to predict the posture data of the robotic arm at the next time step at the current moment to obtain the predicted posture data. Based on the predicted posture data, a control instruction for the robotic arm is generated. According to the control instruction, the robotic arm is controlled to perform an operation on at least one target object, and the step of obtaining the posture data of the robotic arm and the image information of at least one target object is returned to execute until the preset stop condition is reached to execute the target task in the task scenario.

[0087] The robot arm may include M joint motors and an end for mounting an actuator, the M joints enable the end to have m degrees of freedom, and the actuator is used to operate a target object. There are many ways to obtain the posture data of the robot arm and the image information of at least one target object. For example, the angular positions of the M joint motors and the state information of the actuator can be read to obtain the posture data of the robot arm, and the image information of at least one target object can be collected by an image acquisition device.

[0088] After acquiring the posture data of the robot and the image information of at least one target object, the target neural network model can be used to predict the posture data of the robot at the next time step at the current moment based on the posture data and image information, thereby obtaining the predicted posture data. There are many ways to use the target neural network model to predict the posture data of the robot at the next time step at the current moment. For example, the next time step at the current moment can be used as the target time step, and based on the posture data and image information, the target neural network model can be used to predict the posture data of at least one time step after the current time step corresponding to the current moment to obtain the predicted motion sequence data, and the predicted motion sequence data corresponding to the preset number of time steps before the current time step can be screened out from the historical predicted motion sequence data to obtain the historical predicted motion sequence data, and the predicted motion sequence data and the historical motion sequence data can be fused to obtain the predicted posture data of the target time step.

[0089] Among them, there can be multiple ways to use the target neural network model to predict the pose data of at least one time step after the current time step corresponding to the current moment based on the pose data and image information. For example, the target neural network model can be used to extract features from the pose data to obtain second style variable features, perform multi-dimensional feature extraction on the image information to obtain second image features of at least one dimension, fuse the second image features to obtain second target image features, and predict the pose data of at least one time step after the current time step based on the pose data, the second style variable features and the second target image features, thereby obtaining predicted motion sequence data.

[0090] Among them, there are many ways to fuse the predicted motion sequence data and the historical motion sequence data. For example, the pose data of the target time step can be extracted from the predicted motion sequence data to obtain the first candidate pose data, and the pose data of the target time step can be extracted from the historical predicted motion sequence data to obtain the second candidate pose data. The first candidate pose data and the second candidate pose data can be fused to obtain the predicted pose data of the target time step.

[0091] Among them, there are many ways to fuse the first candidate pose data and the second candidate pose data. For example, the pose data fusion weight can be obtained, and based on the pose data fusion weight, the first candidate pose data and the second candidate pose data are weighted respectively to obtain the weighted first pose data and the weighted second pose data, and the weighted first pose data and the weighted second pose data are added together to obtain the predicted pose data of the target time step.

[0092] After the target neural network model is used to predict the posture data of the robot arm at the next time step at the current moment based on the posture data and image information, a control instruction for the robot arm can be generated based on the predicted posture data. The control instruction may include motion instructions and operation instructions. The motion instructions control the robot arm to move, and the operation instructions control the actuator to operate or work on the target object. There are many ways to generate control instructions for the robot arm based on the predicted posture data. For example, the predicted angular position of M joint motors and the predicted state information of the actuator can be extracted from the predicted posture data. Based on the predicted angular position and the predicted state information, motion instructions for the M joint motors and operation instructions for the actuator are generated respectively. The motion instructions and operation instructions are used as control instructions for the robot arm, thereby generating control instructions for the robot arm.

[0093] After generating a control instruction for the robot arm based on the predicted posture data, the robot arm can be controlled to operate on at least one target object according to the control instruction. There are many ways to control the robot arm to operate on at least one target object according to the control instruction, such as controlling the movement of M joint motors of the robot arm through motion instructions, and controlling the actuator to operate on at least one target object through operation instructions to operate on at least one target object.

[0094] After controlling the robotic arm to operate on at least one target object according to the control instructions, the process can return to the step of acquiring the posture data of the robotic arm and the image information of at least one target object until the preset stop condition is reached to complete the target task in the task scenario.

[0095] The preset stop condition can be understood as a preset condition for stopping the robot arm from operating at least one target object. There can be multiple preset stop conditions, for example, the target object reaches a preset state or a specified position, or the number of operations of the robot arm on at least one target object reaches a preset number, or other preset stop conditions, etc.

[0096] From the above, it can be seen that after obtaining at least one modal information of the working area where the robotic arm is located, the present application uses a task recognition model to identify at least one modal information to determine the task scenario, and then screens out a target neural network model from a preset set of neural network models according to the task scenario. Based on the target neural network model, the robotic arm is controlled to operate on at least one target object in the working area, so that the robotic arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robotic arm.

[0097] An embodiment of the present application also provides a control device for a robotic arm, which can be integrated into an electronic device, such as a server or a terminal, and the terminal may include a tablet computer, a laptop computer and / or a personal computer, etc.

[0098] For example, Figure 3 As shown, the control device of the robot arm has a display interface, which can display the acquisition module 310, the identification module 320, the screening module 330 and the control module 340, which can be specifically as follows:

[0099] (1) An acquisition module 310 , which has at least one line of code and is used to acquire at least one modal information of a working area where the robot arm is located, and the working area allows at least one target object to be placed.

[0100] (2) An identification module 320 , which has at least one line of code and is used to identify at least one modal information using a task identification model to determine a task scenario.

[0101] (3) A screening module 330 , which has at least one line of code and is used to screen out a target neural network model from a preset set of neural network models according to a task scenario.

[0102] (4) A control module 340 having at least one line of code and configured to control the robot arm to operate on at least one target object based on the target neural network model.

[0103] In some embodiments, the robot arm control device may further include a training unit 350 having at least one line of code and used to train a neural network model corresponding to a target task to obtain a target neural network. Figure 4 shown.

[0104] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0105] From the above, it can be seen that in this application, after the acquisition module 310 acquires at least one modal information of the working area where the robot arm is located, the identification module 320 uses the task identification model to identify at least one modal information to determine the task scenario, and then the screening module 330 filters out the target neural network model from the preset neural network model set according to the task scenario, and the control module 340 controls the robot arm to operate on at least one target object in the working area based on the target neural network model; so that the robot arm can select the corresponding target neural network model in different task scenarios, thereby realizing autonomous and reliable automatic operation, thereby improving the operating efficiency and reliability of the robot arm.

[0106] The present application also provides an electronic device, such as Figure 5 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0107] The electronic device may include components such as a processor 410 with one or more processing cores, a memory 420 with one or more computer-readable storage media, a power supply 430, and an input unit 440. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0108] The processor 410 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device, and performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 420, and calling data stored in the memory 420.

[0109] The memory 420 can be used to store software programs and modules, and the processor 410 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc.

[0110] The power supply 430 may be logically connected to the processor 410 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system.

[0111] The input unit 440 may be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0112] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 410 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 420 according to the following instructions, and the processor 410 will run the application programs stored in the memory 420, thereby realizing various functions, as follows:

[0113] After obtaining at least one modal information of the working area where the robotic arm is located, the working area allows at least one target object to be placed, and the at least one modal information is identified using a task recognition model to determine the task scenario. Then, according to the task scenario, a target neural network model is screened out from a preset neural network model set, and based on the target neural network model, the robotic arm is controlled to operate on at least one target object.

[0114] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0115] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0116] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the control methods of a robotic arm provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0117] After obtaining at least one modal information of the working area where the robotic arm is located, the working area allows at least one target object to be placed, and the at least one modal information is identified using a task recognition model to determine the task scenario. Then, according to the task scenario, a target neural network model is screened out from a preset neural network model set, and based on the target neural network model, the robotic arm is controlled to operate on at least one target object.

[0118] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0119] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0120] Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the robot arm control methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the robot arm control methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0121] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method provided in various optional implementations of the above-mentioned application and the robot arm control aspect of the robot arm system.

[0122] The above is a detailed introduction to a robot arm control method and a robot arm system provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for controlling a robotic arm, characterized in that: include: Acquire at least one modal information of a working area where the robot arm is located, wherein the working area allows at least one target object to be placed; Using a task recognition model to recognize the at least one modal information to determine a task scenario, wherein the task scenario is a scenario in which an operation needs to be performed on the target object; Obtain a task set corresponding to a preset neural network model set, filter out tasks corresponding to the task scenario from the task set, and obtain a target task, where the target task is a task that the robotic arm needs to perform in the task scenario; Filtering out a neural network model corresponding to the target task from a preset neural network model set to obtain a target neural network model, wherein the target neural network model is trained based on a training data set corresponding to the target task; Based on the target neural network model, the robotic arm is controlled to operate on the at least one target object to perform the target task.

2. The control method according to claim 1, characterized in that: The obtaining of at least one modal information of the working area where the robot arm is located includes: Obtaining an operation instruction for the robotic arm; In response to the operation instruction, at least one modal information of the area where the robot arm is located is obtained.

3. The control method according to claim 2, characterized in that: The obtaining of the control instruction for the robotic arm comprises: Display a task list, wherein the task list includes at least one preset task; In response to a selection operation on the task list, determining task information; Based on the task information, the work instruction is generated.

4. The control method according to claim 2, characterized in that: The obtaining of the operation instruction for the robot arm comprises: Acquiring voice information for the robotic arm; Performing speech recognition on the speech information to obtain task information; Based on the task information, the work instruction is generated.

5. The control method according to claim 2, characterized in that: The step of using a task identification model to identify the at least one modal information to determine a task scenario includes: Using the task identification model, identifying object information of the target object in the at least one modal information; When there is only one target object, taking the object information as first object information, and filtering out the task scene from a preset task scene set according to the first object information; When there are multiple target objects, the object information is used as second object information, the target objects are compared based on the second object information, and the task scene is screened out from a preset task scene set based on the comparison result and the operation instruction.

6. The control method according to claim 5, characterized in that: The step of selecting the task scenario from a preset task scenario set based on the comparison result and the operation instruction includes: When the target objects are of the same type, the task scene is selected from a preset task scene set according to the second object information; When the target objects are objects of different categories, candidate task scenes corresponding to each target object are screened out from a preset task scene set according to the second object information, and the task scene is determined from the candidate task scenes based on the work instruction.

7. The control method according to claim 6, characterized in that: The step of determining the task scenario from the candidate task scenarios based on the job instruction includes: When the job instruction is generated based on the task information, the task scenario is screened out from the candidate task scenarios based on the task information; When the work instruction is generated based on the object detection result of the work area, the attribute information of the candidate task scenes is obtained, and based on the attribute information, the candidate task scenes are sorted to obtain the task scene.

8. The control method according to claim 1, characterized in that: The controlling the robot arm to operate the at least one target object based on the target neural network model to perform the target task includes: Acquire the position data of the robotic arm and the image information of the at least one target object; Based on the posture data and image information, the target neural network model is used to predict the posture data of the robotic arm at the next time step at the current moment to obtain predicted posture data; Based on the predicted posture data, generating a control instruction for the robotic arm; According to the control instruction, controlling the robotic arm to operate on the at least one target object; Return to the step of acquiring the posture data of the robotic arm and the image information of the at least one target object until a preset stop condition is reached, so as to complete the target task in the task scenario.

9. A robotic arm system, characterized in that: The robotic arm system is used to execute the control method according to any one of claims 1 to 8, and the robotic arm system comprises: A robotic arm, comprising M joint motors and an end for mounting an actuator, wherein the M joint motors enable the end to have m degrees of freedom, and the actuator is used to operate a target object; An image acquisition device, configured to acquire image information containing the target object; The control device is configured to obtain an operation instruction for the robotic arm, determine a task scenario through the operation instruction, and select a target neural network model corresponding to the task scenario from a preset neural network model set, so as to control the robotic arm to perform an operation on the target object based on the target neural network model.

10. The robotic arm system according to claim 9, characterized in that: The operation instruction is generated by the image information; or, The robot arm system further includes a voice collection device, which is configured to collect voice information of a user, and the operation instruction is generated through the voice information.

Images