Robot control method, robot and system

Abstract

The invention belongs to the technical field of robots, and discloses a robot control method, a robot and a system.The robot comprises a master control steering engine, m slave control mechanical arms and a shooting device, the shooting device is arranged on at least one slave control mechanical arm, and the method comprises the steps that environment image data shot by the shooting device are obtained; the environment image data are input into a target model to be processed, respective control instructions of the m slave control mechanical arms are obtained, the target model comprises a first network and a second network, the first network is used for calculating joint movement data corresponding to joints in a target mechanical arm, and the second network is used for calculating joint movement data corresponding to the joints in the target mechanical arm; the target mechanical arm is at least one of the m slave control mechanical arms; and the second network is used for performing mapping processing based on the joint movement data corresponding to the joints in the target mechanical arm to obtain respective control instructions of the m slave control mechanical arms. In this way, the accuracy and efficiency of robot control can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot control method, a robot and a system. Background Art

[0002] With the continuous development of computer technology, sensor technology and bionics technology, robots are being used more and more widely. For example, they are used in logistics, reception, guidance, business handling, teaching and scientific research, etc. Usually, a robot body is equipped with a mechanical arm, which is responsible for tasks such as object grasping and object movement.

[0003] In practical applications, the control of the robot's manipulator arm directly affects the robot's operating performance, so it is necessary to know how to better control the manipulator arm. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present application provides a robot control method, a robot and a system, which can improve the accuracy and efficiency of robot control.

[0005] According to a first aspect of the present application, a robot control method is provided, wherein the robot comprises a master control steering engine, m slave control mechanical arms and a photographing device, wherein the photographing device is arranged on at least one of the slave control mechanical arms, and the m slave control mechanical arms are controlled by the master control steering engine, where m is a positive integer, and the method comprises:

[0006] Acquire environmental image data captured by the shooting device, where the environmental image data is used to reflect the target scene in which the robot is located and / or the target task to be performed;

[0007] The environmental image data is input into the target model for processing to obtain control instructions for each of the m slave-controlled robotic arms, and the control instructions are used to control joint motion data corresponding to each joint in the slave-controlled robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave-controlled robotic arms; the second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions for each of the m slave-controlled robotic arms.

[0008] Optionally, the target robotic arm includes a reference robotic arm, the second network includes a joint mapping relationship of the m slave robotic arms relative to the reference robotic arm, the joint mapping relationship is used to indicate a mapping relationship between joint motion data corresponding to each joint in the reference robotic arm and joint motion data corresponding to each joint in the slave robotic arm when having the same base center coordinates, and the mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions for each of the m slave robotic arms includes:

[0009] Based on the joint mapping relationship of the m slave-controlled robotic arms relative to the reference robotic arm, data conversion is performed on the joint motion data corresponding to each joint in the reference robotic arm to obtain the joint motion data corresponding to each joint in the m slave-controlled robotic arms;

[0010] Based on the joint motion data corresponding to each joint in the m slave-controlled robotic arms, corresponding control instructions for each of the m slave-controlled robotic arms are generated.

[0011] Optionally, the target robotic arm includes the m slave robotic arms, the second network includes joint mapping relationships of the m slave robotic arms, the joint mapping relationships are used to indicate mapping relationships of joint motion data corresponding to each joint in the slave robotic arms when having the same base center coordinates, and the mapping processing is performed based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions of each of the m slave robotic arms, including:

[0012] Based on the joint mapping relationship of each of the m slave-controlled robotic arms, data conversion is performed on the joint motion data corresponding to each joint in the m slave-controlled robotic arms to obtain the converted joint data corresponding to each joint in the m slave-controlled robotic arms;

[0013] Based on the conversion joint data corresponding to each joint in the m slave-controlled robotic arms, control instructions for each of the m slave-controlled robotic arms are generated accordingly.

[0014] Optionally, the method further comprises:

[0015] Acquire a first training set, wherein the first training set includes sample environment data captured by the camera, and control motion data corresponding to each joint of the m slave mechanical arms when the master steering engine performs a sample task;

[0016] The first network to be trained is trained based on the first training set to obtain the trained first network.

[0017] Optionally, the method further comprises:

[0018] Acquire a second training set, where the second training set at least includes the base center coordinates corresponding to each of the m slave-controlled robotic arms when the sample task is performed;

[0019] The second network to be trained is trained based on the second training set to obtain the trained second network.

[0020] Optionally, the first training set and / or the second training set is a preprocessed training set, and the preprocessing includes at least one of data cleaning, standardization, and outlier processing.

[0021] Optionally, the method further comprises:

[0022] The m slave-controlled robotic arms are controlled to synchronously execute respective control instructions of the m slave-controlled robotic arms to complete the target task.

[0023] Optionally, the image data also includes depth of field data; and / or the m slave-controlled robotic arms are heterogeneous robotic arms.

[0024] According to the second aspect of the present application, a robot is provided, comprising a master servo, m slave robotic arms and a photographing device, wherein the photographing device is arranged on at least one of the slave robotic arms, and the m slave robotic arms are controlled by the master servo, m is a positive integer, and the robot further comprises: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the above-mentioned robot control method.

[0025] According to a third aspect of the present application, there is provided a robot control system, comprising a master control steering engine, m slave control robotic arms and a photographing device, wherein the photographing device is arranged on at least one of the slave control robotic arms, the m slave control robotic arms are controlled by the master control steering engine, and m is a positive integer;

[0026] After the m slave-controlled robotic arms are powered on, the shooting devices provided on the slave-controlled robotic arms are powered on;

[0027] When the m slave-controlled robotic arms support communication, the shooting device is started to shoot corresponding environmental image data, where the environmental image data is used to reflect the target scene in which the robot is located and / or the target task to be performed;

[0028] The environmental image data is input into the target model for processing to obtain control instructions for each of the m slave-controlled robotic arms, and the control instructions are used to control joint motion data corresponding to each joint in the slave-controlled robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave-controlled robotic arms; the second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions for each of the m slave-controlled robotic arms.

[0029] For the contents not introduced or described in the embodiments of the present application, please refer to the relevant introduction in the aforementioned method embodiments, and the embodiments of the present application are not limited thereto.

[0030] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0031] (1) The robot can directly use the target network to process the environmental image data captured by the shooting device, and obtain the control instructions of the m slave mechanical arms in the robot, thereby controlling the joint motion data corresponding to each joint in the corresponding slave mechanical arms to complete the corresponding target task. In this way, the efficiency and accuracy of robot control can be improved.

[0032] (2) The present application can use the first network to first calculate the joint motion data corresponding to each joint in a reference robotic arm; then use the joint mapping relationship of the m slave robotic arms relative to the reference robotic arm in the second network to map the joint motion data corresponding to each joint in the reference robotic arm, thereby obtaining the joint motion data corresponding to each joint in the m slave robotic arms, and using this to generate control instructions for each of the m slave robotic arms. In this way, the joint motion data corresponding to each joint in other slave robotic arms can be mapped and converted based on the joint motion data corresponding to each joint in the reference robotic arm, and then the control instructions for the corresponding slave robotic arms can be generated; the resource consumption of the entire process can be reduced, and it is also beneficial to improve the accuracy of robot control.

[0033] (3) The present application can use the first network to first calculate the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms; then use the joint mapping relationship of each of the above-mentioned m slave-controlled robotic arms in the second network to map the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms, so as to obtain the conversion motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms, and use this to generate the control instructions of each of the above-mentioned m slave-controlled robotic arms. In this way, the first network can be used to roughly calculate the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms, and then the above-mentioned m joint mapping relationship can be used to perform more accurate mapping transformation calculations, which is conducive to improving the convenience and accuracy of robot control.

[0034] (4) The present application uses the acquired first training set to train the first network to be trained, thereby obtaining a trained first network. Using a small sample training set to train the first network not only reduces the requirement for the amount of sample training data, but also overcomes the limitations of the prior art that relies on a large amount of sample training data in scenarios where data collection is difficult or costly, and is also conducive to improving the performance and accuracy of robot control.

[0035] (5) The present application uses the acquired second training set to train the second network to be trained, thereby obtaining a trained second network. Using a small sample training set to train the second network not only reduces the requirement for the amount of sample training data, but also overcomes the limitations of the prior art that relies on a large amount of sample training data in scenarios where data collection is difficult or costly, and is also conducive to improving the performance and accuracy of robot control.

[0036] (6) The present application uses the pre-processed first training set and / or second training set to train the corresponding network, which is beneficial to improving the precision or accuracy of network training, and is also beneficial to improving the accuracy and reliability of subsequent robot control.

[0037] (7) The present application synchronously controls the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms based on the control instructions of each of the above-mentioned m slave-controlled robotic arms, so as to ensure the synchronization of the robot movements and complete the corresponding target tasks, thereby facilitating improving the synchronization and accuracy of the robot control.

[0038] (8) This application improves the robot's operating accuracy in complex environments by combining image depth data, enhancing the robot's perception of the surrounding environment / scene, and improving the robot's accuracy in identifying and locating objects. And / or, using one master and multiple slave heterogeneous robots to strengthen learning training is conducive to improving the robot's applicability and generalization in parallel use and heterogeneous environments.

[0039] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 It is a schematic structural diagram of a robot control system according to an exemplary embodiment.

[0042] Figure 2 It is a schematic structural diagram of another robot control system according to an exemplary embodiment.

[0043] Figure 3 It is a flowchart of a robot control method according to an exemplary embodiment.

[0044] Figure 4 It is a schematic diagram of the internal structure of a target model according to an exemplary embodiment.

[0045] Figure 5 It is a schematic diagram of a process of first network training according to an exemplary embodiment.

[0046] Figure 6 It is a schematic diagram of a process of second network training according to an exemplary embodiment.

[0047] Figure 7 It is a schematic structural diagram of a robot control device according to an exemplary embodiment.

[0048] Figure 8 It is a schematic diagram of the structure of a robot according to an exemplary embodiment.

[0049] Fig. 9 The figure is a schematic diagram showing the structure of a chip according to an exemplary embodiment. DETAILED DESCRIPTION

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0051] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0052] Traditional robot motion training methods usually rely on a large amount of sample data, which has limitations in scenarios such as data collection being difficult or costly. The overfitting problem in training with small sample data is also an important problem and challenge that is currently difficult to solve.

[0053] At present, the robot control scheme based on neural network usually relies on color (Red Green Blue, RGB) images and joint position information, and lacks the understanding of scene depth. This causes the robot to be unable to accurately judge the distance and spatial relationship of objects when dealing with complex environments, thus affecting the accuracy of robot control operations. For example, it is easily disturbed under low light or strong light conditions, resulting in reduced accuracy of recognition and positioning. For example, when the object is partially obscured, the traditional scheme cannot effectively identify and locate the obscured object. In the traditional robot control scheme based on neural network, a large amount of labeled data is often required to improve the model performance of the network during the neural network training process. The lack of depth information makes it difficult for the network model to capture important spatial features during the learning process, resulting in a low learning rate. In addition, in the traditional robot control scheme, the robot is only equipped with a set of master-slave structures, and the structure of the slave robot arm is often consistent or the same. There are great application limitations in the application scenarios / environments where heterogeneous robots of various models coexist and work together.

[0054] To solve the above problems, this application proposes a robot control method, a robot and a system. Figure 1 FIG. 1 is a schematic diagram showing a structure of a robot control system according to an exemplary embodiment. Figure 1 The robot control system 10 (also referred to as the robot 10) shown may include: a master steering engine 100, m slave robotic arms (represented as 201 to 20m in the figure) and a camera 300. The present application does not limit the number of the master steering engine 100 and the slave robotic arms. The master steering engine 100 may be one or more, and the slave robotic arms may be m, where m is a positive integer that is customized and configured according to actual conditions.

[0055] The master control servo 100 may be responsible for controlling the m slave control manipulators, for example, controlling the joint motion data of each joint in the slave control manipulator, so as to complete the corresponding target task. The target task is the task that the robot wants to perform, which may include but is not limited to, for example, object grasping, handling, assembly, welding, stacking, spraying, grinding or other custom tasks. The joint motion data may refer to data used to describe joint motion, which may include but is not limited to, for example, joint motion angle, joint motion speed, joint motion acceleration or other motion data. In a specific implementation, the master control servo 100 may receive manually input control information, drive each joint of the slave control manipulator, and make it perform the corresponding task according to a predetermined trajectory or action. The present application does not limit the specific implementation method of the above control information. For example, if the master control servo 100 is a remote control servo, the present application may be controlled by a corresponding remote control signal; if the master control servo 100 is a Bluetooth servo, the present application may be controlled by a corresponding Bluetooth signal, etc.

[0056] The above-mentioned shooting device 300 is used to collect corresponding environmental image data, for example, collecting at a set frequency, etc. The set frequency is customized by the system or the user according to actual needs, such as customized according to user preferences, etc. The present application does not limit the type of the above-mentioned shooting device 300. For example, it can be a general depth camera, or other devices for image or video shooting, etc. The above-mentioned environmental image data can include the current environmental data of the robot, and can also include depth of field data. The depth of field data is used to reflect the range of clear objects in the above-mentioned environmental image data, for example, it is used to indicate the distance between the farthest point and the nearest point in the above-mentioned environmental image data. The present application does not limit the installation position of the above-mentioned shooting device 300. Specifically, the above-mentioned shooting device 300 can be set on at least one slave-controlled robotic arm, for example, it can be set on the wrist of a slave-controlled robotic arm. Please refer to Figure 2 FIG. 1 is a schematic diagram of another robot structure according to an exemplary embodiment. Figure 2 The robot control system 10 shown includes two master servos, which are shown as 101 and 102. Each master servo controls two slave manipulators. In the figure, the master servo 101 controls the slave manipulator 201 and the slave manipulator 202. The master servo 102 controls the slave manipulator 203 and the slave manipulator 204. A shooting device 301 is provided at the wrist of the slave manipulator 201, and a shooting device 302 is provided at the wrist of the slave manipulator 203. The shooting device 301 is used to periodically or in real time capture the image data of the environment in which the slave manipulator 201 is located. The shooting device 302 is used to periodically or in real time capture the image data of the environment in which the slave manipulator 203 is located. Figure 2 The components included in the robot are arranged symmetrically, such as left-right symmetrical arrangement as shown in the figure.

[0057] This application does not limit the models or types of the m slave robotic arms. For example, the m slave robotic arms can be isomorphic robotic arms with the same structure or heterogeneous robotic arms with different models / structures. This application does not make too many limitations or details. In practical applications, the master servo 100, the m slave robotic arms and the camera 300 can be arranged on the same horizontal flat plate. Figure 2 As shown, the master servo 101, the master servo 102, the slave mechanical arm 201, the slave mechanical arm 202, the slave mechanical arm 203, the slave mechanical arm 204, the shooting device 301 and the shooting device 302 can be arranged on the same horizontal flat plate. The bottom of each of the master servo 100 and the slave mechanical arm can be provided with a base (also referred to as a chassis). The base of the master servo 100 can bear the basic functions of the master servo 100, such as walking, movement and obstacle avoidance. The base of the slave mechanical arm can bear the basic functions of the slave mechanical arm, such as walking, movement and obstacle avoidance.

[0058] Based on the above examples, see Figure 3 FIG. 1 is a flow chart of a robot control method according to an exemplary embodiment. Figure 3 The method shown can be applied to Figure 1 or Figure 2 In the robot control system 10 shown, the method may include the following implementation steps:

[0059] S301. Acquire environmental image data captured by the shooting device, where the environmental image data is used to reflect the target scene in which the robot is located and / or the target task to be performed.

[0060] In the present application, the above-mentioned environmental image data may refer to the environmental image obtained by shooting the above-mentioned shooting device 300, which is used to reflect the target scene currently located in the above-mentioned system and / or the target task to be performed. The target scene may refer to the environmental scene in which the robot is located, which may include but is not limited to scenes such as object handling and welding. The above-mentioned target task corresponds to the above-mentioned target scene, which may refer to the task waiting to be executed in the above-mentioned target scene, which may include but is not limited to, for example, object grasping, handling, assembly, welding, palletizing, spraying, grinding or other custom tasks.

[0061] The present application does not limit the implementation method for obtaining the above-mentioned environmental image data. For example, the above-mentioned environmental image data can be directly obtained by photographing and collecting through the above-mentioned photographing device 300; or the environmental image data obtained by the above-mentioned photographing device 300 can be obtained from other devices (such as storage devices, or other terminals) through the network. For example, due to limited storage space capacity, the above-mentioned photographing device 300 stores the photographed environmental image data in an external memory, etc.

[0062] S302. Input the environmental image data into the target model for processing to obtain control instructions for each of the m slave-controlled robotic arms, wherein the control instructions are used to control joint motion data corresponding to each joint in the slave-controlled robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave-controlled robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions for each of the m slave-controlled robotic arms.

[0063] The above-mentioned slave-controlled mechanical arm may include one or more joints, for example, the Kuka robot may include 6 joints, and the joint motion data corresponding to the 6 joints may be expressed as (J1, J2, J3, J4, J5, J6). The joint motion angle of each joint has a certain preset angle range, which determines the flexibility and operability of the robot. The preset angle range is pre-defined by the system or R&D personnel, such as the experience range set according to the experience of the R&D personnel, and the statistical range obtained by calculating a series of experimental data, etc., and this application does not make too many restrictions and limitations on this.

[0064] The target model may refer to a model for predicting the joint motion data of each joint in the robot arm, or predicting the control instructions of each joint in the robot arm. The model may include, but is not limited to, for example, a neural network model, a grayscale prediction model, a regression prediction model, a support vector regression (SVR) model, or other models for robot motion prediction.

[0065] By implementing the embodiment of the present application, the robot includes a master control servo, m slave control mechanical arms and a shooting device, wherein the shooting device is arranged on at least one of the slave control mechanical arms, and the m slave control mechanical arms are controlled by the master control servo, and m is a positive integer. The robot obtains the environmental image data shot by the shooting device, and the environmental image data is used to reflect the target scene and / or the target task to be performed by the robot; the environmental image data is input into the target model for processing, and the control instructions of the m slave control mechanical arms are obtained, and the control instructions are used to control the joint motion data corresponding to each joint in the slave control mechanical arm, and the target model includes a first network and a second network, and the first network is used to calculate the joint motion data corresponding to each joint in the target mechanical arm, and the target mechanical arm is at least one of the m slave control mechanical arms; the second network is used to map the joint motion data corresponding to each joint in the target mechanical arm, and obtain the control instructions of the m slave control mechanical arms. In this way, directly using the target model to control the m slave control mechanical arms is conducive to improving the efficiency and accuracy of robot control.

[0066] In step S302, the present application does not limit the internal structure of the target model. Figure 4 FIG. 1 is a schematic diagram showing the internal structure of a target model according to an exemplary embodiment. Figure 4 The target model shown may include a first network 401 and a second network 402. The first network 401 is mainly used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is one or more of the m slave-controlled robotic arms. The second network 402 is mainly used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm, so as to obtain the control instructions of each of the m slave-controlled robotic arms. The present application does not limit the internal structures of the first network 401 and the second network 402. For example, the first network 401 and the second network 402 may be neural networks, deep learning networks or other custom function networks.

[0067] The present application does not limit the specific implementation of the above step S302. The present application can use the trained first network to perform inference calculations on the above environmental image data, so as to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is part or all of the m slave-controlled robotic arms. The trained second network is further used to perform mapping processing on the target robotic arm, so as to obtain the control instructions of each of the m slave-controlled robotic arms. The specific implementation of the above inference calculation depends on the internal structure of the above first network. For example, when the first network is a convolutional neural network, the present application can use the first network to perform corresponding convolutional neural network processing on the above environmental image data, etc. The present application does not make too many limitations and details on this.

[0068] The present application does not limit the specific implementation methods of the above-mentioned mapping process. For example, in one implementation, the above-mentioned target robotic arm may include a reference robotic arm. The above-mentioned second network includes the joint mapping relationships of the above-mentioned m slave robotic arms relative to the above-mentioned reference robotic arm. Based on the above-mentioned m joint mapping relationships, the present application can perform data conversion on the joint motion data corresponding to each joint in the above-mentioned reference robotic arm, so as to obtain the converted motion data corresponding to each joint in the above-mentioned m slave robotic arms, which is the joint motion data corresponding to each joint in the slave robotic arm. Furthermore, the present application can generate corresponding control instructions for each of the m slave robotic arms based on the joint motion data corresponding to each joint in the above-mentioned m slave robotic arms. The control instruction is used to control each joint in the corresponding slave robotic arm to move according to the corresponding joint motion data, so as to complete the corresponding task.

[0069] The above-mentioned joint mapping relationship is used to indicate the mapping relationship between the joint motion data corresponding to each joint in the above-mentioned reference robot arm and the joint motion data corresponding to each joint in the above-mentioned slave robot arm when they have the same base center coordinates. That is, it indicates the mapping relationship between the joint motion data corresponding to each joint in the above-mentioned m slave robot arms and the joint motion data corresponding to each joint in the above-mentioned reference robot arm when the above-mentioned m slave robot arms have the same base center coordinates. For example, combined with Figure 2In the robot shown, when performing the same target task, the base center coordinates of the slave-controlled mechanical arm 201 and the slave-controlled mechanical arm 202 need to be moved to the same position coordinates, for example (x1, y1, z1), assuming that the joint motion data corresponding to each joint in the slave-controlled mechanical arm 201 is (J1, J2, J3, J4, J5, J6), and the joint motion data corresponding to each joint in the slave-controlled mechanical arm 202 is (J7, J8, J9, J10, J11, J12). In this example, the above-mentioned joint mapping relationship can refer to the mapping relationship between (J1, J2, J3, J4, J5, J6) and (J7, J8, J9, J10, J11, J12).

[0070] In another embodiment, the target manipulator may include m slave manipulators, and the second network may include the joint mapping relationships of the m slave manipulators, and may specifically include the joint mapping relationships of the m slave manipulators relative to preset reference points. The preset reference point is a reference point that is pre-defined and set by the system according to actual conditions, such as any reference point selected on a horizontal plate, etc., and this application does not make too many restrictions and details. Based on the m joint mapping relationships, the application may perform data conversion on the joint motion data corresponding to each joint in the m slave manipulators, thereby obtaining the conversion motion data corresponding to each joint in the m slave manipulators, which is the joint motion data corresponding to each joint in the slave manipulators. Further, the application may generate control instructions corresponding to each of the m slave manipulators based on the conversion motion data corresponding to each joint in the m slave manipulators. The control instruction is used to control each joint in the corresponding slave manipulator to move according to the corresponding joint motion data, thereby completing the corresponding task. In this embodiment, the first network can be used to roughly calculate the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms, and then the above-mentioned m joint mapping relationships can be used to perform more accurate mapping transformation calculations, which is conducive to improving the convenience and accuracy of robot control.

[0071] Before executing the above step S302, the present application also needs to train the above target model. The training process can be applied to a training device, which may include but is not limited to, for example, the above robot, terminal device, server, or other device with model training capabilities. Taking the above target model including the first network 401 and the second network 402 as an example, the following introduces the training embodiments involving the two networks respectively.

[0072] See also Figure 5 FIG. 1 is a flow chart showing a first network training process according to an exemplary embodiment. Figure 5 The process shown may include the following implementation steps:

[0073] S501, obtaining a first training set, wherein the first training set includes sample environment data photographed by the photographing device, and control motion data corresponding to each joint in the m slave robotic arms when the master servo performs a sample task.

[0074] In the present application, the first training set may include the sample environmental data photographed by the shooting device 300, and the control motion data corresponding to each joint in the m slave robotic arms when the master servo 100 performs the sample task. For example, in the execution of a certain task, the master servo 100 is manually remotely controlled to control the movements of the m slave robotic arms, such as controlling the joint motion data of each joint in the slave robotic arms. In this process, the joint motion data of each joint in the m slave robotic arms are collected and recorded, and the sample environmental data collected by the shooting device 300 at the set frequency is collected synchronously. For the introduction of the sample environmental data and the sample task, please refer to the aforementioned introduction of the environmental image data and the target task, which will not be repeated here.

[0075] The present application does not limit the implementation method for obtaining the above-mentioned first training set. For example, the present application can first obtain the first initial set. The content of the first initial set can correspond to the relevant introduction of the above-mentioned first training set, which will not be repeated here. Then, the present application can pre-process the above-mentioned first initial set to obtain the above-mentioned first training set. The above-mentioned preprocessing may include but is not limited to at least one of the following: data cleaning, data standardization, normalization, outlier processing or other custom data processing. For example, the robot control system 10 can obtain the above-mentioned first training set by loading, checking and preprocessing, such as the joint motion data of each joint in the above-mentioned m slave-controlled robotic arms.

[0076] S502: Train the first network to be trained based on the first training set to obtain the trained first network.

[0077] This application does not limit the specific implementation method of the above training. For example, a reinforcement learning or imitation learning algorithm can be used to repeatedly iterate the above first network using the sample environment data in the above first training set and the joint motion data of each joint in the above m slave-controlled robotic arms, so as to obtain the trained above first network. This application does not make too many restrictions and details on the specific training within the above network.

[0078] See also Figure 6 FIG. 1 is a flow chart showing a second network training process according to an exemplary embodiment. Figure 6 The process shown may include the following implementation steps:

[0079] S601, obtaining a second training set, where the second training set at least includes the base center coordinates corresponding to each of the m slave-controlled robotic arms when the sample task is performed.

[0080] In the present application, the second training set may include the base center coordinates corresponding to each of the m slave robotic arms when the sample task is completed, and may also include the joint motion data corresponding to each joint in the m slave robotic arms. In a specific implementation, the present application may train the m slave robotic arms, for example Figure 2 In the figure, the left slave robot 201 is paired with the left slave robot 202, and the right slave robot 203 is paired with the right slave robot 204. Specifically, when executing the above sample tasks, the present application can move the base center coordinates of the above m slave robot arms to the same position coordinates, such as (x1, y1, z1), and collect the joint motion data corresponding to each joint in the above m slave robot arms. For example, the joint motion data corresponding to each joint in the above slave robot 201 is (J1, J2, J3, J4, J5, J6), and the corresponding base center coordinates are (x1, y1, z1); when the above slave robot 202 executes the same sample task, its base center coordinates should also be (x1, y1, z1), and the joint motion data corresponding to each joint in the slave robot 202 can be (J7, J8, J9, J10, J11, J12), etc. In this example, the above joint mapping relationship may refer to the mapping relationship between (J1, J2, J3, J4, J5, J6) and (J7, J8, J9, J10, J11, J12).

[0081] The present application does not limit the implementation method for obtaining the above-mentioned second training set. For example, the present application can first obtain the second initial set. The content of the second initial set can correspond to the relevant introduction of the above-mentioned second training set, which will not be repeated here. Then, the present application can preprocess the above-mentioned second initial set to obtain the above-mentioned second training set. The above-mentioned preprocessing may include but is not limited to at least one of the following: data cleaning, data standardization, normalization or other custom data processing. For example, the robot control system 10 can obtain the above-mentioned second training set by loading, checking and preprocessing, such as the base center coordinates of each of the above-mentioned m slave-controlled robotic arms, and the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms.

[0082] S602: Train the second network to be trained based on the second training set to obtain the trained second network; the second network is used to indicate the joint mapping relationship.

[0083] The present application does not limit the specific implementation methods of the above-mentioned training. For example, a reinforcement learning or imitation learning algorithm can be used to repeatedly iterate the above-mentioned second network using the above-mentioned second training set to obtain the trained above-mentioned second network. Regarding the specific training within the above-mentioned network, this application does not make too many restrictions and details here. The above-mentioned second network is used to indicate the above-mentioned joint mapping relationship. The introduction of the above-mentioned joint mapping relationship can be referred to the relevant introduction in the aforementioned embodiment, which will not be repeated here.

[0084] It should be noted that the above-mentioned m slave-controlled robotic arms of the present application can be isomorphic robotic arms with the same structure, or heterogeneous robotic arms with different structures. When the above-mentioned m slave-controlled robotic arms are heterogeneous robotic arms, the present application adopts one-master-multiple-slave robot reinforcement learning training, which is beneficial to improving the applicability and generalization of the robot in a parallel heterogeneous use environment. The number of iterations involved in the above-mentioned network training of the present application is not limited, for example, 20-50 times, etc.; model training is carried out through a small sample training set, which reduces the requirements for the amount of sample training data, overcomes the limitations of the prior art, and is also beneficial to improving the performance and accuracy of the robot control system 10.

[0085] In some optional embodiments, the present application can control the m slave-controlled robotic arms to synchronously / simultaneously execute the control instructions of the m slave-controlled robotic arms to complete the target task in the target scene. That is, the m slave-controlled robotic arms are controlled to execute their corresponding control instructions at the same time, thereby completing the target task. This is conducive to improving the synchronization and accuracy of robot control.

[0086] In some optional embodiments, after the above-mentioned m slave-controlled mechanical arms are powered on, the shooting device 300 installed / set on the above-mentioned slave-controlled mechanical arms is also powered on synchronously. When the above-mentioned robot control system 10 is started, it can be determined whether the above-mentioned m slave-controlled mechanical arms support normal communication, for example, whether the control instructions of the master control servo 100 or the robot control system 10 can be received. If normal communication is not supported, the process can be terminated. On the contrary, if normal communication is supported, the above-mentioned method embodiment of the present application is further executed, for example, the above-mentioned shooting device 300 is further started to shoot and obtain corresponding environmental image data, and the above-mentioned environmental image data is input into the trained target model for processing, and the control instructions of each of the above-mentioned m slave-controlled mechanical arms are obtained. Further, the corresponding slave-controlled mechanical arm movement can be controlled according to the control instruction to complete the corresponding target task. It can be corresponding to the relevant introduction in the aforementioned embodiment, which will not be repeated here.

[0087] In some optional embodiments, after the m slave-controlled mechanical arms are controlled to move according to the corresponding control instructions, the position of the shooting device 300 installed / set on the slave-controlled mechanical arms also changes. At this time, the environmental image data shot by the shooting device 300 will also change, and the new environmental image data can be input into the target model for processing to obtain new control instructions for each of the m slave-controlled mechanical arms. According to the above principle, the movement of the m slave-controlled mechanical arms can be controlled by repeating the operation, and this application will not go into details.

[0088] In some optional embodiments, the above-mentioned environmental image data or the above-mentioned sample environmental data involved in the present application may also include depth data. For the introduction of depth data, please refer to the above-mentioned Figure 1 The relevant introduction in the embodiments is not repeated here. The present application can improve the operation accuracy of the robot in complex environments / scenes by integrating the depth of field data of the image, enhance the robot's perception of the surrounding environment, and improve the accuracy of the robot's recognition and object positioning.

[0089] By implementing the embodiment of the present application, the robot includes a master control servo, m slave control mechanical arms and a shooting device, wherein the shooting device is arranged on at least one of the slave control mechanical arms, and the m slave control mechanical arms are controlled by the master control servo, and m is a positive integer. The robot obtains the environmental image data shot by the shooting device, and the environmental image data is used to reflect the target scene and / or the target task to be performed by the robot; the environmental image data is input into the target model for processing, and the control instructions of the m slave control mechanical arms are obtained, and the control instructions are used to control the joint motion data corresponding to each joint in the slave control mechanical arm, and the target model includes a first network and a second network, and the first network is used to calculate the joint motion data corresponding to each joint in the target mechanical arm, and the target mechanical arm is at least one of the m slave control mechanical arms; the second network is used to map the joint motion data corresponding to each joint in the target mechanical arm, and obtain the control instructions of the m slave control mechanical arms. In this way, directly using the target model to control the m slave control mechanical arms is conducive to improving the efficiency and accuracy of robot control. At the same time, it can also solve technical problems existing in existing technologies, such as overfitting of small samples and lack of depth of field data affecting the operational accuracy of robot control.

[0090] Based on the above examples, see Figure 7 FIG. 1 is a schematic diagram showing the structure of a robot control device according to an exemplary embodiment. Figure 7The device shown can be applied to the above robot control system, which may include a master control steering engine, m slave control mechanical arms and a shooting device, wherein the shooting device is arranged on at least one of the slave control mechanical arms, and the m slave control mechanical arms are controlled by the master control steering engine, where m is a positive integer. The above device may include an acquisition module 701 and a processing module 702. Among them:

[0091] The acquisition module 701 is used to acquire the environmental image data captured by the shooting device, and the environmental image data is used to reflect the target scene where the robot is located and / or the target task to be performed;

[0092] The processing module 702 is used to input the environmental image data into the target model for processing to obtain control instructions for each of the m slave-controlled robotic arms, and the control instructions are used to control the joint motion data corresponding to each joint in the slave-controlled robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave-controlled robotic arms; the second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions for each of the m slave-controlled robotic arms.

[0093] In some embodiments, the target robotic arm includes a reference robotic arm, the second network includes a joint mapping relationship of the m slave robotic arms relative to the reference robotic arm, the joint mapping relationship is used to indicate a mapping relationship between joint motion data corresponding to each joint in the reference robotic arm and joint motion data corresponding to each joint in the slave robotic arm when having the same base center coordinates, and the mapping process based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control instructions of each of the m slave robotic arms includes:

[0094] Based on the joint mapping relationship of the m slave-controlled robotic arms relative to the reference robotic arm, data conversion is performed on the joint motion data corresponding to each joint in the reference robotic arm to obtain the joint motion data corresponding to each joint in the m slave-controlled robotic arms;

[0095] Based on the joint motion data corresponding to each joint in the m slave-controlled robotic arms, corresponding control instructions for each of the m slave-controlled robotic arms are generated.

[0096] In some embodiments, the target robotic arm includes the m slave robotic arms, the second network includes joint mapping relationships of the m slave robotic arms, the joint mapping relationships are used to indicate mapping relationships of joint motion data corresponding to each joint in the slave robotic arms when having the same base center coordinates, and the mapping process is performed based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions of the m slave robotic arms respectively, including:

[0097] Based on the joint mapping relationship of each of the m slave-controlled robotic arms, data conversion is performed on the joint motion data corresponding to each joint in the m slave-controlled robotic arms to obtain the converted joint data corresponding to each joint in the m slave-controlled robotic arms;

[0098] Based on the conversion joint data corresponding to each joint in the m slave-controlled robotic arms, control instructions for each of the m slave-controlled robotic arms are generated accordingly.

[0099] In some embodiments,

[0100] The acquisition module 701 is further used to acquire a first training set, wherein the first training set includes sample environment data captured by the camera and control motion data corresponding to each joint of the m slave mechanical arms when the master servo performs a sample task;

[0101] The processing module 702 is further configured to train the first network to be trained based on the first training set to obtain the trained first network.

[0102] In some embodiments,

[0103] The acquisition module 701 is further used to acquire a second training set, where the second training set at least includes the base center coordinates corresponding to each of the m slave-controlled robotic arms when the sample task is performed;

[0104] The processing module 702 is further configured to train the second network to be trained based on the second training set to obtain the trained second network.

[0105] In some embodiments, the first training set and / or the second training set is a preprocessed training set, and the preprocessing includes at least one of data cleaning, standardization, and outlier processing.

[0106] In some embodiments, the processing module 702 is further used to control the m slave-controlled robotic arms to synchronously execute respective control instructions of the m slave-controlled robotic arms to complete the target task.

[0107] In some embodiments, the image data also includes depth of field data; and / or the m slave-controlled robotic arms are heterogeneous robotic arms.

[0108] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0109] The present application also provides a computer-readable storage medium, which can be applied to the above Figure 1 or Figure 2 In the robot control system 10 shown in the figure, the computer storage medium stores computer program instructions, and when the program instructions are executed by the processor, the steps of the robot control method provided in the present application are implemented.

[0110] See also Figure 8 is a schematic diagram of a robot structure according to an exemplary embodiment. For example, the robot may include the master control steering engine 100, the m slave control mechanical arms (indicated as 201 to 20m in the figure) and the camera 300. Figure 8 The robot shown may also include: a memory 801, a processor 802, and a computer program 803 stored in the memory 801 and running on the processor 802, wherein when the processor 802 executes the computer program 803, the robot can execute the robot control method introduced above.

[0111] The embodiment of the present application further provides a robot control system, comprising a master control steering engine, m slave control mechanical arms and a photographing device, wherein the photographing device is arranged on at least one of the slave control mechanical arms, the m slave control mechanical arms are controlled by the master control steering engine, and m is a positive integer;

[0112] After the m slave-controlled robotic arms are powered on, the shooting devices provided on the slave-controlled robotic arms are powered on;

[0113] When the m slave-controlled mechanical arms support communication, starting the photographing device to photograph corresponding environmental image data;

[0114] The above-mentioned robot control system is used to execute the robot control method embodiment described above in the embodiment of the present application. For example, the environmental image data is input into the target model for processing to obtain control instructions for each of the m slave-controlled robotic arms, and the control instructions are used to control the joint motion data corresponding to each joint in the slave-controlled robotic arm. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave-controlled robotic arms; the second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm, and obtain the control instructions for each of the m slave-controlled robotic arms, etc. For the content not introduced or described in the embodiment of the present application, please refer to the relevant introduction in the aforementioned method embodiment, and the embodiment of the present application does not limit it.

[0115] In a specific implementation, the system may include a memory and a processor, wherein an executable program code is stored in the memory, and the processor is used to call and execute the executable program code to execute a robot control method provided in an embodiment of the present application. The embodiment of the present application can divide the functional modules of the system according to the above method example. For example, it can correspond to each functional module, or two or more functions can be integrated into a processing module, and the above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, which is only a logical function division, and there may be other division methods in actual implementation. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0116] It should be understood that the system provided in the embodiment of the present application is used to execute the above-mentioned robot control method, so the same effect as the above-mentioned implementation method can be achieved. In the case of an integrated unit, the system may include a processing module and a storage module. Among them, when the system is applied to a robot, the processing module can be used to control and manage the robot's movements. The storage module can be used to support the robot to execute mutual program codes, etc. Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of the present application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module can be a memory.

[0117] In addition, the system provided in an embodiment of the present application may also include a chip, which may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a robot control method provided in the above embodiment.

[0118] In another exemplary embodiment, a computer program product is also provided. The computer program product can be applied to the above Figure 1 or Figure 2 The robot control system 10 shown in the figure. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned robot control method when executed by the programmable device.

[0119] See also Fig. 9 FIG. 1 is a schematic diagram showing the structure of a chip according to an exemplary embodiment. Fig. 9 The chip 900 shown can be applied to the above Figure 1 or Figure 2 In the robot control system 10 shown in the figure, the chip 900 may include a processor 901 and an interface 902. Optionally, it may also include a memory 903. The number of the processors 901 may be one or more, and the number of the interfaces 902 may be multiple.

[0120] In one embodiment, for a chip used to implement the method embodiment described in this application:

[0121] The interface 902 is used to receive or output signals;

[0122] The processor 901 is used to execute part or all of the contents in the robot control method embodiment.

[0123] Understandably, the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0124] Understandably, the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0125] It should be noted here that the description of the above storage medium, system and chip embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0126] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the present application. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0127] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A robot control method, characterized in that: The robot comprises a master control steering engine, m slave control mechanical arms and a photographing device, wherein the photographing device is arranged on at least one of the slave control mechanical arms, and the m slave control mechanical arms are controlled by the master control steering engine, where m is a positive integer. The method comprises: Acquire environmental image data captured by the shooting device, where the environmental image data is used to reflect the target scene in which the robot is located and / or the target task to be performed; The environmental image data is input into the target model for processing to obtain control instructions for each of the m slave-controlled robotic arms, and the control instructions are used to control joint motion data corresponding to each joint in the slave-controlled robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave-controlled robotic arms; the second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions for each of the m slave-controlled robotic arms.

2. The method according to claim 1, characterized in that: The target manipulator includes a reference manipulator, the second network includes a joint mapping relationship of the m slave manipulators relative to the reference manipulator, the joint mapping relationship is used to indicate a mapping relationship between joint motion data corresponding to each joint in the reference manipulator when having the same base center coordinates and joint motion data corresponding to each joint in the slave manipulator, and the mapping process based on the joint motion data corresponding to each joint in the target manipulator to obtain control instructions for each of the m slave manipulators includes: Based on the joint mapping relationship of the m slave-controlled robotic arms relative to the reference robotic arm, data conversion is performed on the joint motion data corresponding to each joint in the reference robotic arm to obtain the joint motion data corresponding to each joint in the m slave-controlled robotic arms; Based on the joint motion data corresponding to each joint in the m slave-controlled robotic arms, corresponding control instructions for each of the m slave-controlled robotic arms are generated.

3. The method according to claim 1, characterized in that The target robotic arm includes the m slave robotic arms, the second network includes joint mapping relationships of the m slave robotic arms, the joint mapping relationships are used to indicate mapping relationships of joint motion data corresponding to each joint in the slave robotic arms when they have the same base center coordinates, and the mapping process is performed based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions of the m slave robotic arms respectively, including: Based on the joint mapping relationship of each of the m slave-controlled robotic arms, data conversion is performed on the joint motion data corresponding to each joint in the m slave-controlled robotic arms to obtain the converted joint data corresponding to each joint in the m slave-controlled robotic arms; Based on the conversion joint data corresponding to each joint in the m slave-controlled robotic arms, control instructions for each of the m slave-controlled robotic arms are generated accordingly.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire a first training set, wherein the first training set includes sample environment data captured by the camera, and control motion data corresponding to each joint of the m slave mechanical arms when the master steering engine performs a sample task; The first network to be trained is trained based on the first training set to obtain the trained first network.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a second training set, where the second training set at least includes the base center coordinates corresponding to each of the m slave-controlled robotic arms when the sample task is performed; The second network to be trained is trained based on the second training set to obtain the trained second network.

6. The method according to claim 4 or 5, characterized in that: The first training set and / or the second training set are preprocessed training sets, and the preprocessing includes at least one of data cleaning, standardization, and outlier processing.

7. The method according to claim 6, characterized in that The method further comprises: The m slave-controlled robotic arms are controlled to synchronously execute respective control instructions of the m slave-controlled robotic arms to complete the target task.

8. The method according to claim 6, characterized in that The environmental image data also includes depth of field data; and / or, the m slave-controlled robotic arms are heterogeneous robotic arms.

9. A robot, characterized in that: The robot comprises a master control steering engine, m slave control mechanical arms and a photographing device, wherein the photographing device is arranged on at least one of the slave control mechanical arms, the m slave control mechanical arms are controlled by the master control steering engine, m is a positive integer, and the robot further comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions to implement the steps of the method according to any one of claims 1 to 8.

10. A robot control system, characterized in that: It comprises a master control steering engine, m slave control mechanical arms and a photographing device, wherein the photographing device is arranged on at least one of the slave control mechanical arms, and the m slave control mechanical arms are controlled by the master control steering engine, where m is a positive integer; After the m slave-controlled robotic arms are powered on, the shooting devices provided on the slave-controlled robotic arms are powered on; When the m slave-controlled robotic arms support communication, the shooting device is started to shoot corresponding environmental image data, where the environmental image data is used to reflect the target scene in which the robot is located and / or the target task to be performed; The environmental image data is input into a target model for processing to obtain control instructions for each of the m slave-controlled robotic arms, wherein the control instructions are used to control joint motion data corresponding to each joint in the slave-controlled robotic arms, wherein the target model includes a first network and a second network, wherein the first network is used to calculate joint motion data corresponding to each joint in a target robotic arm, and the target robotic arm is at least one of the m slave-controlled robotic arms; The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain control instructions for each of the m slave robotic arms.

Images