Robot control method, robot and system

Abstract

The application belongs to the technical field of robots, and discloses a robot control method, a robot and a system. The robot comprises a master control steering engine, m slave control mechanical arms and a shooting device. The shooting device is arranged on at least one of the slave control mechanical arms. The method comprises the following steps: acquiring environment image data shot by the shooting device; inputting the environment image data into a target model for processing to obtain control instructions of the m slave control mechanical arms respectively. The target model comprises a first network and a second network. The first network is used for calculating joint motion data corresponding to each joint in a target mechanical arm. The target mechanical arm is at least one of the m slave control mechanical arms. The second network is used for mapping processing based on the joint motion data corresponding to each joint in the target mechanical arm to obtain the control instructions of the m slave control mechanical arms respectively. In this way, the accuracy and efficiency of robot control can be improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a robot control method, robot, and system. Background Technology

[0002] With the continuous development of computer technology, sensor technology, and bionics, robots are being used more and more widely. For example, they are used in logistics handling, customer reception, guidance and explanation, business processing, and teaching and research. Typically, a robotic arm is mounted on the robot body, which performs tasks such as grasping and moving objects.

[0003] In practical applications, the quality of robot arm control directly affects the robot's operational performance; therefore, it is essential to learn how to better control the robot arm. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this application provides a robot control method, robot, and system that can improve the accuracy and efficiency of robot control.

[0005] According to a first aspect of this application, a robot control method is provided, the robot comprising a master control servo motor, m slave control robotic arms, and a camera device, wherein the camera device is disposed on at least one of the slave control robotic arms, and the m slave control robotic arms are controlled by the master control servo motor, where m is a positive integer, the method comprising:

[0006] The environmental image data captured by the imaging device is used to reflect the target scene where the robot is located and / or the target task to be performed.

[0007] The environmental image data is input into the target model for processing to obtain control commands for each of the m slave robotic arms. The control commands are used to control the joint motion data corresponding to each joint in the slave robotic arm. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms.

[0008] Optionally, the target robotic arm includes a reference robotic arm, and the second network includes joint mapping relationships of the m slave robotic arms relative to the reference robotic arm. These joint mapping relationships indicate the mapping relationship between the joint motion data corresponding to each joint in the reference robotic arm and the joint motion data corresponding to each joint in the corresponding slave robotic arm when they have the same base center coordinates. The mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms includes:

[0009] Based on the joint mapping relationship between the m slave robotic arms and the reference robotic arm, the joint motion data corresponding to each joint in the reference robotic arm is converted to obtain the joint motion data corresponding to each joint in the m slave robotic arms.

[0010] Based on the joint motion data corresponding to each joint in the m slave robotic arms, control commands are generated for each of the m slave robotic arms.

[0011] Optionally, the target robotic arm includes the m slave robotic arms, and the second network includes the joint mapping relationship of each of the m slave robotic arms. The joint mapping relationship is used to indicate the mapping relationship of the joint motion data corresponding to each joint in the slave robotic arm when they have the same base center coordinates. The step of mapping based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms includes:

[0012] Based on the joint mapping relationship of each of the m slave robotic arms, the joint motion data corresponding to each joint in the m slave robotic arms is converted to obtain the converted joint data corresponding to each joint in the m slave robotic arms.

[0013] Based on the transition joint data corresponding to each joint in the m slave robotic arms, control commands are generated for each of the m slave robotic arms.

[0014] Optionally, the method further includes:

[0015] Obtain a first training set, which includes sample environment data captured by the imaging device and control motion data of the master control servo motor on each joint of the m slave control robotic arms when performing sample tasks;

[0016] The first network to be trained is trained based on the first training set to obtain the trained first network.

[0017] Optionally, the method further includes:

[0018] Obtain a second training set, which includes at least the base center coordinates of each of the m slave robotic arms when the sample task is completed;

[0019] The second network to be trained is trained based on the second training set to obtain the trained second network.

[0020] Optionally, the first training set and / or the second training set are preprocessed training sets, wherein the preprocessing includes at least one of data cleaning, standardization, and outlier handling.

[0021] Optionally, the method further includes:

[0022] The m slave robotic arms are controlled to synchronously execute their respective control commands in order to complete the target task.

[0023] Optionally, the image data may further include depth data; and / or, the m slave robotic arms may be heterogeneous robotic arms.

[0024] According to a second aspect of this application, a robot is provided, including a master control servo motor, m slave control robotic arms, and a camera device, wherein the camera device is disposed on at least one of the slave control robotic arms, the m slave control robotic arms are controlled by the master control servo motor, and m is a positive integer. The robot further includes: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the robot control method described above.

[0025] According to a third aspect of this application, a robot control system is provided, including a master control servo motor, m slave control robotic arms and a camera, wherein the camera is disposed on at least one of the slave control robotic arms, and the m slave control robotic arms are controlled by the master control servo motor, where m is a positive integer;

[0026] After the m slave robotic arms are powered on, the shooting device installed on the slave robotic arms is powered on.

[0027] When the m slave robotic arms support communication, the imaging device is activated to capture corresponding environmental image data. The environmental image data is used to reflect the target scene where the robot is located and / or the target task to be performed.

[0028] The environmental image data is input into the target model for processing to obtain control commands for each of the m slave robotic arms. The control commands are used to control the joint motion data corresponding to each joint in the slave robotic arm. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms.

[0029] For any content not introduced or described in the embodiments of this application, please refer to the relevant descriptions in the foregoing method embodiments. This application does not limit the scope of the embodiments.

[0030] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0031] (1) The robot can directly process the environmental image data captured by the imaging device using the target network to obtain the control commands for each of the m slave robotic arms in the robot, thereby controlling the joint motion data of each joint in the corresponding slave robotic arm to complete the corresponding target task. In this way, the efficiency and accuracy of robot control can be improved.

[0032] (2) This application can first calculate the joint motion data corresponding to each joint in a reference robotic arm using a first network; then, using the joint mapping relationship between the m slave robotic arms and the reference robotic arm in the second network, the joint motion data corresponding to each joint in the reference robotic arm is mapped and processed to obtain the joint motion data corresponding to each joint in the m slave robotic arms, and the control commands for each of the m slave robotic arms are generated accordingly. In this way, the joint motion data corresponding to each joint in other slave robotic arms can be mapped and converted based on the joint motion data corresponding to each joint in the reference robotic arm, and the corresponding control commands for the slave robotic arms can be generated; this can reduce the resource consumption of the whole process and also help improve the accuracy of robot control.

[0033] (3) This application can first calculate the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms using a first network; then, it can use the joint mapping relationship of each of the above-mentioned m slave-controlled robotic arms in the second network to map the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms, thereby obtaining the transformation motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms, and using this to generate the control commands for each of the above-mentioned m slave-controlled robotic arms. In this way, the joint motion data corresponding to each joint in the above-mentioned m slave-controlled robotic arms can be roughly / coarsely calculated using the first network first, and then the mapping relationship of the above-mentioned m joints can be used to perform a more accurate mapping transformation calculation, which is beneficial to improving the convenience and accuracy of robot control.

[0034] (4) This application uses the acquired first training set to train the first network to be trained, thereby obtaining a trained first network. Training the first network with a small training set reduces the requirement for the amount of training data and overcomes the limitations of existing technologies that rely on a large amount of training data in scenarios where data collection is difficult or costly. It also helps to improve the performance and accuracy of robot control.

[0035] (5) This application uses the acquired second training set to train the second network to be trained, thereby obtaining a trained second network. Using a small training set to train the second network reduces the requirement for the amount of sample training data, overcomes the limitations of existing technologies that rely on a large amount of sample training data in scenarios where data collection is difficult or costly, and also helps to improve the performance and accuracy of robot control.

[0036] (6) This application uses the pre-processed first training set and / or second training set to train the corresponding network, which is beneficial to improve the accuracy or precision of network training, and also beneficial to improve the accuracy and reliability of subsequent robot control.

[0037] (7) Based on the control instructions of the above m slave robotic arms, this application synchronously controls the joint motion data corresponding to each joint in the above m slave robotic arms to ensure the synchronization of robot actions and complete the corresponding target tasks, thereby improving the synchronization and accuracy of robot control.

[0038] (8) This application improves the robot's operational accuracy in complex environments by combining image depth data, enhances the robot's perception of the surrounding environment / scene, and improves the accuracy of the robot in identifying and locating objects. And / or, using a master-slave heterogeneous robot for reinforcement learning training is beneficial to improving the robot's applicability and generalization in parallel use and heterogeneous environments.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 This is a schematic diagram of the structure of a robot control system according to an exemplary embodiment.

[0042] Figure 2 This is a schematic diagram of the structure of another robot control system according to an exemplary embodiment.

[0043] Figure 3 This is a flowchart illustrating a robot control method according to an exemplary embodiment.

[0044] Figure 4 This is a schematic diagram of the internal structure of a target model according to an exemplary embodiment.

[0045] Figure 5 This is a schematic diagram illustrating a process for training a first network according to an exemplary embodiment.

[0046] Figure 6 This is a schematic diagram illustrating a second network training process according to an exemplary embodiment.

[0047] Figure 7 This is a schematic diagram of the structure of a robot control device according to an exemplary embodiment.

[0048] Figure 8 This is a schematic diagram of the structure of a robot according to an exemplary embodiment.

[0049] Figure 9 This is a schematic diagram of the structure of a chip according to an exemplary embodiment. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0051] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0052] Traditional robot motion training methods typically rely on large amounts of sample data, which presents limitations in scenarios where data acquisition is difficult or costly. Furthermore, overfitting under small sample data training conditions remains a significant and difficult problem to solve.

[0053] Current neural network-based robot control schemes typically rely on color (Red, Green, Blue, RGB) images and joint position information, lacking an understanding of scene depth. This leads to the robot's inability to accurately determine the distance and spatial relationships of objects when handling complex environments, thus affecting the precision of robot control operations. For example, it is easily affected by interference in low-light or strong-light conditions, resulting in decreased accuracy in recognition and localization. Furthermore, traditional schemes cannot effectively identify and locate partially occluded objects. In traditional neural network-based robot control schemes, a large amount of labeled data is often required during neural network training to improve model performance. However, the lack of depth information makes it difficult for the network model to capture important spatial features during learning, resulting in a low learning rate. In addition, traditional robot control schemes typically use only one master-slave structure, and the slave robotic arms often have identical or identical structures. This presents significant limitations in application scenarios / environments where multiple heterogeneous robots of different models work collaboratively.

[0054] To address the aforementioned problems, this application proposes a robot control method, a robot, and a system. Please refer to [link / reference]. Figure 1 This is a schematic diagram illustrating the structure of a robot control system according to an exemplary embodiment. Figure 1 The robot control system 10 (or robot 10) shown may include: a master servo motor 100, m slave robotic arms (represented as 201 to 20m in the illustration) and a camera 300. This application does not limit the number of the master servo motor 100 and the slave robotic arms; there may be one or more master servo motors 100, and there may be m slave robotic arms, where m is a positive integer configured according to actual conditions.

[0055] The aforementioned master servo motor 100 can control the aforementioned m slave robotic arms, for example, controlling the joint motion data of each joint in the slave robotic arms to complete the corresponding target task. This target task is the task the robot wants to perform, which may include, but is not limited to, object grasping, handling, assembly, welding, palletizing, spraying, grinding, or other custom tasks. The aforementioned joint motion data can refer to data used to describe joint motion, which may include, but is not limited to, joint motion angle, joint motion speed, joint motion acceleration, or other motion data. In specific implementation, the master servo motor 100 can receive manually input control information and drive each joint of the slave robotic arm to perform the corresponding task according to a predetermined trajectory or action. This application does not limit the specific implementation method of the aforementioned control information. For example, if the aforementioned master servo motor 100 is a remote-controlled servo motor, this application can control it through a corresponding remote-controlled signal; or if the aforementioned master servo motor 100 is a Bluetooth servo motor, this application can control it through a corresponding Bluetooth signal, etc.

[0056] The aforementioned imaging device 300 is used to collect corresponding environmental image data, such as collecting data at a set frequency. This set frequency is customized by the system or user according to actual needs, such as user preferences. This application does not limit the type of the imaging device 300; for example, it can be a general-purpose depth camera or other devices used for image or video capture. The aforementioned environmental image data may include data about the robot's current environment and depth data. This depth data reflects the range of clear objects in the environmental image data, specifically, for example, indicating the distance between the farthest and nearest points in the environmental image data. This application does not limit the installation location of the imaging device 300; specifically, the imaging device 300 can be mounted on at least one slave robotic arm, for example, on the wrist of a slave robotic arm. Please refer to [further details omitted]. Figure 2 This is a schematic diagram illustrating the structure of another robot according to an exemplary embodiment. For example... Figure 2 The robot control system 10 shown includes two master servos, denoted as 101 and 102. Each master servo controls two slave robotic arms. In the illustration, master servo 101 controls slave robotic arms 201 and 202. Master servo 102 controls slave robotic arms 203 and 204. A camera device 301 and a camera device 302 are located at the wrist of slave robotic arm 201 and 203, respectively. The camera devices 301 and 302 are used to periodically or in real-time capture environmental image data of the slave robotic arm 201 and 203, respectively. Figure 2 The components of the robot shown are arranged symmetrically, as shown in the figure.

[0057] This application does not limit the model or type of each of the aforementioned m slave robotic arms. For example, the aforementioned m slave robotic arms can be homogeneous robotic arms with the same structure, or they can be heterogeneous robotic arms with different models / structures. This application will not impose further limitations or details in this regard. In practical applications, the aforementioned master control servo motor 100, the aforementioned m slave robotic arms, and the aforementioned imaging device 300 can be mounted on the same horizontal plate. For example... Figure 2 As shown, the aforementioned master servo motor 101, master servo motor 102, slave robotic arms 201, 202, 203, and 204, as well as the imaging device 301 and 302, can all be mounted on the same horizontal plate. Each of the master servo motor 100 and the slave robotic arm can have a base (also called a chassis) at its bottom. The base of the master servo motor 100 can support its basic functions such as walking, movement, and obstacle avoidance. The base of the slave robotic arm can support its basic functions such as walking, movement, and obstacle avoidance.

[0058] Based on the above embodiments, please refer to Figure 3 This is a flowchart illustrating a robot control method according to an exemplary embodiment. Figure 3 The method shown can be applied to Figure 1 or Figure 2 In the robot control system 10 shown, the method may include the following implementation steps:

[0059] S301. Acquire environmental image data captured by the shooting device, wherein the environmental image data is used to reflect the target scene where the robot is located and / or the target task to be performed.

[0060] In this application, the aforementioned environmental image data may refer to environmental images captured by the aforementioned imaging device 300, which are used to reflect the target scene currently in which the system is located and / or the target task to be performed. The target scene may refer to the environmental scene in which the robot is located, which may include, but is not limited to, scenarios such as object handling and welding. The aforementioned target task corresponds to the aforementioned target scene, and may refer to the task waiting to be performed in the aforementioned target scene, which may include, but is not limited to, object grasping, handling, assembly, welding, palletizing, spraying, polishing, or other custom tasks.

[0061] This application does not limit the implementation method of acquiring the above-mentioned environmental image data. For example, the environmental image data can be directly acquired by capturing the above-mentioned shooting device 300; or the environmental image data captured by the above-mentioned shooting device 300 can be acquired from other devices (such as storage devices or other terminals) through the network. If the storage space capacity is limited, the above-mentioned shooting device 300 may store the captured environmental image data in an external memory, etc.

[0062] S302. The environmental image data is input into the target model for processing to obtain control commands for each of the m slave robotic arms. The control commands are used to control the joint motion data corresponding to each joint in the slave robotic arm. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm. The target robotic arm is at least one of the m slave robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms.

[0063] The aforementioned slave-controlled robotic arm may include one or more joints. For example, a KUKA robot may include six joints, and the joint motion data corresponding to these six joints can be represented as (J1, J2, J3, J4, J5, J6). Each joint has a preset range of motion angles, which determines the robot's flexibility and operability. This preset angle range is pre-defined by the system or the developers, such as an experience-based range or a statistical range calculated from a series of experimental data. This application does not impose further limitations or restrictions on this aspect.

[0064] The aforementioned target model can refer to a model used to predict joint motion data from each joint of a robotic arm, or to predict control commands from each joint of a robotic arm. This model can include, but is not limited to, neural network models, grayscale prediction models, regression prediction models, support vector machine regression (SVR) models, or other models used for robot motion prediction.

[0065] By implementing the embodiments of this application, a robot includes a master control servo motor, m slave control robotic arms, and a camera device. The camera device is mounted on at least one of the slave control robotic arms, and the m slave control robotic arms are controlled by the master control servo motor, where m is a positive integer. The robot acquires environmental image data captured by the camera device. This environmental image data reflects the target scene where the robot is located and / or the target task to be performed. The environmental image data is input into a target model for processing to obtain control commands for each of the m slave control robotic arms. These control commands are used to control the joint motion data corresponding to each joint in the slave control robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, where the target robotic arm is at least one of the m slave control robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave control robotic arms. Thus, directly using the target model to control the m slave control robotic arms improves the efficiency and accuracy of robot control.

[0066] In step S302, this application does not limit the internal structure of the target model described above. For example, please refer to [link to relevant documentation]. Figure 4 This is a schematic diagram illustrating the internal structure of a target model according to an exemplary embodiment. For example... Figure 4 The target model shown may include a first network 401 and a second network 402. The first network 401 is mainly used to calculate the joint motion data corresponding to each joint in the target robotic arm, which is one or more of the aforementioned m slave robotic arms. The second network 402 is mainly used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm, thereby obtaining the control commands for each of the m slave robotic arms. This application does not limit the internal structure of the first network 401 and the second network 402; for example, both the first network 401 and the second network 402 can be neural networks, deep learning networks, or other custom functional networks.

[0067] This application does not limit the specific implementation of step S302 described above. This application can utilize the trained first network to perform inference calculations on the environmental image data, thereby calculating the joint motion data corresponding to each joint in the target robotic arm, where the target robotic arm is some or all of the m slave robotic arms. Further, the trained second network is used to map the target robotic arm, thereby obtaining the control commands for each of the m slave robotic arms. The specific implementation of the inference calculation depends on the internal structure of the first network. For example, if the first network is a convolutional neural network, this application can use the first network to perform corresponding convolutional neural network processing on the environmental image data, etc. This application will not impose further limitations or details on this.

[0068] This application does not limit the specific implementation of the above mapping process. For example, in one embodiment, the target robotic arm may include a reference robotic arm. The second network includes the joint mapping relationships of the m slave robotic arms relative to the reference robotic arm. Based on the m joint mapping relationships, this application can perform data conversion on the joint motion data corresponding to each joint in the reference robotic arm to obtain the converted motion data corresponding to each joint in the m slave robotic arms, which is the joint motion data corresponding to each joint in the slave robotic arm. Further, this application can generate corresponding control commands for each of the m slave robotic arms based on the joint motion data corresponding to each joint in the m slave robotic arms. The control commands are used to control each joint in the corresponding slave robotic arm to move according to the corresponding joint motion data, thereby completing the corresponding task.

[0069] The aforementioned joint mapping relationship is used to indicate the mapping relationship between the joint motion data corresponding to each joint in the reference robotic arm and the joint motion data corresponding to each joint in the corresponding slave robotic arm when they have the same base center coordinates. That is, it indicates the mapping relationship between the joint motion data corresponding to each joint in the m slave robotic arms and the joint motion data corresponding to each joint in the reference robotic arm when they have the same base center coordinates. For example, combined with... Figure 2In the robot shown, when performing the same target task, the base center coordinates of the slave robotic arm 201 and the slave robotic arm 202 need to be moved to the same position coordinates, for example (x1, y1, z1). Assume the joint motion data corresponding to each joint in the slave robotic arm 201 are (J1, J2, J3, J4, J5, J6), and the joint motion data corresponding to each joint in the slave robotic arm 202 are (J7, J8, J9, J10, J11, J12). In this example, the joint mapping relationship can refer to the mapping relationship between (J1, J2, J3, J4, J5, J6) and (J7, J8, J9, J10, J11, J12).

[0070] In another embodiment, the target robotic arm may include m slave robotic arms, and the second network includes the joint mapping relationships of the m slave robotic arms. Specifically, it may include the joint mapping relationships of the m slave robotic arms relative to preset reference points. The preset reference points are reference points pre-defined by the system according to actual conditions, such as any reference point selected on a horizontal plate, etc., which are not further limited or detailed in this application. Based on the m joint mapping relationships, this application can perform data conversion on the joint motion data corresponding to each joint in the m slave robotic arms to obtain the converted motion data corresponding to each joint in the m slave robotic arms, which is the joint motion data corresponding to each joint in the slave robotic arm. Further, based on the converted motion data corresponding to each joint in the m slave robotic arms, this application can generate corresponding control commands for each of the m slave robotic arms. These control commands are used to control each joint in the corresponding slave robotic arm to move according to the corresponding joint motion data, thereby completing the corresponding task. In this embodiment, the joint motion data corresponding to each joint in the above m slave robotic arms can be roughly calculated using the first network first, and then the mapping relationship of the above m joints can be used to perform a more accurate mapping transformation calculation, which is beneficial to improving the convenience and accuracy of robot control.

[0071] Before performing step S302, this application also needs to train the target model. This training process can be applied to a training device, which may include, but is not limited to, the aforementioned robot, terminal device, server, or other devices with model training capabilities. Taking the target model including the first network 401 and the second network 402 as an example, the training embodiments involving each of the two networks are described below.

[0072] Please see Figure 5 This is a schematic diagram illustrating a first network training process according to an exemplary embodiment. For example... Figure 5 The process shown may include the following implementation steps:

[0073] S501. Obtain the first training set, which includes sample environment data captured by the shooting device and control motion data of each joint in the m slave robotic arms when the master control servo motor performs the sample task.

[0074] In this application, the first training set mentioned above may include sample environment data captured by the imaging device 300 and control motion data of the master servo motor 100 on each joint of the m slave robotic arms when performing the sample task. For example, in performing a certain task, the master servo motor 100 is manually remotely controlled to control the movements of the m slave robotic arms, such as controlling the joint motion data of each joint in the slave robotic arms. During this process, the joint motion data of each joint in the m slave robotic arms is collected and recorded, and the sample environment data collected by the imaging device 300 at a set frequency is collected simultaneously. For a description of the sample environment data and the sample task, please refer to the foregoing descriptions of the environmental image data and the target task, which will not be repeated here.

[0075] This application does not limit the implementation method of obtaining the first training set. For example, this application can first obtain a first initial set, the content of which can be referred to in the relevant description of the first training set mentioned above, and will not be repeated here. Then, this application can preprocess the first initial set to obtain the first training set. The preprocessing may include, but is not limited to, at least one of the following: data cleaning, data standardization, normalization, outlier handling, or other custom data processing. For example, the robot control system 10 can obtain the first training set, such as the joint motion data of each joint in the m slave robotic arms, through loading, checking, and preprocessing.

[0076] S502. Train the first network to be trained based on the first training set to obtain the trained first network.

[0077] This application does not limit the specific implementation method of the above training. For example, reinforcement learning or imitation learning algorithms can be used to iteratively train the first network using the sample environment data in the first training set and the joint motion data of each joint in the m slave robotic arms, thereby obtaining the trained first network. This application does not impose further limitations or details on the specific training within the network.

[0078] Please see Figure 6 This is a schematic diagram illustrating a second network training process according to an exemplary embodiment. For example... Figure 6 The process shown may include the following implementation steps:

[0079] S601. Obtain a second training set, which includes at least the center coordinates of the bases of each of the m slave robotic arms when the sample task is completed.

[0080] The second training set mentioned above in this application may include the base center coordinates of each of the m slave-controlled robotic arms after completing the sample task, and may also include joint motion data corresponding to each joint of the m slave-controlled robotic arms. In specific implementations, this application can be used for the m slave-controlled robotic arms, for example... Figure 2 The left slave robotic arm 201 is paired with the left slave robotic arm 202, and the right slave robotic arm 203 is paired with the right slave robotic arm 204. Specifically, when performing the above sample task, this application can move the base center coordinates of each of the above m slave robotic arms to the same position coordinates, for example (x1, y1, z1), and collect the joint motion data corresponding to each joint in the above m slave robotic arms. For example, the joint motion data corresponding to each joint in the above slave robotic arm 201 is (J1, J2, J3, J4, J5, J6), and its corresponding base center coordinates are (x1, y1, z1); when the above slave robotic arm 202 performs the same sample task, its base center coordinates should also be (x1, y1, z1), and the joint motion data corresponding to each joint in the slave robotic arm 202 can be (J7, J8, J9, J10, J11, J12), etc. In this example, the joint mapping relationship mentioned above can refer to the mapping relationship between (J1, J2, J3, J4, J5, J6) and (J7, J8, J9, J10, J11, J12).

[0081] This application does not limit the implementation method for obtaining the second training set. For example, this application can first obtain a second initial set, the content of which can be referred to in the relevant description of the second training set mentioned above, and will not be repeated here. Next, this application can preprocess the second initial set to obtain the second training set. The preprocessing may include, but is not limited to, at least one of the following: data cleaning, data standardization, normalization, or other custom data processing. For example, the robot control system 10 can obtain the second training set by loading, checking, and preprocessing, such as the center coordinates of the bases of the m slave robotic arms and the joint motion data corresponding to each joint in the m slave robotic arms.

[0082] S602. The second network to be trained is trained based on the second training set to obtain the trained second network; the second network is used to indicate the joint mapping relationship.

[0083] This application does not limit the specific implementation method of the above training. For example, reinforcement learning or imitation learning algorithms can be used to iteratively train the second network using the second training set to obtain the trained second network. The specific training details within the network are not described in detail here. The second network is used to indicate the joint mapping relationship. For a description of the joint mapping relationship, please refer to the relevant descriptions in the foregoing embodiments; they will not be repeated here.

[0084] It should be noted that the m slave-controlled robotic arms mentioned above in this application can be homogeneous robotic arms with the same structure or heterogeneous robotic arms with different structures. When the m slave-controlled robotic arms are heterogeneous, this application adopts a one-master-many-slave robot reinforcement learning training, which is beneficial to improving the applicability and generalization of the robot in parallel heterogeneous usage environments. The number of iterations involved in the network training mentioned above in this application is not limited, for example, 20-50 times; model training is performed using a small sample training set, which reduces the requirement for the amount of sample training data, overcomes the limitations of the prior art, and is also beneficial to improving the performance and accuracy of the robot control system 10.

[0085] In some optional embodiments, this application can control the m slave robotic arms to synchronously / simultaneously execute their respective control commands to complete the target task in the target scenario. That is, it simultaneously controls the m slave robotic arms to execute their respective control commands, thereby completing the target task. This improves the synchronization and accuracy of robot control.

[0086] In some optional embodiments, after the m slave robotic arms are powered on, the imaging device 300 installed / set on the slave robotic arms is also powered on synchronously. When the robot control system 10 starts, it can determine whether the m slave robotic arms support normal communication, such as whether they can receive control commands from the master servo motor 100 or the robot control system 10. If normal communication is not supported, the process can be terminated. Otherwise, if normal communication is supported, the method embodiment described above is further executed, for example, the imaging device 300 is further activated to capture corresponding environmental image data, and the environmental image data is input into the trained target model for processing to obtain control commands for each of the m slave robotic arms. The corresponding slave robotic arm can then be controlled to move according to the control command to complete the corresponding target task. This can be referred to in the relevant descriptions in the foregoing embodiments, and will not be repeated here.

[0087] In some optional embodiments, after the m slave robotic arms move according to the corresponding control commands, the position of the imaging device 300 installed / set on the slave robotic arms also changes. At this time, the environmental image data captured by the imaging device 300 will also change. The new environmental image data can be input into the target model for processing to obtain new control commands for each of the m slave robotic arms. By repeating this operation according to the above principle, the movement of the m slave robotic arms can be controlled, which will not be elaborated further in this application.

[0088] In some optional embodiments, the environmental image data or sample environmental data involved in this application may further include depth data. For a description of depth data, please refer to the foregoing. Figure 1 The relevant descriptions in the embodiments described herein will not be repeated here. This application can improve the robot's operational accuracy in complex environments / scenes by integrating depth data from images, enhance the robot's perception of its surroundings, and improve the accuracy of robot recognition and object localization.

[0089] By implementing the embodiments of this application, a robot includes a master control servo motor, m slave control robotic arms, and a camera device. The camera device is mounted on at least one of the slave control robotic arms, and the m slave control robotic arms are controlled by the master control servo motor, where m is a positive integer. The robot acquires environmental image data captured by the camera device. This environmental image data reflects the target scene where the robot is located and / or the target task to be performed. The environmental image data is input into a target model for processing to obtain control commands for each of the m slave control robotic arms. These control commands are used to control the joint motion data corresponding to each joint in the slave control robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, where the target robotic arm is at least one of the m slave control robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave control robotic arms. Thus, directly using the target model to control the m slave control robotic arms improves the efficiency and accuracy of robot control. It can also solve technical problems existing in the current technology, such as overfitting with small samples and the lack of depth data affecting the operational accuracy of robot control.

[0090] Based on the foregoing embodiments, please refer to Figure 7 This is a schematic diagram illustrating the structure of a robot control device according to an exemplary embodiment. Figure 7The illustrated device can be applied to the aforementioned robot control system, which may include a master servo motor, m slave robotic arms, and a camera. The camera is mounted on at least one of the slave robotic arms, and the m slave robotic arms are controlled by the master servo motor, where m is a positive integer. The device may include an acquisition module 701 and a processing module 702. Wherein:

[0091] The acquisition module 701 is used to acquire environmental image data captured by the shooting device. The environmental image data is used to reflect the target scene where the robot is located and / or the target task to be performed.

[0092] The processing module 702 is used to input the environmental image data into the target model for processing to obtain control commands for each of the m slave robotic arms. The control commands are used to control the joint motion data corresponding to each joint in the slave robotic arm. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm. The target robotic arm is at least one of the m slave robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms.

[0093] In some embodiments, the target robotic arm includes a reference robotic arm, and the second network includes joint mapping relationships of the m slave robotic arms relative to the reference robotic arm. These joint mapping relationships indicate the mapping relationship between joint motion data corresponding to each joint in the reference robotic arm and joint motion data corresponding to each joint in the corresponding slave robotic arm when they have the same base center coordinates. The mapping process based on the joint motion data of each joint in the target robotic arm to obtain control commands for each of the m slave robotic arms includes:

[0094] Based on the joint mapping relationship between the m slave robotic arms and the reference robotic arm, the joint motion data corresponding to each joint in the reference robotic arm is converted to obtain the joint motion data corresponding to each joint in the m slave robotic arms.

[0095] Based on the joint motion data corresponding to each joint in the m slave robotic arms, control commands are generated for each of the m slave robotic arms.

[0096] In some embodiments, the target robotic arm includes the m slave robotic arms, and the second network includes the joint mapping relationship of each of the m slave robotic arms. The joint mapping relationship is used to indicate the mapping relationship of the joint motion data corresponding to each joint in the slave robotic arm when they have the same base center coordinates. The step of mapping based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms includes:

[0097] Based on the joint mapping relationship of each of the m slave robotic arms, the joint motion data corresponding to each joint in the m slave robotic arms is converted to obtain the converted joint data corresponding to each joint in the m slave robotic arms.

[0098] Based on the transition joint data corresponding to each joint in the m slave robotic arms, control commands are generated for each of the m slave robotic arms.

[0099] In some embodiments,

[0100] The acquisition module 701 is further configured to acquire a first training set, the first training set including sample environment data captured by the shooting device and control motion data of each joint in the m slave robotic arms when the master control servo motor performs the sample task;

[0101] The processing module 702 is further configured to train the first network to be trained based on the first training set, so as to obtain the trained first network.

[0102] In some embodiments,

[0103] The acquisition module 701 is further configured to acquire a second training set, the second training set including at least the base center coordinates of each of the m slave robotic arms when the sample task is completed;

[0104] The processing module 702 is further configured to train the second network to be trained based on the second training set, so as to obtain the trained second network.

[0105] In some embodiments, the first training set and / or the second training set are preprocessed training sets, wherein the preprocessing includes at least one of data cleaning, standardization, and outlier handling.

[0106] In some embodiments, the processing module 702 is further configured to control the m slave robotic arms to synchronously execute their respective control commands in order to complete the target task.

[0107] In some embodiments, the image data further includes depth data; and / or, the m slave robotic arms are heterogeneous robotic arms.

[0108] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0109] This application also provides a computer-readable storage medium that can be used in the above-mentioned applications. Figure 1 or Figure 2 In the robot control system 10 shown, computer program instructions are stored on the computer storage medium. When these instructions are executed by a processor, they implement the steps of the robot control method provided in this application.

[0110] Please see Figure 8 This is a schematic diagram of the structure of a robot according to an exemplary embodiment. For example, the robot may include components such as the aforementioned main control servo motor 100, the aforementioned m slave robotic arms (represented as 201 to 20m in the diagram), and the aforementioned imaging device 300, etc. Figure 8 The robot shown may also include: a memory 801, a processor 802, and a computer program 803 stored in the memory 801 and running on the processor 802, wherein when the processor 802 executes the computer program 803, the robot can perform the robot control method described above.

[0111] This application embodiment also provides a robot control system, including a master control servo motor, m slave control robotic arms and a camera device, wherein the camera device is disposed on at least one of the slave control robotic arms, and the m slave control robotic arms are controlled by the master control servo motor, where m is a positive integer.

[0112] After the m slave robotic arms are powered on, the shooting device installed on the slave robotic arms is powered on.

[0113] When the m slave robotic arms support communication, the imaging device is activated to capture corresponding environmental image data;

[0114] The aforementioned robot control system is used to execute the robot control method embodiments described in the foregoing of this application. For example, the environmental image data is input into the target model for processing to obtain control instructions for each of the m slave robotic arms. The control instructions are used to control the joint motion data corresponding to each joint in the slave robotic arms. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control instructions for each of the m slave robotic arms, etc. For any content not introduced or described in the embodiments of this application, please refer to the relevant descriptions in the foregoing method embodiments. This application does not limit the scope of the embodiments.

[0115] In specific implementation, the system may include a memory and a processor. The memory stores executable program code, and the processor is used to call and execute the executable program code to execute the robot control method provided in this application embodiment. This application embodiment can divide the system into functional modules based on the above method example. For example, each module can correspond to a specific function, or two or more functions can be integrated into a single processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division; other division methods may exist in actual implementation. It should also be noted that all relevant content of each step involved in the above method embodiment can be referenced to the functional description of the corresponding functional module, and will not be repeated here.

[0116] It should be understood that the system provided in this application embodiment is used to execute the robot control method described above, and therefore can achieve the same effect as the above-described implementation method. When using integrated units, the system may include a processing module and a storage module. When the system is applied to a robot, the processing module can be used to control and manage the robot's movements. The storage module can be used to support the robot in executing mutual program code, etc. The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of Digital Signal Processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0117] In addition, the system provided in this application embodiment may also include a chip, which may include a connected processor and a memory; wherein, the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a robot control method provided in the above embodiment.

[0118] In another exemplary embodiment, a computer program product is also provided, which can be applied as described above. Figure 1 or Figure 2 The robot control system 10 shown includes a computer program product that can be executed by a programmable device, the computer program having code portions for performing the robot control method described above when executed by the programmable device.

[0119] Please see Figure 9 This is a schematic diagram illustrating the structure of a chip according to an exemplary embodiment. For example... Figure 9 The chip 900 shown can be applied to the above. Figure 1 or Figure 2 In the robot control system 10 shown, the chip 900 may include a processor 901 and an interface 902. Optionally, it may also include a memory 903. The number of processors 901 may be one or more, and the number of interfaces 902 may be multiple.

[0120] In one embodiment, regarding the case where the chip is used to implement the method embodiments described in this application:

[0121] The interface 902 is used to receive or output signals;

[0122] The processor 901 is used to execute some or all of the contents of the robot control method embodiment.

[0123] Understandably, the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0124] Understandably, the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0125] It should be noted that the descriptions of the storage medium, system, and chip embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0126] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of this application. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed in this application. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0127] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A robot control method, characterized in that, The robot includes a master control servo motor, m slave control robotic arms, and a camera device. The camera device is mounted on at least one of the slave control robotic arms. The m slave control robotic arms are controlled by the master control servo motor, where m is a positive integer. The method includes: The environmental image data captured by the imaging device is used to reflect the target scene where the robot is located and / or the target task to be performed. The environmental image data is input into the target model for processing to obtain control commands for each of the m slave robotic arms. The control commands are used to control the joint motion data corresponding to each joint in the slave robotic arm. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm, and the target robotic arm is at least one of the m slave robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms. The target robotic arm includes a reference robotic arm. The second network includes joint mapping relationships of the m slave robotic arms relative to the reference robotic arm. These joint mapping relationships indicate the mapping relationship between the joint motion data corresponding to each joint in the reference robotic arm and the joint motion data corresponding to each joint in the corresponding slave robotic arm when they have the same base center coordinates. The mapping processing based on the joint motion data of each joint in the target robotic arm to obtain the control commands for each of the m slave robotic arms includes: Based on the joint mapping relationship between the m slave robotic arms and the reference robotic arm, the joint motion data corresponding to each joint in the reference robotic arm is converted to obtain the joint motion data corresponding to each joint in the m slave robotic arms. Based on the joint motion data corresponding to each joint in the m slave robotic arms, control commands are generated for each of the m slave robotic arms. The target robotic arm includes m slave robotic arms, and the second network includes joint mapping relationships for each of the m slave robotic arms. These joint mapping relationships indicate the mapping relationships of joint motion data corresponding to each joint in the slave robotic arms when they have the same base center coordinates. The mapping processing based on the joint motion data of each joint in the target robotic arm to obtain control commands for each of the m slave robotic arms includes: Based on the joint mapping relationship of each of the m slave robotic arms, the joint motion data corresponding to each joint in the m slave robotic arms is converted to obtain the converted joint data corresponding to each joint in the m slave robotic arms. Based on the transition joint data corresponding to each joint in the m slave robotic arms, control commands are generated for each of the m slave robotic arms.

2. The robot control method according to claim 1, characterized in that, The method further includes: Obtain a first training set, which includes sample environment data captured by the imaging device and control motion data of the master control servo motor on each joint of the m slave control robotic arms when performing sample tasks; The first network to be trained is trained based on the first training set to obtain the trained first network.

3. The robot control method according to claim 1, characterized in that, The method further includes: Obtain a second training set, which includes at least the base center coordinates of each of the m slave robotic arms when the sample task is completed; The second network to be trained is trained based on the second training set to obtain the trained second network.

4. The robot control method according to claim 2 or 3, characterized in that, Training Session 1 The training set and / or the second training set are preprocessed training sets, wherein the preprocessing includes at least one of data cleaning, standardization, and outlier handling.

5. The robot control method according to claim 4, characterized in that, The method also includes include: The m slave robotic arms are controlled to synchronously execute their respective control commands in order to complete the target task.

6. The robot control method according to claim 4, characterized in that, The environmental image The data also includes depth data; and / or, the m slave robotic arms are heterogeneous robotic arms.

7. A robot, characterized in that, Includes a master servo motor, m slave robotic arms, and a camera. The camera is mounted on at least one of the slave robotic arms, and the m slave robotic arms are controlled by the master servo motor, where m is a positive integer. The robot also includes: processor; Memory used to store processor-executable instructions; The processor is configured to execute the executable instructions to implement the steps of the robot control method according to any one of claims 1 to 6.

8. A robot control system, characterized in that, The step of performing the robot control method as described in any one of claims 1 to 6 includes a master servo motor, m slave robotic arms and a camera device, wherein the camera device is disposed on at least one of the slave robotic arms, and the m slave robotic arms are controlled by the master servo motor, where m is a positive integer. After the m slave robotic arms are powered on, the shooting device installed on the slave robotic arms is powered on. When the m slave robotic arms support communication, the imaging device is activated to capture corresponding environmental image data. The environmental image data is used to reflect the target scene where the robot is located and / or the target task to be performed. The environmental image data is input into the target model for processing to obtain control commands for each of the m slave robotic arms. The control commands are used to control the joint motion data corresponding to each joint in the slave robotic arm. The target model includes a first network and a second network. The first network is used to calculate the joint motion data corresponding to each joint in the target robotic arm. The target robotic arm is at least one of the m slave robotic arms. The second network is used to perform mapping processing based on the joint motion data corresponding to each joint in the target robotic arm to obtain the control commands of each of the m slave robotic arms.

Images