A robot training method, device, system, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a robot training method, device, system, electronic equipment and storage medium, relating to the technical field of robots, and the method comprises: acquiring a first signal and a second signal transmitted by a plurality of millimeter wave antennas, the second signal comprising a return signal of the first signal; generating a three-dimensional image of a training personnel according to the first signal and the second signal, the training personnel being located within a coverage range of at least part of the plurality of millimeter wave antennas; detecting the three-dimensional image to obtain target motion information of a target key point of the training personnel; and controlling a target joint point associated with the target key point to move according to the target motion information, according to an association between the key point of the training personnel and the joint point of the robot. The scheme can reduce the training cost of the robot and improve the training accuracy of the robot.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a robot training method, apparatus, system, electronic device, and storage medium. Background Technology

[0002] Before robots are put into use, they need to be trained to accurately complete designated tasks. Currently, robot training is mainly based on exoskeleton devices or virtual reality (VR) devices. However, when using exoskeleton devices to train robots, a customized exoskeleton device needs to be created for each type of robot and each person training the robot (i.e., the trainer), resulting in high training costs. When using VR devices to train robots, network interference can cause delays in the robot's image received by the trainer through the VR device, which obstructs the trainer's field of vision and reduces the accuracy of the robot training. Summary of the Invention

[0003] The purpose of this application is to provide a robot training method, apparatus, system, electronic device, and storage medium to reduce robot training costs and improve robot training accuracy. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a robot training method, the method comprising:

[0005] Acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal;

[0006] A three-dimensional image of the trainee is generated based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas of the plurality of millimeter-wave antennas;

[0007] The target motion information of the target key points of the trainee is obtained by detecting the three-dimensional image.

[0008] Based on the association between the trainer's key points and the robot's joints, and according to the target motion information, the movement of the target joints associated with the target key points is controlled.

[0009] In some embodiments, the step of detecting the three-dimensional image to obtain the target motion information of the target key points of the trainee includes:

[0010] Determine the first region where the target key points are located in the three-dimensional image;

[0011] Search for the target key points within the first region to obtain the target motion information of the target key points.

[0012] In some embodiments, the step of determining the first region where the target key points are located in the three-dimensional image includes:

[0013] Obtain a two-dimensional image of the trainee;

[0014] Identify the second region in the two-dimensional image where the target key point is located;

[0015] Based on the mapping relationship between the two-dimensional image and the three-dimensional image, the first region corresponding to the second region in the three-dimensional image is determined.

[0016] In some embodiments, the step of determining the first region where the target key points are located in the three-dimensional image includes:

[0017] Obtain the historical motion information of the target key points;

[0018] Based on the historical motion information, predict the first region where the target key point is located in the three-dimensional image.

[0019] In some embodiments, the three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time, and a second three-dimensional image corresponding to the signal acquired previously;

[0020] The step of detecting the three-dimensional image to obtain the target motion information of the target key points of the trainee includes:

[0021] By comparing the target key points of the trainee in the first three-dimensional image and the second three-dimensional image, the target motion information of the target key points is obtained.

[0022] In some embodiments, the step of detecting the three-dimensional image to obtain the target motion information of the target key points of the trainee includes:

[0023] The coordinates of the target key points of the trainee in the three-dimensional image are obtained as the target motion information of the target key points of the trainee.

[0024] In some embodiments, the step of controlling the movement of the target joint associated with the target key points according to the target motion information based on the association between the trainer's key points and the robot's joints includes:

[0025] Based on the relationship between the trainer's key points and the robot's joints, the target joints associated with the target key points on the robot are determined.

[0026] Based on the target motion information, control the movement of the target joints on each robot.

[0027] In some embodiments, the target joints include points on the robot that have rotational degrees of freedom; the target key points include the skeletal points of the trainer and / or points obtained based on multiple associated skeletal points.

[0028] Secondly, embodiments of this application provide a robot training device, the device comprising:

[0029] An acquisition module is used to acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal;

[0030] A generation module is configured to generate a three-dimensional image of a trainee based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas of the plurality of millimeter-wave antennas;

[0031] The detection module is used to detect the three-dimensional image and obtain the target motion information of the target key points of the trainee;

[0032] The control module is used to control the movement of the target joints associated with the target key points according to the relationship between the trainer's key points and the robot's joints and the target motion information.

[0033] In some embodiments, the detection module is specifically used for:

[0034] Determine the first region where the target key points are located in the three-dimensional image;

[0035] Search for the target key points within the first region to obtain the target motion information of the target key points.

[0036] In some embodiments, the detection module is specifically used for:

[0037] Obtain a two-dimensional image of the trainee;

[0038] Identify the second region in the two-dimensional image where the target key point is located;

[0039] Based on the mapping relationship between the two-dimensional image and the three-dimensional image, the first region corresponding to the second region in the three-dimensional image is determined.

[0040] In some embodiments, the detection module is specifically used for:

[0041] Obtain the historical motion information of the target key points;

[0042] Based on the historical motion information, predict the first region where the target key point is located in the three-dimensional image.

[0043] In some embodiments, the three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time, and a second three-dimensional image corresponding to the signal acquired previously;

[0044] The detection module is specifically used to compare the target key points of the trainee in the first three-dimensional image and the second three-dimensional image to obtain the target motion information of the target key points.

[0045] In some embodiments, the detection module is specifically used to obtain the coordinates of the target key points of the trainee in the three-dimensional image, as the target motion information of the target key points of the trainee.

[0046] In some embodiments, the control module is specifically used for:

[0047] Based on the relationship between the trainer's key points and the robot's joints, the target joints associated with the target key points on the robot are determined.

[0048] Based on the target motion information, control the movement of the target joints on each robot.

[0049] In some embodiments, the target joints include points on the robot that have rotational degrees of freedom; the target key points include the skeletal points of the trainer and / or points obtained based on multiple associated skeletal points.

[0050] Thirdly, embodiments of this application provide a robot training system, including multiple millimeter-wave antennas and electronic devices;

[0051] The plurality of millimeter-wave antennas are used to transmit a first signal and receive a second signal, the second signal including the echo signal of the first signal; and to transmit the first signal and the second signal to the electronic device;

[0052] The electronic device is configured to generate a three-dimensional image of a trainee based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas among the plurality of millimeter-wave antennas; detect the three-dimensional image to obtain target motion information of the target key points of the trainee; and control the movement of the target key points associated with the target joints according to the target motion information based on the association between the key points of the trainee and the joints of the robot.

[0053] In some embodiments, the plurality of millimeter-wave antennas are located above the activity area of ​​the trainee, and the projection of the plurality of millimeter-wave antennas onto the plane of the activity area is located at the boundary of the activity area.

[0054] In some embodiments, the electronic device is specifically configured to: determine a first region in the three-dimensional image where the target key point is located; search for the target key point in the first region to obtain target motion information of the target key point.

[0055] In some embodiments, the system further includes a camera;

[0056] The camera is used to capture two-dimensional images of the trainees;

[0057] The electronic device is specifically used for: acquiring a two-dimensional image of the trainee captured by the camera; identifying a second region in the two-dimensional image where the target key point is located; and determining a first region in the three-dimensional image corresponding to the second region based on the mapping relationship between the two-dimensional image and the three-dimensional image.

[0058] In some embodiments, the camera is located above the activity area of ​​the trainee, and the projection of the camera onto the plane of the activity area is located at the boundary of the activity area.

[0059] In some embodiments, the electronic device is specifically configured to: acquire historical motion information of the target key point; and predict a first region in the three-dimensional image where the target key point is located based on the historical motion information.

[0060] In some embodiments, the three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time, and a second three-dimensional image corresponding to the signal acquired previously;

[0061] The electronic device is specifically used to: compare the target key points of the trainee in the first three-dimensional image and the second three-dimensional image to obtain the target motion information of the target key points.

[0062] In some embodiments, the electronic device is specifically used to: acquire the coordinates of the target key points of the trainee in the three-dimensional image, as the target motion information of the target key points of the trainee.

[0063] In some embodiments, the target joints include points on the robot that have rotational degrees of freedom; the target key points include the skeletal points of the trainer and / or points obtained based on multiple associated skeletal points.

[0064] Fourthly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0065] Memory, used to store computer programs;

[0066] The processor, when executing a program stored in memory, implements any of the robot training methods described above.

[0067] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the robot training methods described above.

[0068] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the robot training methods described above.

[0069] Beneficial effects of the embodiments in this application:

[0070] In the technical solution provided in this application embodiment, the key points of the trainer are associated with the joints of the robot. The trainer is located within the coverage area of ​​multiple millimeter-wave antennas. Signals emitted by the millimeter-wave antennas are used to generate a three-dimensional image of the trainer, and then the target motion information of the target key points is obtained from this three-dimensional image. This information is used to control the movement of the target joints and complete the robot training. In this application embodiment, the key points of the trainer are directly associated with the joints of the robot. As long as the trainer is within the coverage area of ​​the millimeter-wave antennas, the corresponding key points can be collected, completing the robot training. No specialized customized equipment is required, reducing the robot training cost. Furthermore, the millimeter-wave antenna signal has low latency and high accuracy, and the trainer's field of vision is not obstructed during robot training, improving the robot training accuracy.

[0071] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0073] Figure 1 This is a schematic diagram of a first structure of a robot training system provided in an embodiment of this application;

[0074] Figure 2a A first schematic diagram illustrating the coverage area of ​​a millimeter-wave antenna provided in an embodiment of this application;

[0075] Figure 2b A second schematic diagram illustrating the coverage area of ​​the millimeter-wave antenna provided in an embodiment of this application;

[0076] Figure 2c A third schematic diagram illustrating the coverage area of ​​the millimeter-wave antenna provided in this application embodiment;

[0077] Figure 2d A fourth schematic diagram illustrating the coverage area of ​​the millimeter-wave antenna provided in this application embodiment;

[0078] Figure 2e A fifth schematic diagram illustrating the coverage area of ​​the millimeter-wave antenna provided in this application embodiment;

[0079] Figure 3a A three-dimensional schematic diagram of the mounting positions of the millimeter-wave antenna and camera provided in the embodiments of this application;

[0080] Figure 3b A top view of the mounting position of the millimeter-wave antenna and camera provided in an embodiment of this application;

[0081] Figure 4 This is a schematic diagram of a second structure of the robot training system provided in the embodiments of this application;

[0082] Figure 5 A schematic diagram illustrating the coverage area of ​​a camera provided in an embodiment of this application;

[0083] Figure 6 A schematic diagram illustrating the determination of a first region provided in an embodiment of this application;

[0084] Figure 7 A signaling diagram for interaction between the robot training system provided in the embodiments of this application;

[0085] Figure 8a This is a first schematic diagram illustrating the joint movement of a robot according to an embodiment of this application.

[0086] Figure 8b This is a second schematic diagram illustrating the movement of joints in a robot according to an embodiment of this application.

[0087] Figure 9 This is a schematic diagram of a first flowchart of a robot training method provided in an embodiment of this application;

[0088] Figure 10 This is a second flowchart illustrating the robot training method provided in the embodiments of this application;

[0089] Figure 11 A schematic diagram of a robot training device provided in an embodiment of this application;

[0090] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0091] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0092] Remote control: Users control the robot's movement and task performance using a remote control, computer, or mobile device. Remote control, also known as teleoperation, is a common method used for training robots.

[0093] Currently, there are two main methods for training robots:

[0094] The first method involves training the robot using an exoskeleton device.

[0095] In this training method, the trainee wears an exoskeleton device on their body, performs actions, and the exoskeleton device collects the actions performed by the trainee and then controls the robot to perform corresponding actions.

[0096] In this training method, exoskeleton devices are worn by trainees. However, trainees have different body shapes, which requires custom-made exoskeleton devices for different trainees, resulting in poor adaptability of the exoskeleton devices to the trainees.

[0097] Furthermore, different robot models exhibit variations in their degrees of freedom and parameters. For instance, one robot might have a dexterous hand as its end effector, possessing five fingers with nine degrees of freedom; while another might have a suction cup as its end effector, with only one degree of freedom. Similarly, a robot arm can have seven degrees of freedom, typically manifested in the joints, but different robot models have different specifications, such as variations in the placement of the joints. Therefore, training different robot models requires custom-designed exoskeleton devices tailored to each model.

[0098] Customizing exoskeleton equipment for each type of robot and each trainer increases the training cost of the robot.

[0099] The second method involves training the robot using VR devices.

[0100] In this training method, the VR equipment includes a head-mounted headset and controllers. The trainee wears the headset and holds the controllers. A camera captures images of the robot and transmits them to the headset; the headset displays the robot's image; the trainee manipulates the controllers based on the image displayed on the headset, thereby controlling the robot to perform corresponding actions.

[0101] In this training method, the camera transmits the robot's posture to the head-mounted device via the network, allowing the trainee to see the robot's posture and control its movements. However, the network status changes in real time, and due to network issues, there is a delay in the trainee receiving the robot's image through the VR device. This obstructs the trainee's field of vision and reduces the accuracy of robot training.

[0102] To address the aforementioned problems, embodiments of this application provide a robot training system, such as... Figure 1 As shown, it includes multiple millimeter-wave antennas 11 and electronic devices 12;

[0103] Multiple millimeter-wave antennas 11 are used to transmit a first signal to the trainees and receive a second signal, the second signal being the echo signal of the first signal; and to transmit the first and second signals to an electronic device 12.

[0104] Electronic device 12 is used to generate a three-dimensional image of the trainee based on the first signal and the second signal; to detect the three-dimensional image to obtain the target motion information of the target key points of the trainee; and to control the movement of the target key points associated with the target joints according to the correlation between the key points of the trainee and the joints of the robot.

[0105] It should be noted that the electronic device 12 can be a standalone physical device or integrated into the robot. Therefore, when the electronic device 12 is integrated into the robot, the robot training system can be considered as including multiple millimeter-wave antennas 11 and the robot itself.

[0106] In the technical solution provided in this application embodiment, the key points of the trainer are associated with the joints of the robot. The trainer is located within the coverage area of ​​multiple millimeter-wave antennas. Signals emitted by the millimeter-wave antennas are used to generate a three-dimensional image of the trainer, and then the target motion information of the target key points is obtained from this three-dimensional image. This information is used to control the movement of the target joints and complete the robot training. In this application embodiment, the key points of the trainer are directly associated with the joints of the robot. As long as the trainer is within the coverage area of ​​the millimeter-wave antennas, the corresponding key points can be collected, completing the robot training. No specialized customized equipment is required, reducing the robot training cost. Furthermore, the millimeter-wave antenna signal has low latency and high accuracy, and the trainer's field of vision is not obstructed during robot training, improving the robot training accuracy.

[0107] In this embodiment, the multiple millimeter-wave antennas 11 and the electronic device 12 can be connected via wired or wireless means, without limitation. To improve the accuracy of the 3D image, the more millimeter-wave antennas 11, the better. Conversely, to reduce computational complexity, improve the efficiency of 3D image generation, and enhance robot training efficiency, the fewer millimeter-wave antennas 11, the better. The final number of millimeter-wave antennas 11 deployed is typically determined by comprehensively considering at least two requirements from multiple dimensions, including environmental factors, imaging accuracy, imaging efficiency, computational power, and training efficiency.

[0108] When training the robot, the trainer moves within a designated activity area. In this embodiment, the coverage areas of the multiple millimeter-wave antennas 11 may be the same or different.

[0109] When multiple millimeter-wave antennas 11 have the same coverage area, the active area is located within the coverage area of ​​each millimeter-wave antenna 11, meaning the coverage area of ​​each millimeter-wave antenna 11 is greater than or equal to the active area, in order to accurately acquire three-dimensional images of the trainees. Figure 2a As shown, the activity area of ​​the trainees is within the coverage area of ​​millimeter-wave antennas a to d.

[0110] When the coverage areas of multiple millimeter-wave antennas 11 are different, the location of the active area can be divided into the following cases.

[0111] Scenario 1: The activity area lies within the intersection of the coverage areas of multiple millimeter-wave antennas 11, in order to accurately acquire three-dimensional images of the trainees. For example... Figure 2b As shown, the intersection of the coverage areas of millimeter-wave antennas a to d is region A, and the active area is located within region A.

[0112] Scenario 2: The activity area lies within the intersection of the coverage areas of several millimeter-wave antennas 11, in order to accurately acquire three-dimensional images of the trainees. For example... Figure 2c As shown, among millimeter-wave antennas a to d, the intersection of the coverage areas of millimeter-wave antennas a to c is region B, and the active area is located within region B.

[0113] Scenario 3: The activity area lies within the union of the coverage areas of multiple millimeter-wave antennas 11, thereby expanding the activity range of the trainees. For example... Figure 2d As shown, the union of the coverage areas of millimeter-wave antennas a to d includes the active area.

[0114] Case 4: The activity area is located within the union of the coverage areas of some millimeter-wave antennas 11, to accommodate robot training with a small range of movement for trainees, reduce the number of signals processed by subsequent electronic devices 12, and lower the burden on electronic devices 12. For example... Figure 2e As shown, among millimeter-wave antennas a to d, the union of the coverage areas of millimeter-wave antennas a to b includes the active area.

[0115] By deploying multiple millimeter-wave antennas 11 in the activity area with the same or different coverage areas, the training personnel can be located within the coverage area of ​​at least a portion of the multiple millimeter-wave antennas 11.

[0116] In one example, multiple millimeter-wave antennas 11 are positioned above the activity area of ​​the trainee, and the projections of the multiple millimeter-wave antennas 11 onto the plane of the activity area are located at the boundary of the activity area. This ensures that a complete image of the trainee is acquired.

[0117] For example, the robot training system includes four millimeter-wave antennas 11, and the training area is such as... Figure 3a and Figure 3b The quadrilateral region shown has four millimeter-wave antennas 11 positioned above the active area, with their projections onto the plane of the active area located at the four vertices. This allows the millimeter-wave antennas 11 to transmit signals from the top, receiving signal feedback after striking the trainee.

[0118] In this embodiment, the multiple millimeter-wave antennas 11 may be partially located above the activity area of ​​the trainee, partially located to the side of the trainee, or the multiple millimeter-wave antennas 11 may be located to the side of the trainee. The specific deployment position of the multiple millimeter-wave antennas 11 is not limited, as long as the collected signal can generate a three-dimensional image of the trainee.

[0119] The millimeter-wave antenna 11 can simultaneously deploy a transmitter and a receiver. The transmitter of the millimeter-wave antenna 11 transmits a signal (such as a first signal) to the activity area. A trainee located within the activity area blocks the first signal, thereby reflecting the first signal and forming an echo signal of the first signal, i.e., a second signal. The receiver of the millimeter-wave antenna 11 receives the second signal, which includes the echo signal of the first signal transmitted by the millimeter-wave antenna 11 itself, and may also include the echo signal of the first signal transmitted by other millimeter-wave antennas 11. After acquiring the first and second signals, the millimeter-wave antenna 11 can directly transmit the first and second signals to the electronic device 12, or it can process the first and second signals to generate a differential signal and transmit the differential signal to the electronic device 12. The differential signal can reflect the difference between the transmission time of the first signal and the reception time of the second signal, as well as the energy difference, thereby determining the trainee, and the distance between the corresponding point on the trainee and the millimeter-wave antenna 11; based on this distance, the electronic device 12 can generate a three-dimensional image of the trainee.

[0120] Millimeter-wave antennas have the following characteristics: 1) narrow beamwidth, which allows for the resolution of small targets at closer distances or clearer observation of target details; 2) small size, making them convenient for deployment in indoor settings. Utilizing these characteristics, millimeter-wave antennas can capture highly precise human movements, facilitating accurate robot training.

[0121] In this embodiment, key points are the points where the trainer and the robot are associated with each other's joints, and joints are points on the robot with rotational degrees of freedom. Key points include the trainer's skeletal points and / or points obtained from multiple associated skeletal points. Associated skeletal points can be multiple adjacent skeletal points, or multiple skeletal points that are mutually affected when performing a certain action.

[0122] For example, the robot's end effector is a dexterous hand with five fingers. The distal joint of each finger is associated with the skeletal point at the end of the trainer's finger, that is, the key points include the trainer's skeletal points.

[0123] For example, the robot's end effector is a gripper with two fingers. The distal joint of one finger can be associated with the bone point at the end of the trainee's thumb, and the distal joint of the other finger can be associated with the middle position of the bone points at the ends of the trainee's other four fingers. The bone points at the ends of the trainee's other four fingers are the associated bone points, and the middle position of the bone points at the ends of the trainee's other four fingers is the point calculated based on multiple associated bone points. That is, the key point includes the trainee's bone points and the point obtained based on multiple associated bone points.

[0124] For example, the robot's end effector is a suction cup, which does not have fingers. The middle joint of the suction cup can be associated with the middle position of the skeletal points at the ends of the trainee's five fingers. The skeletal points at the ends of the trainee's five fingers are associated skeletal points, and the middle position of the skeletal points at the ends of the trainee's five fingers is a point calculated based on multiple associated skeletal points. That is, the key point includes the point calculated based on the skeletal points.

[0125] In this embodiment, the key points of the trainer are associated with the joints of the robot, which can eliminate the differences in the rotational degrees of freedom and parameters of the joints caused by different robot models.

[0126] For example, two robot models may have the same number of degrees of freedom in their arms, each with seven degrees of freedom, but the distance parameters between the joints on the arms are different. In one robot, the distance between the wrist and elbow joints is L1, while in the other robot it is L2. In this embodiment, the wrist joints of the two robots are associated with the skeletal points of the trainer's wrist, and the elbow joints of the two robots are associated with the skeletal points of the trainer's elbow.

[0127] In this embodiment, the differences in parameters and degrees of freedom among different robot models also include the shoulders, waist, and lower body limbs. Regardless of whether the robot models are the same, it is sufficient to associate the trainee's key points with the robot's joints. For example, if the robot's lower body limbs are two legs, the trainee's feet, knees, waist, and other key points can be associated with the robot's feet, legs, waist, and other joints, respectively. If the robot's lower body limbs are wheeled, the trainee's knee key points can be associated with the robot's lower body joints. Further details will not be elaborated upon.

[0128] Motion information may include one or more of the following: direction of movement, distance of movement, speed of movement, and position coordinates.

[0129] After acquiring signals emitted by multiple millimeter-wave antennas 11, the electronic device 12 uses these signals to generate a three-dimensional image of the trainee. In this embodiment, the electronic device 12 can generate a three-dimensional image using the first and second signals from all millimeter-wave antennas 11, or it can acquire the first and second signals from a portion of the millimeter-wave antennas 11 based on the trainee's activity area and use these signals to generate a three-dimensional image; there is no limitation on this.

[0130] After generating the 3D image, the electronic device 12 detects the 3D image to obtain the motion information of the target key points of the trainee (i.e., target motion information). In this embodiment, the electronic device 12 can use any of the following methods to detect the 3D image and obtain the target motion information.

[0131] Method 1: Determine the first region in the 3D image where the target key point is located; search for the target key point within the first region to obtain the target motion information of the target key point.

[0132] In Method 1, the electronic device 12 may determine the first region using any of the following methods.

[0133] Method 1.1: Determine the first region using a two-dimensional image.

[0134] In this embodiment of the application, the robot training system may further include at least one camera 13, such as Figure 4 As shown. Camera 13 can be a Red Green Blue (RGB) camera, a YUV camera (YUV color encoding uses luminance and chrominance, where Y represents luminance (Luma), i.e., grayscale value, and U and V represent chrominance (Chrominance), used to describe hue and saturation), or other types of cameras. There are no restrictions on this, as long as it can acquire two-dimensional images of the trainees.

[0135] To accurately acquire two-dimensional images of the trainees and assist in determining the first region, the activity area is located within the coverage area of ​​each camera 13, meaning the coverage area of ​​camera 13 is greater than or equal to the activity area. Figure 5 As shown. In one example, camera 13 is positioned above the activity area of ​​the trainee, and the projection of camera 13 onto the plane of the activity area is located at the boundary of the activity area. This ensures that a complete image of the trainee is captured.

[0136] For example, the robot training system includes four cameras (13), and the training area is as follows: Figure 3a and Figure 3b The quadrilateral area shown has four cameras 13 positioned above the activity area, with their projections onto the plane of the activity area located at the four vertices. This allows the cameras 13 to capture two-dimensional images of the trainees from above, avoiding occlusion.

[0137] In this embodiment, the camera 13 can also be deployed to the side of the trainee, as long as the two-dimensional image captured by the camera 13 can eliminate occlusion and detect the trainee's key points. For example, if the trainee's key points are located in front of the trainee, one or more cameras 13 can be deployed on one side of the trainee's front to capture a two-dimensional image of the trainee's front; or, if the trainee's key points are located in both the front and back of the trainee, one or more cameras 13 can be deployed on one side of the front and one side of the back of the trainee respectively to capture a two-dimensional image of the trainee's front and a two-dimensional image of the trainee's back; or, if the trainee rotates and moves within the activity area, multiple cameras 13 can be deployed around the activity area, and the multiple frames of two-dimensional images captured by the multiple cameras 13 can be processed to obtain an occluded two-dimensional image, such as a panoramic image. The specific deployment location of the camera 13 is not limited.

[0138] Furthermore, in this embodiment, camera 13 can be deployed as a standalone physical device, or it can be integrated into electronic device 12 or a robot. Wherein, camera 13 integrated into a robot can be understood as: using a camera deployed on the robot itself to acquire two-dimensional images to assist in determining the first region.

[0139] For example, when training multiple robots, cameras on some robots can be used to capture two-dimensional images to help determine a first region for training other robots. Here, to improve the quality of the captured two-dimensional images, electronic device 12 can adjust the robot's pose so that the camera on the robot captures two-dimensional images that meet the requirements. For example, if the camera is mounted on the robot's hand, electronic device 12 can adjust the robot's arm to extend upwards, and simultaneously adjust the lower half of the robot to extend upwards, thereby increasing the camera's height to achieve a top-down view of the trainee and obtain an unobstructed two-dimensional image.

[0140] In method 1.1, the electronic device 12 can preset the mapping relationship between the two-dimensional image and the three-dimensional image, such as the calibration relationship between the coordinate system of the two-dimensional image and the coordinate system of the three-dimensional image, or the association relationship between points in the two-dimensional image and regions in the three-dimensional image. The electronic device 12 can also perform calibration after acquiring the two-dimensional image to obtain the mapping relationship between the two-dimensional image and the three-dimensional image.

[0141] Camera 13 can periodically acquire two-dimensional images of trainees. After acquiring the two-dimensional images, if camera 13 is deployed as an independent physical device, it can send the acquired two-dimensional images to electronic device 12. If camera 13 is integrated into electronic device 12, electronic device 12 can directly acquire the two-dimensional images acquired by camera 13. Electronic device 12 acquires the two-dimensional images of trainees acquired by camera 13; identifies the region (such as the second region) where the target key points are located in the two-dimensional image; based on the mapping relationship between the two-dimensional image and the three-dimensional image, electronic device 12 can determine the first region corresponding to the second region in the three-dimensional image.

[0142] In this embodiment, the electronic device 12 can utilize a convolutional neural network or other image recognition algorithms to accurately identify the second region where the target key point is located from a two-dimensional image. The second region is a region in the two-dimensional image. According to the mapping relationship between two-dimensional and three-dimensional images, the electronic device 12 can map the second region onto the three-dimensional image to obtain the corresponding first region. At this time, the electronic device 12 reduces the search area of ​​the target key point from the entire three-dimensional image to the first region, thereby reducing the search area of ​​the target key point and improving both the search efficiency and accuracy.

[0143] Method 1.2: Predict the first region using historical motion information.

[0144] In this embodiment, the three-dimensional image of the trainee generated based on a single acquired signal is simply referred to as the three-dimensional image corresponding to that acquired signal. Each time the electronic device 12 acquires a three-dimensional image of the trainee, it can obtain motion information of the target key points from the three-dimensional image.

[0145] After acquiring the three-dimensional image of the trainee, the electronic device 12 can obtain the historical motion information of the target key points, such as the direction of movement and the speed of movement; based on the historical motion information, it can predict the first region where the target key points are located in the three-dimensional image.

[0146] Taking the three-dimensional image corresponding to the signal acquired this time as the first three-dimensional image and the three-dimensional image corresponding to the signal acquired last time as the second three-dimensional image as an example, after acquiring the first three-dimensional image of the trainee this time, the electronic device 12 can acquire the motion information of the target key points in the second three-dimensional image as historical motion information; based on the historical motion information, predict the first region where the target key points are located in the first three-dimensional image.

[0147] For example, electronic device 12 acquired three-dimensional image 1 and three-dimensional image 2 successively. Figure 6In the image acquisition scenario shown, after acquiring 3D image 1, electronic device 12 acquires motion information such as the movement direction (as shown by the arrow) and movement speed v of key point P (i.e., the target key point) based on 3D image 1. After acquiring 3D image 2, electronic device 12 predicts the distance L that key point P will move in the movement direction shown by the arrow, based on the movement direction and movement speed v, and the frame rate of the 3D image, thus obtaining point P'. The region Q with a preset radius r centered on point P' is then used as the first region.

[0148] In this embodiment, the electronic device 12 predicts the location of the target key point in the first region based on historical motion information, reducing the search area for the target key point from the entire 3D image to the first region. This reduces the search area for the target key point, improving both the search efficiency and accuracy. Furthermore, it eliminates the need for additional cameras or other equipment, reducing the equipment cost of the robot training system.

[0149] Method 1.3 uses two-dimensional images and historical motion information to determine the first region.

[0150] In this embodiment, the electronic device 12 can combine methods 1.1 and 1.2 described above to determine the first region, thereby further improving the search efficiency and accuracy of the target key points. See details below. Figure 7 The signaling diagram shown illustrates the interaction between the robot training system and other systems.

[0151] In step S71, the multiple millimeter-wave antennas 11 transmit a first signal at a first frequency and receive a second signal.

[0152] In step S72, the multiple millimeter-wave antennas 11 transmit a first signal and a second signal to the electronic device 12.

[0153] In step S73, the electronic device 12 generates a three-dimensional image of the trainee based on the first signal and the second signal.

[0154] Here, multiple millimeter-wave antennas 11 transmit a first signal at a first frequency, and correspondingly, electronic devices 12 generate a three-dimensional image of the trainee at the first frequency.

[0155] In step S74, camera 13 acquires two-dimensional images of the trainees at a second frequency. The first frequency is greater than the second frequency.

[0156] In this embodiment of the application, the execution order of steps S71 and S74 is not limited.

[0157] In step S75, camera 13 sends a two-dimensional image to electronic device 12.

[0158] Here, camera 13 acquires two-dimensional images of the trainees at a second frequency, and correspondingly, electronic device 12 receives the two-dimensional images at the same second frequency. Since the first frequency is greater than or equal to the second frequency, the number of two-dimensional images processed by electronic device 12 is less than or equal to the number of three-dimensional images, thus reducing the workload of electronic device 12.

[0159] In step S76, after generating the three-dimensional image, the electronic device 12 determines whether it has received the two-dimensional image; if yes, then proceed to step S77; if no, then proceed to step S79.

[0160] In step S77, the electronic device 12 identifies the second region where the target key point is located in the two-dimensional image.

[0161] In step S78, the electronic device 12 determines the first region corresponding to the second region in the three-dimensional image based on the mapping relationship between the two-dimensional image and the three-dimensional image. Then, step S711 is executed.

[0162] In step S76, if a two-dimensional image is received simultaneously with the generation of the three-dimensional image, the electronic device 12 executes steps S77 to S78 to determine the first region using the two-dimensional image, thereby improving the accuracy of determining the target key points and eliminating errors in historical motion information. The specific process for determining the first region using a two-dimensional image can be found in the relevant description in Method 1.1 above, and will not be repeated here.

[0163] In step S79, the electronic device 12 acquires the historical motion information of the target key points.

[0164] In step S710, the electronic device 12 predicts the first region where the target key points are located in the 3D image based on historical motion information. Then, step S711 is executed.

[0165] In step S76, if a two-dimensional image is not received while generating the three-dimensional image, the electronic device 12 executes steps S79 to S710 to determine the first region using historical motion information, thereby improving the accuracy and efficiency of determining the target key points. The specific process for determining the first region using historical motion information can be found in the relevant description in Method 1.2 above, and will not be repeated here.

[0166] In step S711, the electronic device 12 searches for target key points in the first area and obtains the target motion information of the target key points.

[0167] Electronic device 12 searches for target key points within the first area. After determining the location of the target key points, it can determine the target motion information of the target key points, and then control the joint movement of the robot. The specific method for determining the target motion information will be described in detail below, and will not be elaborated here.

[0168] In this embodiment, the electronic device 12 integrates two-dimensional images and historical motion information to determine the first region. The two complement each other, improving the accuracy and efficiency of determining target key points.

[0169] Furthermore, in this embodiment, when the first frequency equals the second frequency, the electronic device 12 also acquires a two-dimensional image when generating a three-dimensional image. In this case, the electronic device 12 can consistently determine the first region according to steps S77-S78, which maximizes the accuracy of determining the target key points.

[0170] Method 2 involves searching for target key points in a 3D image to obtain target motion information for those key points.

[0171] In this embodiment of the application, the electronic device 12 directly performs a full-image search on the three-dimensional image to obtain the target key points, and then determines the target motion information. The specific determination method will be described in detail below, and will not be elaborated here.

[0172] In this embodiment of the application, the electronic device 12 may also use other methods to determine the target key points and target motion information, and there is no limitation on this.

[0173] After acquiring the key points of the target, the electronic device 12 can determine the target motion information using any of the following methods.

[0174] Method 1: Compare the target key points of the trainee in the first 3D image and the second 3D image to obtain the target motion information of the target key points.

[0175] The first 3D image is the 3D image corresponding to the signal acquired this time, and the second 3D image is the 3D image corresponding to the signal acquired last time. After acquiring the first 3D image and searching for the target key point from the first 3D image, the electronic device 12 acquires the target key point previously searched from the second 3D image, compares the target key points searched in the two searches, and obtains target motion information such as the moving direction, moving speed, and moving distance of the target key point.

[0176] For example, electronic device 12 acquires three-dimensional image 1 and three-dimensional image 2 successively. The coordinates of key point P found in three-dimensional image 1 are (2, 3, 2), and the coordinates of key point P found in three-dimensional image 2 are (4, 3, 3). Comparing key point P in three-dimensional image 1 and three-dimensional image 2, electronic device 12 determines that the relative movement direction of key point P is (4, 3, 3) - (2, 3, 2) = (2, 0, 1), and the square of the movement distance is 2. 2 +0 2 +1 2 =5.

[0177] Method 2: Obtain the coordinates of the target key points of the trainee in the 3D image, as the target motion information of the target key points of the trainee.

[0178] In this embodiment of the application, based on a fixed three-dimensional coordinate system, after the electronic device 12 searches for the target key point from the first three-dimensional image, it determines the coordinates of the target key point and uses the coordinates as the target motion information of the target key point for the trainee.

[0179] For example, the electronic device 12 acquires three-dimensional image 1 and three-dimensional image 2 in succession. After searching for key point P in three-dimensional image 1, the coordinates of key point P in the three-dimensional coordinate system are determined to be (2, 3, 2), and (2, 3, 2) are used as the target motion information of key point P. After searching for key point P in three-dimensional image 2, the coordinates of key point P in the three-dimensional coordinate system are determined to be (4, 3, 3), and (4, 3, 3) are used as the new target motion information of key point P.

[0180] In this embodiment, the electronic device 12 can preset the association between the key points of the trainer and the joints of the robot, or it can perform calibration after acquiring the key points of the trainer and the joints of the robot to obtain the association between the key points of the trainer and the joints of the robot. After determining the target key points, the electronic device 12 can determine the joints associated with the target key points, i.e., the target joints, based on the above association between key points and joints; and then control the movement of the target joints according to the target motion information.

[0181] For example, electronic device 12 acquires 3D image 1 and 3D image 2 successively. Comparing key point P in 3D image 1 and 3D image 2, electronic device 12 obtains that the relative movement direction of key point P is (2, 0, -1), and the square of the movement distance is 5. Then, electronic device 12 can send motion information such as (2, 0, -1) and 5 to the robot. Subsequently, the robot establishes a 3D coordinate system with the current position of the joint point W associated with key point P as the origin, such as... Figure 8a As shown, control the joint point W to move to (2, 0, -1).

[0182] For example, electronic device 12 acquires three-dimensional image 1 and three-dimensional image 2 successively. In three-dimensional image 1, the coordinates of key point P are found to be (2, 3, 2). (2, 3, 2) is used as the target motion information of key point P, and motion information such as (2, 3, 2) is sent to the robot. Then, in the three-dimensional coordinate system XYZ with O as the origin, such as... Figure 8b As shown, the joint W associated with the robot's control key point P moves to (2, 3, 2).

[0183] The coordinates of key point P in the 3D image 2 are found to be (4, 3, 3). (4, 3, 3) is used as the target motion information of key point P, and motion information such as (4, 3, 3) is sent to the robot. Then, in the 3D coordinate system XYZ with O as the origin, the robot controls the joint point W associated with key point P to move to (4, 3, 3).

[0184] When electronic device 12 is a standalone physical device, it can be a server, mobile terminal, personal computer, etc. After acquiring the target motion information, electronic device 12 determines the movement information of the target joint based on the association between the trainer's key points and the robot's joints, and sends a control command to the robot, which specifies the movement information of the target joint. Upon receiving the control command, the robot moves the target joint according to the movement information included in the control command, thus enabling electronic device 12 to control the movement of the target joint. The movement information can include one or more of the following: movement direction, movement distance, movement speed, and position coordinates.

[0185] When the electronic device 12 is integrated into the robot, after acquiring the target motion information, it can directly control the movement of the target joints associated with the target key points according to the relationship between the trainer's key points and the robot's joints. Here, there is no need to transmit control commands.

[0186] In some embodiments, the electronic device 12 may also set the control ratio between the trainer and the robot, so as to complete the training of the robot performing long-distance tasks in a smaller activity area, or to complete the fine training of the robot performing short-distance tasks in a larger activity area.

[0187] The control ratio can be set according to actual needs. For example, if the control ratio is 1:2, the electronic device 12 will control the target joint to move 2 meters if it detects that the target key point has moved 1 meter. As another example, if the control ratio is 2:1, the electronic device 12 will control the target joint to move 0.5 meters if it detects that the target key point has moved 1 meter.

[0188] In some embodiments, to improve training efficiency, the electronic device 12 can preset the association between the trainee's key points and the joints of multiple robots. After determining the target motion information of the target key points, the electronic device 12 determines the target joints associated with the target key points on multiple robots according to the association between the trainee's key points and the robot's joints; and controls the movement of the target joints on each robot according to the target motion information.

[0189] For example, electronic device 12 can preset the association relationship 1 between the key points of the trainee and the joints of robot 1, and preset the association relationship 2 between the key points of the trainee and the joints of robot 2. After determining the target motion information of the target key point, electronic device 12 determines the joint 1 associated with the target key point on robot 1 according to association relationship 1, and controls the movement of joint 1 on robot 1 according to the target motion information; electronic device 12 determines the joint 2 associated with the target key point on robot 2 according to association relationship 2, and controls the movement of joint 2 on robot 2 according to the target motion information.

[0190] In this embodiment, the electronic device 12 trains multiple robots simultaneously, greatly improving the training efficiency. Here, the robots being trained simultaneously can be of the same or different models. For robots of the same model, the relationship between the trainer's key points and the robot's joints can be the same or different.

[0191] Corresponding to the robot training system described above, this application also provides a robot training method, such as... Figure 9 As shown, this method can be applied to the above-mentioned electronic device 12 and includes the following steps.

[0192] Step S91: Acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal;

[0193] Step S92: Generate a three-dimensional image of the trainee based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas among the multiple millimeter-wave antennas;

[0194] Step S93: Detect the three-dimensional image to obtain the target motion information of the target key points of the trainee;

[0195] Step S94: Based on the relationship between the trainer's key points and the robot's joints, control the movement of the target joints associated with the target key points according to the target motion information.

[0196] In the technical solution provided in this application embodiment, the key points of the trainer are associated with the joints of the robot. The trainer is located within the coverage area of ​​multiple millimeter-wave antennas. Signals emitted by the millimeter-wave antennas are used to generate a three-dimensional image of the trainer, and then the target motion information of the target key points is obtained from this three-dimensional image. This information is used to control the movement of the target joints and complete the robot training. In this application embodiment, the key points of the trainer are directly associated with the joints of the robot. As long as the trainer is within the coverage area of ​​the millimeter-wave antennas, the corresponding key points can be collected, completing the robot training. No specialized customized equipment is required, reducing the robot training cost. Furthermore, the millimeter-wave antenna signal has low latency and high accuracy, and the trainer's field of vision is not obstructed during robot training, improving the robot training accuracy.

[0197] In some embodiments, such as Figure 10 As shown, a robot training method is also provided, which may include the following steps.

[0198] Step S101: Acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal;

[0199] Step S102: Generate a three-dimensional image of the trainee based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas among the multiple millimeter-wave antennas;

[0200] Step S103: Determine the first region where the target key points are located in the 3D image;

[0201] Step S104: Search for target key points in the first region to obtain target motion information of the target key points;

[0202] Step S105: Based on the relationship between the trainer's key points and the robot's joints, control the movement of the target joints associated with the target key points according to the target motion information.

[0203] In the technical solution provided in this application embodiment, the electronic device reduces the search area of ​​the target key point from the entire three-dimensional image to a first area, thereby reducing the search area of ​​the target key point and improving both the search efficiency and the accuracy of the target key point search.

[0204] In some embodiments, step S103 may be: acquiring a two-dimensional image of the trainee; identifying a second region in the two-dimensional image where the target key point is located; and determining a first region in the three-dimensional image corresponding to the second region based on the mapping relationship between the two-dimensional image and the three-dimensional image.

[0205] In some embodiments, step S103 may be: obtaining historical motion information of target key points; and predicting the first region where the target key points are located in the three-dimensional image based on the historical motion information.

[0206] In some embodiments, the three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time and a second three-dimensional image corresponding to the signal acquired last time. In this case, step S93 above can be: comparing the target key points of the trainee in the first three-dimensional image and the second three-dimensional image to obtain the target motion information of the target key points.

[0207] In some embodiments, step S93 above may be: obtaining the coordinates of the target key points of the trainee in the three-dimensional image, as the target motion information of the target key points of the trainee.

[0208] In some embodiments, step S94 or step S105 may be: determining the target joints associated with the target key points on multiple robots based on the association between the key points of the trainer and the joints of the robot; and controlling the movement of the target joints on each robot according to the target motion information.

[0209] In some embodiments, target joints include individual points on the robot that have rotational degrees of freedom; target key points include skeletal points of the trainer and / or points obtained based on multiple associated skeletal points.

[0210] Corresponding to the robot training system described above, this application also provides a robot training device, such as... Figure 11 As shown, this method can be applied to the above-mentioned electronic device 12, including:

[0211] The acquisition module 111 is used to acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal;

[0212] The generation module 112 is used to generate a three-dimensional image of the trainee based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas among the plurality of millimeter-wave antennas;

[0213] The detection module 113 is used to detect the three-dimensional image and obtain the target motion information of the target key points of the trainee;

[0214] The control module 114 is used to control the movement of the target joints associated with the target key points according to the relationship between the trainer's key points and the robot's joints and the target motion information.

[0215] In some embodiments, the detection module 113 may be specifically used for:

[0216] Determine the first region where the target key points are located in the 3D image;

[0217] Search for target key points within the first region to obtain target motion information for the target key points.

[0218] In some embodiments, the detection module 113 may be specifically used for:

[0219] Acquire two-dimensional images of the trainees;

[0220] Identify the second region where key points of a target are located in a two-dimensional image;

[0221] Based on the mapping relationship between the two-dimensional image and the three-dimensional image, the first region corresponding to the second region in the three-dimensional image is determined.

[0222] In some embodiments, the detection module 113 may be specifically used for:

[0223] Obtain historical movement information of key target points;

[0224] Based on historical motion information, predict the first region where the target key points are located in the 3D image.

[0225] In some embodiments, the three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time, and a second three-dimensional image corresponding to the signal acquired previously;

[0226] The detection module 113 can be used to compare the target key points of the trainee in the first three-dimensional image and the second three-dimensional image to obtain the target motion information of the target key points.

[0227] In some embodiments, the detection module 113 can be specifically used to obtain the coordinates of the target key points of the trainee in the three-dimensional image, as the target motion information of the target key points of the trainee.

[0228] In some embodiments, the control module 114 may be specifically used for:

[0229] Based on the relationship between the trainer's key points and the robot's joints, the target joints associated with multiple target key points on the robot are determined.

[0230] Based on the target motion information, control the movement of the target joints on each robot.

[0231] In some embodiments, target joints include individual points on the robot that have rotational degrees of freedom; target key points include skeletal points of the trainer and / or points obtained based on multiple associated skeletal points.

[0232] In the technical solution provided in this application embodiment, the key points of the trainer are associated with the joints of the robot. The trainer is located within the coverage area of ​​multiple millimeter-wave antennas. Signals emitted by the millimeter-wave antennas are used to generate a three-dimensional image of the trainer, and then the target motion information of the target key points is obtained from this three-dimensional image. This information is used to control the movement of the target joints and complete the robot training. In this application embodiment, the key points of the trainer are directly associated with the joints of the robot. As long as the trainer is within the coverage area of ​​the millimeter-wave antennas, the corresponding key points can be collected, completing the robot training. No specialized customized equipment is required, reducing the robot training cost. Furthermore, the millimeter-wave antenna signal has low latency and high accuracy, and the trainer's field of vision is not obstructed during robot training, improving the robot training accuracy.

[0233] Corresponding to the robot training method described above, this application also provides an electronic device, such as... Figure 12 As shown, it includes a processor 121, a communication interface 122, a memory 123, and a communication bus 124, wherein the processor 121, the communication interface 122, and the memory 123 communicate with each other through the communication bus 124.

[0234] Memory 123 is used to store computer programs;

[0235] When processor 121 executes the program stored in memory 123, it performs the following steps:

[0236] Acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal;

[0237] Based on the first and second signals, a three-dimensional image of the trainee is generated, and the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas among the multiple millimeter-wave antennas.

[0238] By detecting the 3D image, the target motion information of the target key points of the trainee can be obtained;

[0239] Based on the relationship between the trainer's key points and the robot's joints, the movement of the target joints associated with the target key points is controlled according to the target motion information.

[0240] In the technical solution provided in this application embodiment, the key points of the trainer are associated with the joints of the robot. The trainer is located within the coverage area of ​​multiple millimeter-wave antennas. Signals emitted by the millimeter-wave antennas are used to generate a three-dimensional image of the trainer, and then the target motion information of the target key points is obtained from this three-dimensional image. This information is used to control the movement of the target joints and complete the robot training. In this application embodiment, the key points of the trainer are directly associated with the joints of the robot. As long as the trainer is within the coverage area of ​​the millimeter-wave antennas, the corresponding key points can be collected, completing the robot training. No specialized customized equipment is required, reducing the robot training cost. Furthermore, the millimeter-wave antenna signal has low latency and high accuracy, and the trainer's field of vision is not obstructed during robot training, improving the robot training accuracy.

[0241] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0242] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0243] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0244] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0245] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the robot training methods described above.

[0246] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the robot training methods described above.

[0247] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, electronic device, or data center to another website, computer, electronic device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as an electronic device or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0248] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0249] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of methods, apparatus, electronic devices, storage media, and program products are basically similar to the system embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the system embodiments.

[0250] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A robot training method, characterized in that, The method includes: Acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal; A three-dimensional image of the trainee is generated based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas of the plurality of millimeter-wave antennas; The target motion information of the target key points of the trainee is obtained by detecting the three-dimensional image. Based on the association between the trainer's key points and the robot's joints, and according to the target motion information, the movement of the target joints associated with the target key points is controlled.

2. The method according to claim 1, characterized in that, The step of detecting the three-dimensional image to obtain the target motion information of the target key points of the trainee includes: Determine the first region where the target key points are located in the three-dimensional image; Search for the target key points within the first region to obtain the target motion information of the target key points.

3. The method according to claim 2, characterized in that, The step of determining the first region where the target key points are located in the three-dimensional image includes: Acquire a two-dimensional image of the trainee; identify a second region in the two-dimensional image where the target key point is located; determine a first region in the three-dimensional image corresponding to the second region based on the mapping relationship between the two-dimensional and three-dimensional images; or Obtain historical motion information of the target key point; predict the first region where the target key point is located in the three-dimensional image based on the historical motion information.

4. The method according to any one of claims 1 to 3, characterized in that, The three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time, and a second three-dimensional image corresponding to the signal acquired previously; the step of detecting the three-dimensional image to obtain the target motion information of the trainee's target key points includes: comparing the trainee's target key points in the first three-dimensional image and the second three-dimensional image to obtain the target motion information of the target key points; or, obtaining the coordinates of the trainee's target key points in the three-dimensional image as the target motion information of the trainee's target key points; and / or The step of controlling the movement of the target joints associated with the target key points according to the association between the trainer's key points and the robot's joints, and in accordance with the target motion information, includes: determining the target joints associated with the target key points on multiple robots according to the association between the trainer's key points and the robot's joints; controlling the movement of the target joints on each robot according to the target motion information; and / or The target joints include points on the robot that have rotational degrees of freedom; the target key points include the skeletal points of the trainee and / or points obtained based on multiple associated skeletal points.

5. A robot training device, characterized in that, The device includes: An acquisition module is used to acquire a first signal and a second signal transmitted by multiple millimeter-wave antennas, wherein the second signal includes the echo signal of the first signal; A generation module is configured to generate a three-dimensional image of a trainee based on the first signal and the second signal, wherein the trainee is located within the coverage area of ​​at least a portion of the millimeter-wave antennas among the plurality of millimeter-wave antennas; The detection module is used to detect the three-dimensional image and obtain the target motion information of the target key points of the trainee; The control module is used to control the movement of the target joints associated with the target key points according to the relationship between the trainer's key points and the robot's joints and the target motion information.

6. The apparatus according to claim 5, characterized in that, The detection module is specifically used for: determining a first region where the target key points are located in the three-dimensional image; searching for the target key points within the first region to obtain the target motion information of the target key points; and / or The detection module is specifically used for: acquiring a two-dimensional image of the trainee; identifying a second region in the two-dimensional image where the target key point is located; determining a first region in the three-dimensional image corresponding to the second region based on the mapping relationship between the two-dimensional image and the three-dimensional image; or, acquiring historical motion information of the target key point; and predicting the first region in the three-dimensional image where the target key point is located based on the historical motion information. and / or The three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time, and a second three-dimensional image corresponding to the signal acquired last time; the detection module is specifically used to compare the target key points of the trainee in the first three-dimensional image and the second three-dimensional image to obtain the target motion information of the target key points; or, to obtain the coordinates of the target key points of the trainee in the three-dimensional image as the target motion information of the target key points of the trainee. and / or The control module is specifically used to: determine the target joints associated with the target key points on multiple robots based on the relationship between the key points of the trainer and the joints of the robot; and control the movement of the target joints on each robot according to the target motion information. and / or The target joints include points on the robot that have rotational degrees of freedom; the target key points include the skeletal points of the trainee and / or points obtained based on multiple associated skeletal points.

7. A robot training system, characterized in that, Includes multiple millimeter-wave antennas and electronic devices; The plurality of millimeter-wave antennas are used to transmit a first signal and receive a second signal, the second signal including the echo signal of the first signal; and to transmit the first signal and the second signal to the electronic device; The electronic device is configured to generate a three-dimensional image of a trainee based on the first signal and the second signal, the trainee being located within the coverage area of ​​at least a portion of the millimeter-wave antennas of the plurality of millimeter-wave antennas; The target motion information of the target key points of the trainee is obtained by detecting the three-dimensional image. Based on the relationship between the trainer's key points and the robot's joints, the movement of the target joints associated with the target key points is controlled according to the target motion information.

8. The system according to claim 7, characterized in that, The plurality of millimeter-wave antennas are located above the activity area of ​​the trainee, and the projections of the plurality of millimeter-wave antennas onto the plane of the activity area are located at the boundary of the activity area; and / or The electronic device is specifically configured to: determine a first region where a target key point is located in the three-dimensional image; search for the target key point within the first region to obtain target motion information of the target key point; and / or The system further includes a camera; the camera is used to acquire two-dimensional images of the trainee; the electronic device is specifically used to: acquire the two-dimensional image of the trainee acquired by the camera; identify a second region in the two-dimensional image where the target key point is located; determine a first region in the three-dimensional image corresponding to the second region based on the mapping relationship between the two-dimensional image and the three-dimensional image; or, acquire historical motion information of the target key point; predict the first region in the three-dimensional image where the target key point is located based on the historical motion information. and / or The camera is located above the activity area of ​​the trainee, and the projection of the camera onto the plane of the activity area is located at the boundary of the activity area; and / or The three-dimensional image includes a first three-dimensional image corresponding to the signal acquired this time, and a second three-dimensional image corresponding to the signal acquired last time; the electronic device is specifically used to: compare the target key points of the trainee in the first three-dimensional image and the second three-dimensional image to obtain the target motion information of the target key points; or, acquire the coordinates of the target key points of the trainee in the three-dimensional image as the target motion information of the target key points of the trainee. and / or The target joints include points on the robot that have rotational degrees of freedom; the target key points include the skeletal points of the trainee and / or points obtained based on multiple associated skeletal points.

9. An electronic device, characterized in that, The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements the method described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 4.

Images