A teaching method and system for a vision-based humanoid robot

Abstract

The embodiment of the present application relates to the technical field of robots, and discloses a teaching method and system for a visual humanoid robot, which comprises the following steps: obtaining human body teaching images of a teaching person in a current scene through a camera assembly; identifying and analyzing the human body teaching images of continuous frames to obtain human body key point coordinates in each movement stage, and calculating joint movement states of each control joint of a corresponding robot; generating simulation control instructions for each joint of the humanoid robot according to the joint movement states of each control joint, and sending the simulation control instructions to the corresponding humanoid robot for action simulation. The teaching method for the visual humanoid robot in the embodiment of the present application obtains human body teaching images of a teaching person through a camera assembly, and the teaching person does not need to wear complex sensor equipment, thereby reducing the teaching cost and operation difficulty.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a teaching method and system for a vision-based humanoid robot. BACKGROUND

[0002] Currently, the teaching scheme of AI-based humanoid robots mainly relies on manual programming, trajectory point entry interpolation, and motion capture teaching. However, these traditional methods have many shortcomings: the operation threshold is high, manual programming requires professional personnel to write complex teaching programs and plan paths, which requires extremely high programming skills; trajectory point entry interpolation also requires professional personnel to accurately operate to determine key trajectory points and perform interpolation calculation; and motion capture teaching relies on professional equipment and operation, which is difficult for ordinary personnel to master. Therefore, designing an efficient robot teaching scheme has become a technical problem to be solved by those skilled in the art. SUMMARY

[0003] To address the defects, the embodiment of the present application discloses a teaching method for a vision-based humanoid robot, which can efficiently complete the teaching of the robot.

[0004] The first aspect of the embodiment of the present application discloses a teaching method for a vision-based humanoid robot, comprising:

[0005] obtaining a human teaching image of a teaching personnel in a current scene through a camera assembly;

[0006] identifying and analyzing the human teaching images of consecutive frames to obtain human key point coordinates in each movement stage, determining robot joints corresponding to each human key point, and calculating joint motion states of each control joint of the corresponding robot;

[0007] generating simulation control instructions for each joint of the humanoid robot according to the joint motion states of each control joint, sending the simulation control instructions to the corresponding humanoid robot for action simulation, and saving the simulation control instructions when the action simulation result meets the requirements.

[0008] The embodiment of the present application obtains the human teaching image of the teaching personnel through the camera assembly, without the need for the teaching personnel to wear complex sensor equipment, thereby reducing the teaching cost and operation difficulty.

[0009] As an optional implementation, in the first aspect of the embodiment of the present application, the identification and analysis of the human teaching images of consecutive frames to obtain human key point coordinates in each movement stage, the determination of robot joints corresponding to each human key point, and the calculation of joint motion states of each control joint of the corresponding robot comprise:

[0010] The human demonstration image of continuous frames is recognized and analyzed to obtain human key point coordinates in each movement stage, the robot joints corresponding to each human key point are determined, and the human key points are input into a motion reorientation formula to calculate the joint motion state of each control joint of the corresponding robot. ;

[0011] wherein, represents human skeleton key point coordinates; represents kinematic constraints, including joint angle limit , self-collision avoidance constraint ; represents an optimization function for minimizing the position error between the human body and the robot end.

[0012] The embodiment of the application establishes a direct correlation between human skeleton key point coordinates and robot joint motion state through the motion reorientation formula, and realizes accurate mapping from human action to robot action.

[0013] As an optional implementation, in the first aspect of the embodiment of the application, the human demonstration image includes an upper limb demonstration image and a lower limb demonstration image;

[0014] The human demonstration image of continuous frames is recognized and analyzed to obtain human key point coordinates in each movement stage, the robot joints corresponding to each human key point are determined, and the joint motion state of each control joint of the corresponding robot is calculated, including:

[0015] The upper limb demonstration image of continuous frames is recognized to determine the upper limb key point coordinates in each movement stage, and the corresponding human joint angle is determined according to the upper limb key point coordinates in each movement stage;

[0016] The correspondence between the upper limb key point and the robot upper limb joint is determined, and the human joint angle is input into an upper limb mapping formula to obtain the robot joint angle of each upper limb joint of the corresponding robot; the upper limb mapping formula is:

[0017] ; wherein, respectively represent the shoulder joint angle, elbow joint angle and wrist joint angle of the robot;

[0018] respectively represent the corresponding shoulder joint angle, elbow joint angle and wrist joint angle of the human body; K is a 3*3 diagonal scaling matrix, used to match the joint motion range difference between the human body and the robot; is a 3*3 rotation order conversion matrix, used to convert the coordinates of the human body and the robot; respectively are the offset amounts of the corresponding joints of the robot and the human body;

[0019] identify the lower limb demonstration images of continuous frames to determine the lower limb key point coordinates in each movement stage, and determine the correspondence between the lower limb key points and the robot lower limb joints according to a human-machine mapping relationship;

[0020] convert the lower limb key point coordinates into robot lower limb joint coordinates in a robot coordinate system, and input the robot lower limb joint coordinates into a lower limb motion mapping formula for inverse kinematics calculation to obtain robot joint angles of each robot lower limb joint, the lower limb motion mapping formula being:

[0021]

[0022] wherein, is a robot lower limb joint angle vector; represents 3D coordinate positions of human ankle joints and hip joints; represents an inverse kinematics solver.

[0023] The embodiment of the present application adopts different mapping logics according to the motion characteristics and structural differences of the upper limbs and the lower limbs, so that the calculation of the robot upper and lower limb joint angles is more in line with the respective motion laws, and the accuracy of the robot in replicating human actions is improved.

[0024] As an optional implementation, in the first aspect of the embodiment of the present application, the human demonstration image further includes a hand demonstration action; and the demonstration method further includes:

[0025] A high-precision camera is used to acquire the hand action, to determine the mapping relationship between the finger joint nodes and the robot finger joints, and to generate finger joint control instructions of the humanoid robot.

[0026] The embodiment of the present application establishes the accurate mapping relationship between the finger joint nodes and the robot finger joints, so that the robot can accurately imitate the hand actions of the demonstration personnel, including various complex gripping, pinching, gestures and other fine operations, greatly improving the flexibility and accuracy of the robot hand action.

[0027] As an optional implementation, in the first aspect of the embodiment of the present application, the human demonstration image of continuous frames is identified and analyzed to obtain human key point coordinates in each movement stage, and each human key point corresponding robot joint is determined, including:

[0028] The human demonstration image of continuous frames is preprocessed to obtain a preprocessed image, and the preprocessing includes image size adjustment, normalization processing and color space conversion;

[0029] The human body posture estimation model is used to analyze the preprocessed image to obtain human body key point coordinates and corresponding confidence in each movement stage, and low-confidence key point coordinates are filtered out.

[0030] According to the human body posture estimation model, the human body key points are determined, the corresponding human joint information is determined according to the set key point mapping rule, and the robot joints corresponding to each human body key point are determined.

[0031] According to the human body posture estimation model, the human body key points are determined, the corresponding human joint information is determined according to the set key point mapping rule, and the robot joints corresponding to each human body key point are determined.

[0032] As an optional implementation, in the first aspect of the embodiment of the present application, after the simulation control instruction is sent to the corresponding humanoid robot for motion simulation, the method further comprises:

[0033] According to the simulation control instruction, a corresponding robot joint sequence is generated, and the robot joint sequence comprises a robot execution timestamp, a joint name list, a trajectory point array and a timestamp mapping table.

[0034] The human motion video and the motion execution screen of the humanoid robot are played synchronously, wherein each frame of image in the human motion video is aligned with the motion of the humanoid robot through the timestamp mapping table.

[0035] When motion adjustment is needed, the key frame is dragged left and right along the time axis according to user selection or the joint angle data is modified in the joint angle panel; the position of the key frame on the time axis is updated in real time, and the robot posture change is previewed in the simulation screen.

[0036] The embodiment of the present application enhances the interactivity between the demonstrator and the system through the visual time axis, the joint angle panel and the synchronous playback function. The demonstrator can actively participate in the motion optimization process and flexibly control the details of the robot motion according to actual needs.

[0037] As an optional implementation, in the first aspect of the embodiment of the present application, the human body demonstration image of the continuous frames is identified and analyzed to obtain human body key point coordinates in each movement stage, the robot joints corresponding to each human body key point are determined, and the joint motion state of each control joint of the corresponding robot is calculated, comprising:

[0038] The human body demonstration image of the continuous frames is input into a self-supervised pre-training model based on Transformer for processing to obtain an action feature sequence and an action semantic vector.

[0039] The action feature sequence and the action semantic vector are subjected to a contrast learning process, so that features of similar actions are close in a hidden space, and features of different actions are far away in the hidden space.

[0040] According to the action feature sequence and the action semantic vector subjected to the contrast learning process, a standardized motion data is reconstructed by a model based on a Transformer, and joint motion states of respective control joints of the corresponding robot are generated based on the standardized motion data.

[0041] The scheme of the embodiment of the present application realizes the mapping of human body actions to robot joint motions by combining self-supervised learning and contrast learning of a Transformer architecture, and can indeed significantly reduce the computational burden of inverse kinematics solving, and can improve the understanding of complex actions.

[0042] The second aspect of the embodiment of the present application discloses a visual-based humanoid robot teaching system, comprising:

[0043] An image acquisition module is configured to acquire human body teaching images of a teaching person in a current scene by a camera assembly;

[0044] An analysis module is configured to identify and analyze the continuous frame human body teaching images to obtain human body key point coordinates in each movement stage, determine robot joints corresponding to each human body key point, and calculate joint motion states of respective control joints of the corresponding robot;

[0045] An action execution module is configured to generate simulation control instructions of respective joints of the humanoid robot according to the joint motion states of the respective control joints, send the simulation control instructions to the corresponding humanoid robot for action simulation, and save the simulation control instructions when the action simulation result meets the requirements.

[0046] The third aspect of the embodiment of the present application discloses an electronic device, comprising a memory storing executable program codes, and a processor coupled with the memory; the processor invokes the executable program codes stored in the memory, and is configured to execute the visual-based humanoid robot teaching method disclosed in the first aspect of the embodiment of the present application.

[0047] The fourth aspect of the embodiment of the present application discloses a computer readable storage medium storing a computer program, wherein the computer program causes a computer to execute the visual-based humanoid robot teaching method disclosed in the first aspect of the embodiment of the present application.

[0048] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0049] The teaching method of the visual-based humanoid robot in the embodiment of the present application acquires the human teaching image of the teaching personnel through the camera assembly, without the need for the teaching personnel to wear complicated sensor equipment, thereby reducing the teaching cost and operation difficulty. The teaching personnel can naturally make various actions, and the robot can automatically capture learning, so that the teaching process is more efficient and intuitive, and the teaching efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0051] Figure 1 is a flowchart of the teaching method of the visual-based humanoid robot disclosed by the embodiment of the present application;

[0052] Figure 2 is a flowchart of the joint motion state calculation disclosed by the embodiment of the present application;

[0053] Figure 3 is a flowchart of the action simulation disclosed by the embodiment of the present application;

[0054] Figure 4 is another flowchart of the joint motion state calculation disclosed by the embodiment of the present application;

[0055] Figure 5 is a structural diagram of the teaching system of the visual-based humanoid robot provided by the embodiment of the present application;

[0056] Figure 6 is a structural diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0058] It should be noted that the terms "first", "second", "third", "fourth" and the like in the description and claims of the application are used for distinguishing between similar elements and not necessarily for describing a specific sequential order. The terms "comprising", "having", "including" and "containing" and any variations thereof used herein are intended to cover a non-exclusive inclusion such that a process, method, system, product or apparatus that comprises, has, includes or contains a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, system, product or apparatus.

[0059] Embodiment one

[0060] Please refer to Figure 1 , Figure 1 is a flowchart of the teaching method of the visual-based humanoid robot according to the embodiments of the present application. The execution subject of the method described in the embodiments of the present application is composed of software or / and hardware, which can receive relevant information through wired or / and wireless mode and can send certain instructions. Of course, it can also have certain processing function and storage function. The execution subject can control multiple devices, such as remote physical servers or cloud servers and related software, or can be a local host or server and related software that performs relevant operations on devices placed somewhere. In some scenarios, it can also control multiple storage devices, which can be placed in the same place or different places as the devices. As shown in Figure 1 The teaching method of the visual-based humanoid robot includes the following steps:

[0061] S101: obtaining the human teaching image of the teaching personnel in the current scene through the camera assembly;

[0062] S102: identifying and analyzing the human teaching image of the continuous frames to obtain the human key point coordinates in each moving stage, determining the robot joints corresponding to each human key point, and calculating the joint motion state of each control joint of the corresponding robot;

[0063] S103: generating simulation control instructions for each joint of the humanoid robot according to the joint motion state of each control joint, sending the simulation control instructions to the corresponding humanoid robot for action simulation, and saving the simulation control instructions when the action simulation result meets the requirements.

[0064] The embodiment of the present application acquires the human demonstration image of the demonstrator through the camera assembly, without the need for the demonstrator to wear complex sensor equipment, thereby reducing the demonstration cost and operation difficulty. The demonstrator can naturally make various actions, and the robot can automatically capture learning, so that the demonstration process is more efficient and intuitive, the demonstration efficiency is improved, and non-professionals can conveniently operate and guide the robot.

[0065] The embodiment of the present application identifies and analyzes the continuous frame human demonstration image, can determine the human key point coordinates and correspond to the robot joints, and further calculate the joint motion state. This enables the robot to accurately capture the action details of the demonstrator, realizes high-precision action imitation, and enables the robot to complete complex human-like actions, thereby improving the human-likeness and accuracy of the robot action.

[0066] The embodiment of the present application generates simulation control instructions according to the joint motion state and performs action simulation, and only saves the instructions when the result meets the requirements. This helps to ensure that the action performed by the robot meets the expectations, discovers problems in advance through simulation verification and adjustment, avoids faults or dangers that may be caused by directly executing incorrect instructions on the actual robot, and improves the reliability of the instructions and the stability of the robot operation.

[0067] The method is based on visual recognition, and as long as the camera can clearly acquire the image, it can be applied in various environments in theory, and has good adaptability to different scenes. At the same time, by adjusting the correspondence between the human key points and the robot joints and the algorithm parameters, the method is suitable for humanoid robots of different models and structures, and has a certain universality. The simulation control instructions that meet the requirements are saved, which is convenient for subsequent calling, and the robot can repeatedly execute the corresponding actions according to these instructions at any time, which is beneficial to the robot to quickly master various skills, realize skill accumulation and reuse, and facilitate the robot to quickly switch in different task scenes, thereby improving the working efficiency and practicality of the robot. The scheme of the embodiment of the present application realizes the whole process of image acquisition, key point extraction, motion mapping and robot control, converts the intuitive action of the user into a control sequence executable by the robot, and can complete the demonstration without professional programming knowledge.

[0068] When a specific implementation is performed, the camera also needs to be calibrated to obtain corresponding internal and external parameter matrices to assist in realizing accurate calibration of specific pixel points.

[0069] More preferably, the continuous frame human demonstration image is identified and analyzed to obtain the human key point coordinates in each movement stage, the robot joints corresponding to each human key point are determined, and the joint motion state of each control joint of the corresponding robot is calculated, including:

[0070] The human body demonstration image of continuous frames is recognized and analyzed to obtain human body key point coordinates in each movement stage, the robot joints corresponding to each human body key point are determined, and the human body key points are input into a motion reorientation formula to calculate the joint motion state of each control joint of the corresponding robot. ;

[0071] wherein, represents the human skeleton key point coordinates; represents the kinematic constraint combination, including joint angle limit , self-collision avoidance constraint ; represents an optimization function for minimizing the position error between the human body and the robot end. is the angle of the i th joint of the robot, , are the minimum and maximum allowable rotation angles of the joint, respectively; , are two different links of the robot, dist( ) represents the shortest distance between the links, is a safety distance threshold, and the above constraints ensure that the links do not collide during movement.

[0072] The motion reorientation formula of the embodiment of the present application establishes a direct correlation between the human skeleton key point coordinates and the robot joint motion state, realizing accurate mapping from human action to robot action. This enables the robot to more accurately reproduce the actions of the demonstration personnel, reducing the deviation in the action transmission process. The joint angle limit in the kinematic constraint combination ensures that the robot joint motion is within a safe range, preventing damage from excessive joint motion; self-collision avoidance prevents collisions between the various components of the robot, ensuring the safety and stability of the robot motion and prolonging the service life of the robot.

[0073] The optimization function of the embodiment of the present application is used to minimize the position error between the human body and the robot end, so that the position of the robot end effector is as consistent as possible with the position of the corresponding part of the human body, improving the consistency of the robot action and the demonstration action, making the robot action more in line with the intention of the demonstration personnel. The robot joint motion state is calculated through explicit mathematical formulas and constraint conditions, making the entire demonstration method more standardized and scientific. At the same time, it also provides a clear basis for subsequent algorithm optimization and parameter adjustment, enhancing the operability and scalability of the method.

[0074] More preferably, as shown in Figure 2 , the human body demonstration image includes an upper limb demonstration image and a lower limb demonstration image;

[0075] The human demonstration image of the continuous frames is recognized and analyzed to obtain human key point coordinates in each movement stage, the robot joints corresponding to each human key point are determined, and the joint motion state of each control joint of the corresponding robot is calculated, including:

[0076] S1021: The upper limb demonstration image of the continuous frames is recognized to determine the upper limb key point coordinates in each movement stage, and the corresponding human joint angle is determined according to the upper limb key point coordinates in each movement stage;

[0077] S1022: The corresponding relationship between the upper limb key point and the robot upper limb joint is determined, and the human joint angle is input into the upper limb mapping formula to obtain the robot joint angle of each upper limb joint of the corresponding robot; the upper limb mapping formula is:

[0078] ; wherein, respectively represent the shoulder joint angle, the elbow joint angle and the wrist joint angle of the robot;

[0079] respectively represent the corresponding shoulder joint angle, elbow joint angle and wrist joint angle of the human body; K is a 3*3 diagonal scaling matrix, used to match the joint motion range difference between the human body and the robot; is a 3*3 rotation sequence conversion matrix, used to convert the coordinates of the human body and the robot; respectively are the offsets of the corresponding joints of the robot and the human body;

[0080] S1023: The lower limb demonstration image of the continuous frames is recognized to determine the lower limb key point coordinates in each movement stage, and the corresponding relationship between the lower limb key point and the robot lower limb joint is determined according to the human-robot mapping relationship;

[0081] S1024: The lower limb key point coordinates are converted into robot coordinate system under the machine lower limb joint coordinates, and the machine lower limb joint coordinates are input into the lower limb motion mapping formula for inverse kinematics calculation to obtain the robot joint angle of each robot lower limb joint, and the lower limb motion mapping formula is:

[0082]

[0083] wherein, is a robot lower limb joint angle vector; represents the 3D coordinate position of the human ankle joint and the hip joint; represents an inverse kinematics solver.

[0084] According to the motion characteristics and structural differences of the upper and lower limbs, different mapping logics are adopted, so that the calculation of the joint angles of the upper and lower limbs of the robot is more in line with the respective motion laws. The upper limbs directly establish the correlation of the shoulder, elbow and wrist joints of the human body and the robot through a special angle mapping formula, which can accurately transfer the fine movements of the upper limbs; the lower limbs are more suitable for processing lower limb movements involving overall displacement and support through inverse kinematics solving, which overall improves the accuracy of the robot in replicating human movements.

[0085] In the upper limb mapping formula, the angle scaling matrix K can effectively match the different joint motion ranges of the human body and the robot, and the rotation sequence conversion matrix The difference between the two coordinate systems is solved, and the offset β can compensate for the inherent deviation caused by the structural size. These parameter settings make the robot better adapt to the physical structural differences between the human body and itself, and avoid motion distortion caused by structural differences.

[0086] The lower limbs use a mapping formula based on inverse kinematics to solve based on the 3D coordinates of the ankle and hip joints of the human body, which can ensure that the combination of joint angles of the robot lower limbs in simulating human movements conforms to the kinematic laws, especially in complex lower limb movements such as walking and turning, which can ensure the stability of the robot's center of gravity and motion balance, and reduce the risk of tipping over.

[0087] By separately processing the upper and lower limbs, independent adjustment and optimization can be performed according to the teaching needs of different limbs. For example, the mapping accuracy can be enhanced for fine operations of the upper limbs (such as grasping and gestures), and the stability control can be enhanced for lower limb movement, so that the method can adapt to different types of teaching scenarios and motion requirements, and the flexibility of the overall scheme is enhanced.

[0088] The embodiment of the application separates the overall human movement into upper and lower limb processing, reducing the complexity of single action analysis. By processing each limb movement with targeted algorithms and formulas, key information can be more efficiently extracted from the teaching image and converted into joint angles executable by the robot, improving the efficiency of action analysis and conversion, and making real-time teaching possible.

[0089] More preferably, the human teaching image further includes a hand teaching action; the teaching method further includes:

[0090] A high-precision camera is used to acquire hand movements, determine the mapping relationship between the finger joint nodes and the finger joints of the robot, and generate finger joint control instructions for the humanoid robot.

[0091] By adopting a high-precision camera, subtle movements of the hand and slight changes of the finger joints can be captured, thereby providing high-quality raw data for subsequent movement analysis. By establishing a precise mapping relationship between the finger joints and the robot finger joints, the robot can accurately imitate the hand movements of the demonstrator, including various complex gripping, pinching, gestures and other delicate operations, thereby greatly improving the flexibility and accuracy of the hand movements of the robot. On the basis of the original upper limb and lower limb demonstration, the processing of hand demonstration movements is added, so that the entire demonstration process is more complete and can cover most of the important movements of the human body. By learning complete human movements, including subtle movements of the hand, the robot can more comprehensively understand and execute the intentions of the demonstrator, thereby improving the coordination and continuity of the overall movements.

[0092] More preferably, the human demonstration image of the continuous frame is identified and analyzed to obtain the human key point coordinates in each movement stage, and the robot joints corresponding to each human key point are determined, including:

[0093] The human demonstration image of the continuous frame is preprocessed to obtain a preprocessed image, and the preprocessing includes image size adjustment, normalization processing and color space conversion;

[0094] The preprocessed image is analyzed by a human pose estimation model to obtain the human key point coordinates in each movement stage and the corresponding confidence, and the low-confidence key point coordinates are filtered out;

[0095] According to the human pose estimation model, the human key points are determined according to the set key point mapping rule to determine the corresponding human joint information, and the robot joints corresponding to each human key point are determined.

[0096] The image data format can be unified by the preprocessing operations such as image size adjustment, normalization processing and color space conversion, the interference of factors such as light and scale on subsequent recognition is reduced, high-quality input data is provided for the human pose estimation model, thereby improving the accuracy and efficiency of model analysis, and laying a good foundation for subsequent key point coordinate extraction. The human pose estimation model outputs the key point coordinates with a confidence, and by filtering the low-confidence key point coordinates, the inaccurate or blurred key point information can be eliminated, the remaining key point coordinates have high reliability, and the adverse effects of the error data on subsequent joint mapping and motion state calculation are avoided.

[0097] The human key points determined based on the human posture estimation model are determined according to the set key point mapping rule to correspond to the human joint information, and are further associated to the robot joints to establish a clear and standard mapping relationship, so that the human action information can be accurately transmitted to the corresponding joints of the robot, and reliable basis is provided for subsequent joint motion state calculation and control instruction generation. The pre-processing step reduces the influence of environmental interference, the confidence filtering mechanism reduces the interference of error data, and the standardized mapping rule guarantees the stability of the joint correspondence relationship, which collectively improves the robustness of the entire recognition and analysis process, so that the method can maintain good recognition and analysis effect under different scenes and conditions.

[0098] The human key point coordinates and corresponding robot joint information obtained through the above processing have the characteristics of high accuracy and strong reliability, and the joint motion state calculated and the control instruction generated based thereon are more accurate, thereby guaranteeing the effectiveness of the subsequent action simulation result and the accuracy of the actual action of the robot.

[0099] The human posture estimation model in the embodiment of the application adopts a MediaPipePose or OpenPose model, and real-time detects 33 key skeleton points (such as head, shoulder, elbow, wrist, hip, knee, ankle, etc.) of a human body, and outputs 2D pixel coordinates (x, y) and confidence (0-1) of each point. The advantages of MediaPipe are: lightweight design, support for edge-side real-time inference, and suitability for dynamic motion capture.

[0100] The specific inference process is as follows: input the pre-processed image, extract the feature map through CNN; the graph convolution network (GCN) predicts the key point position, and optimizes the coordinates in combination with the human body topology (such as the connection relationship of shoulder-elbow-wrist); filter low-precision points with a confidence less than 0.6, and output a high-reliability key point sequence.

[0101] After the key point sequence is determined, 2D to 3D coordinate conversion can be performed: if a monocular camera is used: the human body proportion prior (such as shoulder width ≈ 0.5m) is used in combination with a triangulation algorithm to estimate the 3D coordinates of the key points; if a depth camera is used: the 3D coordinates are directly converted through depth information, and the accuracy is higher.

[0102] More preferably, as shown in Figure 3 After the simulation control instruction is sent to the corresponding humanoid robot for action simulation, the method further comprises:

[0103] S1031: generating a corresponding robot joint sequence according to the simulation control instruction, the robot joint sequence comprising a robot execution timestamp, a joint name list, a trajectory point array, and a timestamp mapping table;

[0104] S1032: synchronize playing the human action video and the action execution picture of the humanoid robot, wherein each frame of image in the human action video is mapped with the action of the humanoid robot through a time stamp mapping table;

[0105] S1033: when action adjustment is needed, the key frame is dragged left or right along the time axis according to the user's selection or the joint angle data is modified in the joint angle panel; the position of the key frame on the time axis is updated in real time, and the change of the robot posture is previewed in the simulation picture.

[0106] The embodiment of the application provides a structured and traceable data basis for action simulation by generating a robot joint sequence containing an execution time stamp, a joint name list, a trajectory point array and a time stamp mapping table. The motion trajectory and time node of each joint are clear and explicit, facilitating accurate analysis of the details of the robot action and ensuring the transparency and verifiability of the action simulation process.

[0107] Synchronizing playing the human action video and the action execution picture of the robot through the time stamp mapping table enables the demonstrator to intuitively compare the action differences between the two, quickly find the deviations between the robot action and the human demonstration action, and provide clear reference for subsequent action adjustment, thereby improving the efficiency and accuracy of action verification.

[0108] Supporting the user to adjust the action through two ways of dragging the key frame or modifying the joint angle data is intuitive and flexible. Whether it is the adjustment of the action rhythm on the overall time axis or the fine correction of the angle of a single joint, it can be quickly realized, meeting the adjustment needs in different scenarios and reducing the threshold of action optimization.

[0109] The modification of the key frame can be reflected in the time axis position and the simulation picture in real time, so that the demonstrator can immediately preview the change of the robot posture and judge whether the adjustment effect meets the expectation in time. This real-time feedback mechanism reduces the cycle of adjustment-verification, improves the efficiency of action optimization, and ensures that the final action instruction is more accurate. Through the visual time axis, joint angle panel and synchronous playing function, the interactivity between the demonstrator and the system is enhanced. The demonstrator can actively participate in the action optimization process, flexibly control the details of the robot action according to the actual needs, so that the demonstration result is more suitable for the actual application scenario, and the controllability and final effect of the entire demonstration process are improved.

[0110] More preferably, as shown in Figure 4 the human demonstration image of the continuous frame is recognized and analyzed to obtain the human key point coordinates in each moving stage, the robot joints corresponding to each human key point are determined, and the joint motion state of each control joint of the corresponding robot is calculated, including:

[0111] S102a: input the continuous frame human demonstration image into the Transformer-based self-supervised pre-training model for processing to obtain an action feature sequence and an action semantic vector;

[0112] S102b: perform contrastive learning processing on the action feature sequence and the action semantic vector to make the features of similar actions close in the hidden space and the features of different actions far apart in the hidden space;

[0113] S102c: reconstruct standardized motion data from the Transformer-based model according to the action feature sequence and the action semantic vector after the contrastive learning processing, and generate joint motion states of respective control joints of the corresponding robot based on the standardized motion data.

[0114] The Transformer-based self-supervised pre-training model of the embodiment has strong sequence modeling capability, can effectively capture the time sequence features and spatial correlations of actions from the continuous frame human demonstration image, and extract more representative action feature sequences and semantic vectors. The self-supervised learning method does not require a large amount of manually labeled data, can be trained using a large amount of unlabeled images, and enhances the generalization capability of the model to different scenes and different individual actions.

[0115] The contrastive learning makes similar action features close in the hidden space and different action features far apart, which can enhance the discriminability of the action features and make the model more accurate in distinguishing subtle action differences. At the same time, the construction of the action semantic vector helps the model to understand the high-level semantic information of the action, such as the intention and purpose of the action, rather than just the surface posture change, which improves the understanding ability of the model to complex actions. The Transformer model reconstructs standardized motion data from the processed action features, which can unify the action data format and scale under different demonstration personnel and different shooting conditions, and reduce the noise caused by individual differences and environmental interference. The standardized motion data provides consistent input for the subsequent calculation of the robot joint motion state, which helps to improve the stability and consistency of the robot action in different scenes.

[0116] The joint motion state is generated by combining the timing characteristics, semantic information and standardized motion data, so that the joint motion of the robot not only matches the human body action in the posture, but also is more in line with the human motion law in the dynamic characteristics such as motion rhythm and force change. For complex continuous actions, this method can better maintain the continuity and naturalness of the action and reduce the sense of lag or strangeness in the action conversion process. The model architecture based on the Transformer and the self-supervised learning mechanism have good scalability, and the processing capacity for new actions and complex actions can be continuously improved through continuous learning. At the same time, the contrast learning framework enables the model to flexibly adapt to different types of humanoid robot joint structures, and only needs to adjust the mapping relationship of the motion data to the joint state, so that it can be applied to different models, improving the universality of the method.

[0117] In addition to the above effects, the scheme of the embodiment of the application realizes the mapping of human body actions to robot joint motion by combining the Transformer architecture of self-supervised learning and contrast learning, which can indeed significantly reduce the computational burden of inverse kinematics solving. The decoder part adopts a Transformer or diffusion model to reconstruct the standardized motion data (such as joint angle sequence) from the feature sequence after contrast learning. The standardization process realizes physical feasibility through predefined robot motion constraints (such as joint limits and speed limits).

[0118] The traditional IK needs to solve nonlinear equations in real time, while the scheme of the embodiment of the application directly outputs the joint state through an end-to-end model, which has the advantages that the model inference complexity is lower than the iterative calculation of numerical IK. The model learns the reasonable action distribution from the data, which can avoid the singularity problem in IK; through the semantic vector constraint, the model can automatically select the joint solution that best meets the teaching intention.

[0119] The vision-based teaching method of the humanoid robot in the embodiment of the application acquires the human body teaching image of the teaching personnel through the camera assembly, without the need for the teaching personnel to wear complex sensor equipment, thereby reducing the teaching cost and operation difficulty. The teaching personnel can naturally make various actions, and the robot can automatically capture and learn, so that the teaching process is more efficient and intuitive, and the teaching efficiency is improved.

[0120] Embodiment two

[0121] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of the vision-based teaching system of the humanoid robot disclosed by the embodiment of the application. As Figure 5 indicated, the vision-based teaching system of the humanoid robot can include:

[0122] The image acquisition module 21 is configured to acquire the human body teaching image of the teaching personnel in the current scene through the camera assembly.

[0123] The analysis module 22 is used for recognizing and analyzing the human demonstration image of the continuous frames to obtain the human key point coordinates in each movement stage, determining the robot joints corresponding to each human key point, and calculating the joint motion state of each control joint of the corresponding robot;

[0124] The action execution module 23 is used for generating the simulation control instruction of each joint of the humanoid robot according to the joint motion state of each control joint, sending the simulation control instruction to the corresponding humanoid robot for action simulation, and saving the simulation control instruction when the action simulation result meets the requirement.

[0125] The teaching method of the humanoid robot based on vision in the embodiment of the application obtains the human demonstration image of the demonstrator through the camera assembly, without the need for the demonstrator to wear complex sensor equipment, thereby reducing the teaching cost and operation difficulty. The demonstrator can naturally make various actions, and the robot can automatically capture and learn, so that the teaching process is more efficient and intuitive, and the teaching efficiency is improved.

[0126] Embodiment three

[0127] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of an electronic device disclosed in the embodiment of the application. The electronic device can be a computer, a server, and the like. Of course, in some cases, it can also be a smart device such as a mobile phone, a tablet computer, and a monitoring terminal, and an image acquisition device with a processing function. As shown in Figure 6 , the electronic device can include:

[0128] a memory 510 storing executable program codes;

[0129] a processor 520 coupled with the memory 510;

[0130] The processor 520 calls the executable program codes stored in the memory 510 to execute part or all of the steps in the teaching method of the humanoid robot based on vision in the embodiment one.

[0131] The embodiment of the application discloses a computer readable storage medium storing a computer program, wherein the computer program causes a computer to execute part or all of the steps in the teaching method of the humanoid robot based on vision in the embodiment one.

[0132] The embodiment of the application further discloses a computer program product, wherein when the computer program product runs on a computer, it causes the computer to execute part or all of the steps in the teaching method of the humanoid robot based on vision in the embodiment one.

[0133] The application further discloses an application publishing platform, which is used for publishing the computer program product.

[0134] In various embodiments of the present application, it should be understood that the size of the serial number of the processes does not mean the inevitable sequence of execution, and the execution sequence of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0135] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0136] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0137] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-accessible memory. Based on such understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of steps for causing a computer device (which can be a personal computer, a server or a network device, and specifically can be a processor in the computer device) to execute some or all of the steps of the method described in each embodiment of the present application.

[0138] In the embodiments provided by the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that the determination of B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0139] Those skilled in the art can understand that part or all of the steps in the various methods of the embodiments can be completed by instructing the relevant hardware by a program, and the program can be stored in a computer readable storage medium, including a Read-Only Memory (ROM), a Random Access Memory (RAM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an Electrically-Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other medium that can be used to carry or store data in a computer readable manner.

[0140] The teaching method, system, electronic device and storage medium of the visual-based humanoid robot are described in detail above, and the principle and implementation manner of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A teaching method for a vision-based humanoid robot, characterized by, The method comprises: acquiring a human demonstration image of a demonstrator in a current scene through a camera assembly; identifying and analyzing the human demonstration images of continuous frames to obtain human key point coordinates in each movement stage, determining robot joints corresponding to each human key point, and calculating joint motion states of each control joint of the corresponding robot; The identification and analysis of the human demonstration images of continuous frames to obtain human key point coordinates in each movement stage, determine robot joints corresponding to each human key point, and calculate joint motion states of each control joint of the corresponding robot, comprises: The human body demonstration image of continuous frames is recognized and analyzed to obtain human body key point coordinates in each movement stage, the robot joints corresponding to each human body key point are determined, and the human body key points are input into a motion reorientation formula to calculate the joint motion state of each control joint of the corresponding robot; the motion reorientation formula is: ; wherein, represents human skeleton key point coordinates; represents kinematic constraint combination, including joint angle limit , self-collision avoidance constraint ; represents an optimization function for minimizing the position error of the human body and the robot end. The identification and analysis of the human demonstration images of continuous frames to obtain human key point coordinates in each movement stage, determine robot joints corresponding to each human key point, comprises: preprocessing the human demonstration images of continuous frames to obtain preprocessed images, wherein the preprocessing comprises image size adjustment, normalization processing, and color space conversion; analyzing the preprocessed images through a human pose estimation model to obtain human key point coordinates in each movement stage and corresponding confidence, and filtering out low-confidence key point coordinates; determining human key point corresponding human joint information according to a set key point mapping rule, and determining robot joints corresponding to each human key point according to the human pose estimation model; generating simulation control instructions for each joint of the humanoid robot according to the joint motion states of each control joint, sending the simulation control instructions to the corresponding humanoid robot for motion simulation, and saving the simulation control instructions when the motion simulation result meets the requirements.

2. The teaching method of a vision-based humanoid robot according to claim 1, wherein The human demonstration images comprise upper limb demonstration images and lower limb demonstration images; The identification and analysis of the human demonstration images of continuous frames to obtain human key point coordinates in each movement stage, determine robot joints corresponding to each human key point, and calculate joint motion states of each control joint of the corresponding robot, comprises: identifying the upper limb demonstration images of continuous frames to determine upper limb key point coordinates in each movement stage, and determining corresponding human joint angles according to the upper limb key point coordinates in each movement stage; determining the correspondence between the upper limb key points and the robot upper limb joints, and inputting the human joint angles into an upper limb mapping formula to obtain robot joint angles of each upper limb joint of the corresponding robot; the upper limb mapping formula is: ; wherein, respectively represent the shoulder joint angle, the elbow joint angle, and the wrist joint angle of the robot. respectively represent the corresponding shoulder joint angle, elbow joint angle and wrist joint angle of the human body; K is 3 3 a pair of diagonal scaling matrices, used to match the difference in joint motion range between the human body and the robot; is 3 3 a rotation sequence conversion matrix, used to convert the coordinates of the human body and the robot; respectively are the offset amounts of the corresponding joints of the robot and the human body; identifying the lower limb demonstration images of continuous frames to determine lower limb key point coordinates in each movement stage, and determining the correspondence between the lower limb key points and the robot lower limb joints according to a human-robot mapping relationship; converting the lower limb key point coordinates into robot coordinate system-based robot lower limb joint coordinates, and inputting the robot lower limb joint coordinates into a lower limb motion mapping formula for inverse kinematics calculation to obtain robot joint angles of each robot lower limb joint; the lower limb motion mapping formula is: wherein, is a robot lower limb joint angle vector; represents 3D coordinate positions of human ankle and hip joints; represents inverse kinematics solver.

3. The teaching method of a vision-based humanoid robot according to claim 2, wherein The human demonstration images further comprise hand demonstration actions; the demonstration method further comprises: A high-precision camera is used to acquire hand movements, and a mapping relationship between finger joints and robot finger joints is determined; and a finger joint control instruction of a humanoid robot is generated.

4. The teaching method of a vision-based humanoid robot according to claim 1, wherein After the simulation control instruction is sent to the corresponding humanoid robot for movement simulation, the method further includes: generating a corresponding robot joint sequence according to the simulation control instruction, the robot joint sequence including a robot execution timestamp, a joint name list, a trajectory point array, and a timestamp mapping table; synchronously playing a human movement video and a movement execution screen of the humanoid robot, wherein each frame of image in the human movement video is aligned with the movement of the humanoid robot through the timestamp mapping table; when movement adjustment is needed, performing left-right dragging operation along a time axis according to a user-selected key frame or modifying joint angle data in a joint angle panel; and updating a position of the key frame on the time axis in real time and previewing a robot posture change in a simulation screen.

5. The teaching method of a vision-based humanoid robot according to claim 1, wherein The method includes: inputting the continuous frames of human demonstration images into a Transformer-based self-supervised pre-training model for processing to obtain an action feature sequence and an action semantic vector; performing comparative learning processing on the action feature sequence and the action semantic vector to make features of similar actions close to each other in a hidden space and features of different actions far away from each other in the hidden space; reconstructing standardized motion data from the action feature sequence and the action semantic vector after the comparative learning processing by the Transformer-based model, and generating joint motion states of each control joint of the corresponding robot based on the standardized motion data.

6. A teaching system for a vision-based humanoid robot, characterized by The method includes: an image acquisition module configured to acquire human demonstration images of a demonstration person in a current scene through a camera assembly; an analysis module configured to identify and analyze the continuous frames of human demonstration images to obtain human key point coordinates in each movement stage, determine robot joints corresponding to each human key point, and calculate joint motion states of each control joint of the corresponding robot; The method includes: The human body demonstration image of continuous frames is recognized and analyzed to obtain human body key point coordinates in each movement stage, the robot joints corresponding to each human body key point are determined, and the human body key points are input into a motion reorientation formula to calculate the joint motion state of each control joint of the corresponding robot; the motion reorientation formula is: ; wherein, represents the human skeleton key point coordinates; represents the kinematic constraint combination, including joint angle limiting , self-collision avoidance constraint ; represents the optimization function for minimizing the human and robot end position error; the human demonstration image of the continuous frame is identified and analyzed to obtain the human key point coordinates in each moving stage, and the robot joints corresponding to each human key point are determined, including: preprocessing the continuous frames of human demonstration images to obtain preprocessed images, the preprocessing including image size adjustment, normalization processing, and color space conversion; analyzing the preprocessed images through a human pose estimation model to obtain human key point coordinates and corresponding confidence levels in each movement stage, and filtering out low-confidence key point coordinates; determining human key points according to the human pose estimation model, determining corresponding human joint information according to a set key point mapping rule, and determining robot joints corresponding to each human key point. The action execution module is configured to generate simulation control instructions of each joint of the humanoid robot according to the joint motion state of each control joint, and send the simulation control instructions to the corresponding humanoid robot for action simulation; and when the action simulation result meets the requirement, save the simulation control instructions.

7. An electronic device, comprising: The method comprises the following steps: a memory storing executable program codes; a processor coupled with the memory; the processor invokes the executable program codes stored in the memory, and executes the teaching method of the visual-based humanoid robot according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program enables the computer to execute the teaching method of the visual-based humanoid robot according to any one of claims 1 to 5.

Images