Multi-vision sensor calibration method of humanoid robot and related equipment

Abstract

The embodiment of the invention provides a multi-vision sensor calibration method of a humanoid robot and related equipment, and belongs to the technical field of sensor calibration. The calibration data can be collected in real time, the absolute external parameters of the visual sensor are determined according to the calibration object, the relative external parameters between different sensors are further obtained, all the absolute external parameters and the relative external parameters serve as reference data, the optimized external parameter set of the visual sensor is established, and the visual sensor is calibrated through the finally obtained optimized external parameter set. Through mutual correction of the multiple vision sensors, the reliability of calibrated external parameters is improved, the external parameters of the vision sensors of the humanoid robot are accurately corrected, and normal operation of the humanoid robot is maintained.

Description

Technical Field

[0001] This application relates to the field of sensor calibration technology, and in particular to a multi-vision sensor calibration method and related equipment for a humanoid robot. Background Technology

[0002] The extrinsic parameters of a humanoid robot's visual sensors (the rotation matrix R and translation vector t of the sensor in the world coordinate system) are the core parameters for solving 3D environmental information, directly determining the accuracy of perception results such as target distance and posture. Furthermore, the consistency of extrinsic parameters among multiple sensors is a prerequisite for cross-sensor data fusion (such as the fusion of depth data from a binocular camera and semantic data from a monocular camera). However, humanoid robots need to move frequently in actual work (such as walking, turning their heads, and swinging their arms). Sensor poses can experience "extrinsic parameter drift" due to limb vibrations and joint deformations. Static calibration results cannot be corrected in real time, leading to a decrease in perception accuracy over time.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a multi-vision sensor calibration method and related equipment for humanoid robots, so as to accurately correct the extrinsic parameters of the humanoid robot's vision sensors and maintain the normal operation of the humanoid robot.

[0005] To achieve the above objectives, one aspect of this application proposes a multi-vision sensor calibration method for a humanoid robot, the method comprising: Acquire calibration data of the humanoid robot, the calibration data including the humanoid robot's torso pose data, the humanoid robot's joint motion data, the humanoid robot's position data, and an environmental image containing multiple calibration objects around the humanoid robot, each of the environmental images including at least three calibration objects, and the humanoid robot is fixed with multiple vision sensors; Based on the pixel position of the calibration object in the environmental image, determine the absolute coordinates of the calibration object in the world coordinate system, and determine the initial position of the visual sensor in the world coordinate system; The absolute extrinsic parameters of the visual sensor are determined by combining the pixel position, the absolute coordinates, and the initial position. Determine the relative extrinsic parameters between two adjacent vision sensors, wherein the environmental images projected by the two adjacent vision sensors contain at least one identical calibration object; An objective function is established by combining the absolute extrinsic parameters and the relative extrinsic parameters. An optimized extrinsic parameter set corresponding to the objective function is obtained. The absolute extrinsic parameters are modified using the optimized extrinsic parameter set, and the visual sensor is calibrated using the modified absolute extrinsic parameters.

[0006] In some embodiments, determining the absolute coordinates of the calibration object in the world coordinate system based on the pixel position of the calibration object in the environmental image includes: The ORB-RANSAC algorithm was used to extract the two-dimensional pixel position of the calibration object in the environmental image. Obtain the map data stored in the humanoid robot; By combining the map data and the two-dimensional pixel positions, the absolute coordinates of the calibration object in the world coordinate system are determined.

[0007] In some embodiments, determining the absolute extrinsic parameters of the visual sensor by combining the pixel position, the absolute coordinates, and the initial position includes: By combining the pixel position, the absolute coordinates, and the initial position, an improved PnP function is constructed; The PnP function is iterated using the LM method until the absolute extrinsic parameters of the vision sensor are determined.

[0008] In some embodiments, determining the initial position of the visual sensor in the world coordinate system includes: Based on the joint motion data and the robot DH parameter table, the end pose of the humanoid robot limb is determined by combining the forward kinematics model. The vision sensor is installed on the humanoid robot limb, and the end of the limb is the end away from the joint. By combining the installation offset of the visual sensor in the humanoid robot's limb with the position data, the initial position of the visual sensor in the world coordinate system is determined.

[0009] In some embodiments, the step of combining the absolute extrinsic parameters and the relative extrinsic parameters to establish the objective function and obtaining the optimized extrinsic parameter set corresponding to the objective function includes: The reprojection error term of the visual sensor is determined based on the pixel position, the absolute coordinates, and the absolute extrinsic parameters; The consistency error term between the visual sensors is determined based on the relative extrinsic parameters and the absolute extrinsic parameters; Determine the weight of the consistency error term; By combining the reprojection error term, the consistency error term, and the weights of the consistency error term, an objective function is established; Obtain the optimized extrinsic parameter set corresponding to the objective function.

[0010] In some embodiments, before determining the absolute coordinates of the calibration object in the world coordinate system based on the pixel position of the calibration object in the environmental image, and determining the initial position of the visual sensor in the world coordinate system, the method further includes: Determine the first instantaneous change value of the joint movement velocity, and determine the second instantaneous change value of the trunk posture number; When the first instantaneous change value is greater than the first threshold, and / or the second instantaneous change value is greater than the second threshold, the absolute coordinates of the calibration object in the world coordinate system are determined based on the pixel position of the calibration object in the environmental image, and the initial position of the visual sensor in the world coordinate system is determined.

[0011] To achieve the above objectives, another aspect of this application provides a multi-vision sensor calibration device for a humanoid robot, the device comprising: The acquisition module is used to acquire the calibration data of the humanoid robot. The calibration data includes the torso pose data of the humanoid robot, the joint motion data of the humanoid robot, the position data of the humanoid robot, and an environmental image containing multiple calibration objects around the humanoid robot. Each environmental image includes at least three calibration objects, and the humanoid robot is fixed with multiple vision sensors. The first determining module is used to determine the absolute coordinates of the calibrator in the world coordinate system based on the pixel position of the calibrator in the environmental image, and to determine the initial position of the visual sensor in the world coordinate system. The second determining module is used to determine the absolute extrinsic parameters of the visual sensor by combining the pixel position, the absolute coordinates, and the initial position. The third determining module is used to determine the relative extrinsic parameters between two adjacent visual sensors, wherein the environmental images emitted by the two adjacent visual sensors contain at least one identical calibration object. The optimization module is used to establish an objective function by combining the absolute extrinsic parameters and the relative extrinsic parameters, obtain the optimized extrinsic parameter set corresponding to the objective function, modify the absolute extrinsic parameters using the optimized extrinsic parameter set, and calibrate the visual sensor using the modified absolute extrinsic parameters. This application proposes a humanoid robot, including: a controller, a visual sensor, a motion sensing component, and a positioning sensing component. The controller is connected to the visual sensor, the motion sensing component, and the positioning sensing component. The visual sensor is used to acquire environmental images of landmarks around the humanoid robot. The motion sensing component is used to acquire the torso pose data and joint motion data of the humanoid robot. The positioning sensing component is used to acquire the positioning data of the humanoid robot. The controller is used to acquire calibration data of the humanoid robot. The calibration data includes the humanoid robot's torso pose data, joint motion data, position data, and an environmental image containing multiple calibration objects around the humanoid robot. Each environmental image includes at least three calibration objects, and the humanoid robot is fixed with multiple vision sensors. Based on the pixel positions of the calibration objects in the environmental image, the controller determines the absolute coordinates of the calibration objects in the world coordinate system and determines the initial position of the vision sensors in the world coordinate system. Combining the pixel positions, absolute coordinates, and initial positions, the controller determines the absolute extrinsic parameters of the vision sensors. The controller also determines the relative extrinsic parameters between two adjacent vision sensors, where the environmental images projected by two adjacent vision sensors contain at least one identical calibration object. Finally, the controller establishes an objective function based on the absolute and relative extrinsic parameters, obtains an optimized extrinsic parameter set corresponding to the objective function, modifies the absolute extrinsic parameters using the optimized extrinsic parameter set, and calibrates the vision sensors using the modified absolute extrinsic parameters.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the methods described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above. The embodiments of this application include at least the following beneficial effects: This application provides a multi-vision sensor calibration method, device, electronic device, storage medium, and program product for a humanoid robot. This solution obtains the calibration data of the humanoid robot; determines the absolute coordinates of the calibration object in the world coordinate system based on the pixel position of the calibration object in the environmental image, and determines the initial position of the vision sensor in the world coordinate system; determines the absolute extrinsic parameters of the vision sensor by combining the pixel position, absolute coordinates, and initial position; determines the relative extrinsic parameters between two adjacent vision sensors, wherein the environmental images projected by the two adjacent vision sensors contain at least one identical calibration object; establishes an objective function by combining the absolute and relative extrinsic parameters, obtains the optimized extrinsic parameter set corresponding to the objective function, modifies the absolute extrinsic parameters using the optimized extrinsic parameter set, and calibrates the vision sensor using the modified absolute extrinsic parameters. This embodiment can collect calibration data in real time, determine the absolute extrinsic parameters of the vision sensor based on the calibration object, and further obtain the relative extrinsic parameters between different sensors. Using all absolute and relative extrinsic parameters as reference data, an optimized extrinsic parameter set for the vision sensor is established. The vision sensor is calibrated with the finally obtained optimized extrinsic parameter set. By using multiple vision sensors to mutually correct each other, the reliability of the calibrated extrinsic parameters is improved, the extrinsic parameters of the humanoid robot's vision sensor are accurately corrected, and the normal operation of the humanoid robot is maintained. Attached Figure Description

[0015] Figure 1 This is a structural diagram of the humanoid robot provided in an embodiment of this application; Figure 2 This is a flowchart of a multi-vision sensor calibration method for a humanoid robot provided in an embodiment of this application; Figure 3 This is a schematic diagram of a vision sensor in a humanoid robot provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the multi-vision sensor calibration device for a humanoid robot provided in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0018] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0019] 1) Humanoid robot: A type of robot designed to mimic human appearance and behavior, especially those with a body similar to that of a human.

[0020] 2) The improved PnP (Pose from N Points) function is an algorithm that calculates the position and orientation of the camera relative to the 3D object based on a set of 3D points on the known object and their corresponding 2D projections in the image.

[0021] 3) The LM algorithm (Levenberg-Marquardt method) is a parameter estimation method for solving nonlinear least squares problems. It achieves parameter optimization by combining the steepest descent method and the Gauss-Newton method.

[0022] 4) The ORB algorithm (Oriented FAST and Rotated BRIEF) is a fast computer vision algorithm for feature point extraction and description. It combines FAST feature detection and BRIEF descriptor and improves its noise resistance and rotation invariance.

[0023] 5) RANSAC (Random Sample Consensus) is an efficient method for estimating mathematical model parameters from a sample set containing outliers.

[0024] 6) The DH parameter table is a standardized method used in robot kinematics to describe the relative position and orientation between adjacent links.

[0025] 7) The Bundle-Adjustment algorithm, also known as the bundle adjustment method, uses the LM algorithm to minimize the error between the observed image point coordinates and the predicted image point coordinates.

[0026] 8) Reprojection error is a metric in computer vision that measures the difference between a 3D point projected onto an image plane and the actual observed point.

[0027] The extrinsic parameters of a humanoid robot's visual sensors (the rotation matrix R and translation vector t of the sensor in the world coordinate system) are the core parameters for solving 3D environmental information, directly determining the accuracy of perception results such as target distance and posture. Furthermore, the consistency of extrinsic parameters among multiple sensors is a prerequisite for cross-sensor data fusion (such as the fusion of depth data from a binocular camera and semantic data from a monocular camera). Mainstream sensor calibration methods employ a "calibration board + static photography" approach, requiring the robot to be fixed in a preset environment and extrinsic parameters to be calculated by photographing a dedicated calibration board such as a checkerboard. However, humanoid robots require frequent movement in actual work (such as walking, turning their heads, and swinging their arms), and sensor poses can experience "extrinsic parameter drift" due to limb vibrations and joint deformations. Static calibration results cannot be corrected in real time, leading to a decrease in perception accuracy over time.

[0028] In view of this, this application provides a method, apparatus, device and medium for using multiple vision sensors on a humanoid robot.

[0029] The multi-vision sensor calibration method for humanoid robots provided in this application relates to the field of information technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the multi-vision sensor calibration method for humanoid robots, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] This embodiment's solution is applied to a humanoid robot, specifically, to the controller of the humanoid robot. The humanoid robot provided in this embodiment, such as... Figure 1 As shown, its hardware configuration mainly includes: 1) Visual sensor 12 ① Head-mounted binocular camera: resolution 1280×720, focal length 800px, FOV 60° (global environmental awareness). ② Torso monocular camera: resolution 1280×720, focal length 800px, FOV 90° (mid-range perception); 3. Right arm depth camera: resolution 640×480, focal length 600px, ranging range 0.5-5m (operational perception). 2) Motion sensing component 13 ① Joint encoder: 12 axes (2 axes for head, 1 axis for torso, and 9 axes for limbs), sampling rate 100Hz, angle accuracy ±0.01° ② Torso IMU: 9-axis (accelerometer ±16g, gyroscope ±2000dps), sampling rate 200Hz, attitude angle accuracy ±0.1° 3) Positioning and sensing component 14 ① Laser SLAM module: LiDAR ranging range 0.1-10m, positioning accuracy 0.1m (indoor); ② GPS module: positioning accuracy 1m (outdoor).

[0032] 4) Controller 11 Main control unit: NVIDIA Jetson AGX Xavier (GPU accelerated); FPGA chip: Xilinx Zynq (hardware-accelerated matrix operations); Data bus (synchronization delay < 1ms).

[0033] The controller mentioned in this embodiment is the main control unit in the robot hardware configuration described above. The visual sensor, motion sensing component, and positioning sensing component are used to acquire corresponding sensor data. The controller is connected to the visual sensor, motion sensing component, and positioning sensing component through an FPGA chip and a data bus, respectively, and acquires the sensor data for further processing.

[0034] The controller obtains environmental images via vision sensors (head binocular camera at 15fps, torso monocular camera at 15fps, and arm depth camera at 30fps) that consist of at least three RGB / depth images containing calibration objects.

[0035] The controller obtains the joint angles output by the 12-axis joint encoder of the humanoid robot through the motion sensing component. and speed of movement .

[0036] The controller obtains the humanoid robot's pose angles through motion sensing components. (Roll / Pitch / Yaw) and acceleration.

[0037] The controller obtains the transformation relationship between the humanoid robot's base coordinate system and world coordinate system through the positioning and sensing components. (Identity matrix) (The origin of the base coincides with the origin of the world).

[0038] Figure 2 This is an optional flowchart of a multi-vision sensor calibration method for humanoid robots provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S201 to S205.

[0039] Step S201: Acquire calibration data of the humanoid robot. The calibration data includes the humanoid robot's torso pose data, joint motion data, position data, and an environmental image containing multiple markers around the humanoid robot. Each environmental image includes at least three markers. The humanoid robot is equipped with multiple visual sensors, such as... Figure 3 As shown, Figure 3 The image shows the robotic arm 31 of a humanoid robot, which is equipped with a vision sensor 32.

[0040] After acquiring calibration data in real time, this embodiment can also verify the instantaneous change value of the calibration data in real time, and determine whether to execute step S202 based on the instantaneous change value.

[0041] Specifically, this includes: determining the first instantaneous change value of joint movement velocity, and determining the second instantaneous change value of trunk posture digits; When the first instantaneous change value is greater than the first threshold, and / or the second instantaneous change value is greater than the second threshold, the absolute coordinates of the calibration object in the world coordinate system are determined based on the pixel position of the calibration object in the environmental image, and the initial position of the vision sensor in the world coordinate system is determined.

[0042] For example, the speed of robot joint movement ( ) and trunk acceleration ( ),when or Trigger step S202.

[0043] Example illustration: Upon detecting the movement speed of the robot's right arm joint (Vigorous exercise) triggers dynamic updates, that is, parameter calibration is performed on the vision sensor installed on the right arm.

[0044] Step S202: Based on the pixel position of the calibration object in the environmental image, determine the absolute coordinates of the calibration object in the world coordinate system, and determine the initial position of the vision sensor in the world coordinate system.

[0045] In some embodiments, determining the absolute coordinates of the calibration object in the world coordinate system includes: The ORB-RANSAC algorithm is used to extract the two-dimensional pixel positions of the calibration objects in the environmental image. ; Retrieve map data stored in the humanoid robot; By combining map data and 2D pixel locations, the absolute coordinates of the calibration object in the world coordinate system are determined. .

[0046] The preprocessing workflow for environmental images containing calibration objects includes: grayscale conversion → Gaussian filtering → Canny edge detection. Then, the ORB algorithm is used to extract feature points of natural calibration objects P1, P2, and P3, and the BRIEF descriptor is calculated. Mismatched points are removed using the RANSAC algorithm (with an inlier threshold of 2px), resulting in the two-dimensional pixel coordinates of each calibration object. .

[0047] By combining home CAD drawings (pre-existing map data), the absolute coordinates of the calibration object in the world coordinate system are obtained.

[0048] In some embodiments, determining the initial position of the vision sensor in the world coordinate system includes: Based on joint motion data and robot DH parameter table, combined with forward kinematics model, the end pose of humanoid robot limb is determined. Visual sensors are installed on humanoid robot limbs, with the end of the limb being the one furthest from the joint. By combining the installation offset and position data of the vision sensor in the humanoid robot's limbs, the initial position of the vision sensor in the world coordinate system is determined. .

[0049] The limbs (such as the head and arms) of a humanoid robot are composed of "links and joints". The visual sensors are fixed to the links. Among the links, those farther away from the joints are considered to be the limb ends.

[0050] First, the end-effector pose of the limb is calculated when the joints move the limb. Specifically, this is based on the robot's DH parameter table and joint angles. (k is the number of joints), calculate the pose of the limb end where sensor i is located relative to the base coordinate system using the forward kinematics formula.

[0051] (1) in, Let m be the rotation matrix for the m-th joint. Let m be the length vector of the m-th link. For the link rotation matrix, Let be the translation vector of the link. Let n be the rotation matrix of the nth joint, used to describe the rotation of the nth joint about its axis. Post-attitude transformation; It is the angle of the nth joint, that is, the actual rotation angle of the joint (collected by the joint encoder, which is one of the input parameters for forward kinematics calculation).

[0052] Compensation for sensor mounting offset: (2) in, Let i be the pose of sensor i relative to the base coordinate system. Let i be the rotational pose of sensor i relative to the base. The translational pose of sensor i relative to the base, and the pose deviation of the sensor coordinate system relative to its end effector, including rotational offset. Translation offset .

[0053] Convert to world coordinates: (3) in, This represents the transformation relationship from the base coordinate system to the world coordinate system.

[0054] Step S203: Combine pixel position, absolute coordinates and initial position to determine the absolute extrinsic parameters of the vision sensor.

[0055] An improved PnP function is constructed by combining pixel position, absolute coordinates, and initial position; The PnP function is iterated using the LM method until the absolute extrinsic parameters of the vision sensor are determined. ).

[0056] The objective function of the improved PnP algorithm is: (4) in, For camera perspective projection function, . The focal length is the intrinsic parameter of the sensor. This is the principal point of the image.

[0057] The weights of the kinematic pose regularization term are used to constrain the deviation between the sensor pose and the initial kinematic pose during the optimization process. Let Frobenius be the matrix norm.

[0058] The absolute extrinsic parameters of the visual sensor are finally obtained by iterative optimization using the LM algorithm.

[0059] Step S204: Determine the relative extrinsic parameters between two adjacent visual sensors, wherein the environmental images projected by the two adjacent visual sensors contain at least one identical calibration object.

[0060] For any sensor pair ( Match the feature points of the same natural calibration object captured by both images, and calculate the fundamental matrix. And decompose to obtain the essential matrix SVD decomposition Derive the relative extrinsic parameters of sensor a with respect to sensor b .

[0061] Step S205: Establish the objective function by combining absolute and relative extrinsic parameters, obtain the optimized extrinsic parameter set corresponding to the objective function, modify the absolute extrinsic parameters using the optimized extrinsic parameter set, and calibrate the vision sensor using the modified absolute extrinsic parameters. The projection error of the relative extrinsic parameters in the objective function is minimized.

[0062] With all absolute external parameters and relative external parameters To constrain the objective function, construct the objective function. (5) The adaptive weighted bundle-adjustment algorithm is used for iterative optimization, and the optimized extrinsic parameter set is output. ;in, This is the single-sensor reprojection error term. This is the consistency error term between sensors. These are weighting coefficients that are dynamically adjusted during the optimization process.

[0063] Among them, weight The calculation method is as follows: (6) Basic weights; The average reprojection error is: (7) The average consistency error is: (8) To avoid the minimum value where the denominator is zero.

[0064] This embodiment can collect calibration data in real time, determine the absolute extrinsic parameters of the vision sensor based on the calibration object, and further obtain the relative extrinsic parameters between different sensors. Using all absolute and relative extrinsic parameters as reference data, an optimized extrinsic parameter set for the vision sensor is established. The vision sensor is calibrated with the finally obtained optimized extrinsic parameter set. By using multiple vision sensors to mutually correct each other, the reliability of the calibrated extrinsic parameters is improved, the extrinsic parameters of the humanoid robot's vision sensor are accurately corrected, and the normal operation of the humanoid robot is maintained.

[0065] In a home environment, the robot continuously moved for 30 minutes (including walking, turning its head, swinging its arm, and grasping) to verify the calibration effect. The verification results are shown in Table 1.1.

[0066] Table 1.1 Verification and comparison results of the calibration method in this embodiment Let's take a specific example to illustrate this: The pixels of the calibration object extracted from the calibration object captured by the head camera of the humanoid robot are:

[0067] The pixels of the calibration object extracted from the calibration object captured by the camera on the humanoid robot's torso are:

[0068] The pixels of the calibration object extracted from the calibration object captured by the camera on the humanoid robot arm are:

[0069] By combining home CAD drawings (prior map data), obtain the absolute coordinates of the calibration object in the world coordinate system:

[0070] According to the robot DH parameter table (neck link length) torso height ), combined with the cervical joint angle (Looking up and down) (Yaw), calculate the pose of the head and limb extremities relative to the base: Rotation matrix:

[0071] Translation vector:

[0072] The mounting offset of the head camera relative to the extremities of the head and limbs is:

[0073] but

[0074]

[0075] Perform world coordinate system transformation. hour:

[0076]

[0077] Similarly, calculate the initial pose of the torso camera (associated with torso joint angles) and the arm camera (associated with shoulder / elbow / wrist joint angles). .

[0078] Calculate absolute extrinsic parameters by improving the PnP algorithm: Substitute head camera internal parameters Initial pose Natural markers and Construct an improved PnP objective function

[0079] The weight of the kinematic regularization term

[0080] Iterative optimization was performed using the Levenberg-Marquardt algorithm (50 iterations, convergence threshold 10). -6 ), to obtain the absolute extrinsic parameters of the head camera:

[0081]

[0082] Average reprojection error (Outperforms the traditional PnP algorithm by 2.8px) Repeat the above steps to obtain the absolute extrinsic parameters of the torso camera. Arm camera absolute external reference .

[0083] Constructing the relative extrinsic parameters between cameras (taking head camera A and torso camera B as an example): Extract images from A and B. ORB feature points, to obtain matching point pairs ; Solving the fundamental matrix from A to B using the 8-point algorithm ,satisfy The calculation yields:

[0084] according to Calculate the essential matrix Perform SVD decomposition, combined with P By applying an absolute coordinate scale constraint of 1, we obtain the relative extrinsic parameters: (Rotation of A relative to B); (Translation of A relative to B).

[0085] Similarly, calculate the relative extrinsic parameters of the head camera and the arm camera. Relative extrinsic parameters of torso camera and arm camera .

[0086] Substitute all absolute and relative extrinsic parameters to calculate the initial average reprojection error. Initial average consistency error Then adaptive weights Construct the objective function:

[0087] by To optimize the variables, an FPGA-accelerated Bundle-Adjustment algorithm was used for iterative optimization (100 iterations, convergence condition: variable update amount < 10). -6 The rate of change of the objective function is <10 -5 Output the globally optimal set of extrinsic parameters: Head camera:

[0088] Torso camera:

[0089] Arm camera:

[0090] After optimization: (Maximum positioning deviation between sensors < 2.5cm).

[0091] Please see Figure 4 This application also provides a multi-vision sensor calibration device for a humanoid robot, which can implement the above-described method. The device includes: The acquisition module 41 is used to acquire the calibration data of the humanoid robot. The calibration data includes the torso pose data of the humanoid robot, the joint motion data of the humanoid robot, the position data of the humanoid robot, and an environmental image containing multiple calibration objects around the humanoid robot. Each environmental image includes at least three calibration objects, and the humanoid robot is fixed with multiple vision sensors. The first determining module 42 is used to determine the absolute coordinates of the calibrator in the world coordinate system based on the pixel position of the calibrator in the environmental image, and to determine the initial position of the visual sensor in the world coordinate system. The second determining module 43 is used to determine the absolute extrinsic parameters of the visual sensor by combining the pixel position, the absolute coordinates and the initial position; The third determining module 44 is used to determine the relative extrinsic parameters between two adjacent visual sensors, wherein the environmental images emitted by the two adjacent visual sensors contain at least one identical calibration object. The optimization module 45 is used to establish an objective function by combining the absolute extrinsic parameters and the relative extrinsic parameters, obtain the optimized extrinsic parameter set corresponding to the objective function, modify the absolute extrinsic parameters using the optimized extrinsic parameter set, and calibrate the visual sensor using the modified absolute extrinsic parameters.

[0092] In some embodiments, the first determining module 42 is configured to: The ORB-RANSAC algorithm was used to extract the two-dimensional pixel position of the calibration object in the environmental image. Obtain the map data stored in the humanoid robot; By combining the map data and the two-dimensional pixel positions, the absolute coordinates of the calibration object in the world coordinate system are determined.

[0093] In some embodiments, the second determining module 43 is used to combine the pixel position, the absolute coordinates and the initial position to construct an improved PnP function; The PnP function is iterated using the LM method until the absolute extrinsic parameters of the vision sensor are determined.

[0094] In some embodiments, the second determining module 43 is used to determine the end pose of the humanoid robot limb end based on the joint motion data and the robot DH parameter table, combined with the forward kinematics model, wherein the vision sensor is installed on the humanoid robot limb, and the end of the limb is the end away from the joint; By combining the installation offset of the visual sensor in the humanoid robot's limb with the position data, the initial position of the visual sensor in the world coordinate system is determined.

[0095] In some embodiments, the optimization module 45 is configured to determine the reprojection error term of the visual sensor based on the pixel position, the absolute coordinates, and the absolute extrinsic parameters; The consistency error term between the visual sensors is determined based on the relative extrinsic parameters and the absolute extrinsic parameters; Determine the weight of the consistency error term; By combining the reprojection error term, the consistency error term, and the weights of the consistency error term, an objective function is established; Obtain the optimized extrinsic parameter set corresponding to the objective function.

[0096] In some embodiments, the first determining module 42 is configured to: Determine the first instantaneous change value of the joint movement velocity, and determine the second instantaneous change value of the trunk posture number; When the first instantaneous change value is greater than the first threshold, and / or the second instantaneous change value is greater than the second threshold, the absolute coordinates of the calibration object in the world coordinate system are determined based on the pixel position of the calibration object in the environmental image, and the initial position of the visual sensor in the world coordinate system is determined.

[0097] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0098] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0099] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0100] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the methods described in the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0101] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0102] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0103] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0104] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0105] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0106] The multi-vision sensor calibration method, apparatus, electronic device, storage medium, and program product for humanoid robots provided in this application embodiment acquire calibration data of the humanoid robot; determine the absolute coordinates of the calibration object in the world coordinate system based on the pixel position of the calibration object in the environmental image, and determine the initial position of the vision sensor in the world coordinate system; determine the absolute extrinsic parameters of the vision sensor by combining the pixel position, absolute coordinates, and initial position; determine the relative extrinsic parameters between two adjacent vision sensors, wherein the environmental images projected by the two adjacent vision sensors contain at least one identical calibration object; establish an objective function by combining the absolute and relative extrinsic parameters, obtain the optimized extrinsic parameter set corresponding to the objective function, modify the absolute extrinsic parameters using the optimized extrinsic parameter set, and calibrate the vision sensor using the modified absolute extrinsic parameters. This embodiment can collect calibration data in real time, determine the absolute extrinsic parameters of the vision sensor based on the calibration object, and further obtain the relative extrinsic parameters between different sensors. Using all absolute and relative extrinsic parameters as reference data, an optimized extrinsic parameter set for the vision sensor is established. The vision sensor is calibrated with the finally obtained optimized extrinsic parameter set. By using multiple vision sensors to mutually correct each other, the reliability of the calibrated extrinsic parameters is improved, the extrinsic parameters of the humanoid robot's vision sensor are accurately corrected, and the normal operation of the humanoid robot is maintained.

[0107] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0108] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0111] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0112] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0114] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for calibrating multiple vision sensors in a humanoid robot, characterized in that, include: Acquire calibration data of the humanoid robot, the calibration data including the humanoid robot's torso pose data, the humanoid robot's joint motion data, the humanoid robot's position data, and an environmental image containing multiple calibration objects around the humanoid robot, each of the environmental images including at least three calibration objects, and the humanoid robot is fixed with multiple vision sensors; Based on the pixel position of the calibration object in the environmental image, determine the absolute coordinates of the calibration object in the world coordinate system, and determine the initial position of the visual sensor in the world coordinate system; The absolute extrinsic parameters of the visual sensor are determined by combining the pixel position, the absolute coordinates, and the initial position. Determine the relative extrinsic parameters between two adjacent vision sensors, wherein the environmental images projected by the two adjacent vision sensors contain at least one identical calibration object; An objective function is established by combining the absolute extrinsic parameters and the relative extrinsic parameters. An optimized extrinsic parameter set corresponding to the objective function is obtained. The absolute extrinsic parameters are modified using the optimized extrinsic parameter set, and the visual sensor is calibrated using the modified absolute extrinsic parameters.

2. The method according to claim 1, characterized in that, Determining the absolute coordinates of the calibration object in the world coordinate system based on the pixel position of the calibration object in the environmental image includes: The ORB-RANSAC algorithm was used to extract the two-dimensional pixel position of the calibration object in the environmental image. Obtain the map data stored in the humanoid robot; By combining the map data and the two-dimensional pixel positions, the absolute coordinates of the calibration object in the world coordinate system are determined.

3. The method according to claim 1, characterized in that, Determining the absolute extrinsic parameters of the visual sensor by combining the pixel position, the absolute coordinates, and the initial position includes: By combining the pixel position, the absolute coordinates, and the initial position, an improved PnP function is constructed; The PnP function is iterated using the LM method until the absolute extrinsic parameters of the vision sensor are determined.

4. The method according to claim 1, characterized in that, Determining the initial position of the visual sensor in the world coordinate system includes: Based on the joint motion data and the robot DH parameter table, the end pose of the humanoid robot limb is determined by combining the forward kinematics model. The vision sensor is installed on the humanoid robot limb, and the end of the limb is the end away from the joint. By combining the installation offset of the visual sensor in the humanoid robot's limb with the position data, the initial position of the visual sensor in the world coordinate system is determined.

5. The method according to claim 1, characterized in that, The step of establishing an objective function by combining the absolute extrinsic parameters and the relative extrinsic parameters, and obtaining the optimized extrinsic parameter set corresponding to the objective function, includes: The reprojection error term of the visual sensor is determined based on the pixel position, the absolute coordinates, and the absolute extrinsic parameters; The consistency error term between the visual sensors is determined based on the relative extrinsic parameters and the absolute extrinsic parameters; Determine the weight of the consistency error term; By combining the reprojection error term, the consistency error term, and the weights of the consistency error term, an objective function is established; Obtain the optimized extrinsic parameter set corresponding to the objective function.

6. The method according to claim 1, characterized in that, Before determining the absolute coordinates of the calibration object in the world coordinate system based on the pixel position of the calibration object in the environmental image, and determining the initial position of the visual sensor in the world coordinate system, the method further includes: Determine the first instantaneous change value of the joint movement velocity, and determine the second instantaneous change value of the trunk posture number; When the first instantaneous change value is greater than the first threshold, and / or the second instantaneous change value is greater than the second threshold, the absolute coordinates of the calibration object in the world coordinate system are determined based on the pixel position of the calibration object in the environmental image, and the initial position of the visual sensor in the world coordinate system is determined.

7. A humanoid robot, characterized in that, include: The system includes a controller, a vision sensor, a motion sensing component, and a positioning sensing component. The controller is connected to the vision sensor, the motion sensing component, and the positioning sensing component, respectively. The vision sensor is used to acquire environmental images of the markers around the humanoid robot. The motion sensing component is used to acquire the torso pose data and joint motion data of the humanoid robot. The positioning sensing component is used to acquire the positioning data of the humanoid robot. The controller is used to acquire calibration data of the humanoid robot. The calibration data includes the humanoid robot's torso pose data, the humanoid robot's joint motion data, the humanoid robot's position data, and an environmental image containing multiple calibration objects around the humanoid robot. Each environmental image includes at least three calibration objects, and the humanoid robot is fixed with multiple vision sensors. Based on the pixel position of the calibration objects in the environmental image, the controller determines the absolute coordinates of the calibration objects in the world coordinate system and determines the initial position of the vision sensors in the world coordinate system. By combining the pixel position, the absolute coordinates, and the initial position, the absolute extrinsic parameters of the vision sensor are determined; the relative extrinsic parameters between two adjacent vision sensors are determined, wherein the environmental images projected by the two adjacent vision sensors contain at least one identical calibration object; an objective function is established by combining the absolute extrinsic parameters and the relative extrinsic parameters, and an optimized extrinsic parameter set corresponding to the objective function is obtained; the absolute extrinsic parameters are modified using the optimized extrinsic parameter set, and the vision sensor is calibrated using the modified absolute extrinsic parameters.

8. A multi-vision sensor device for a humanoid robot, characterized in that, The device includes: The acquisition module is used to acquire the calibration data of the humanoid robot. The calibration data includes the torso pose data of the humanoid robot, the joint motion data of the humanoid robot, the position data of the humanoid robot, and an environmental image containing multiple calibration objects around the humanoid robot. Each environmental image includes at least three calibration objects, and the humanoid robot is fixed with multiple vision sensors. The first determining module is used to determine the absolute coordinates of the calibrator in the world coordinate system based on the pixel position of the calibrator in the environmental image, and to determine the initial position of the visual sensor in the world coordinate system. The second determining module is used to determine the absolute extrinsic parameters of the visual sensor by combining the pixel position, the absolute coordinates, and the initial position. The third determining module is used to determine the relative extrinsic parameters between two adjacent visual sensors, wherein the environmental images emitted by the two adjacent visual sensors contain at least one identical calibration object. An optimization module is used to establish an objective function by combining the absolute extrinsic parameters and the relative extrinsic parameters, obtain the optimized extrinsic parameter set corresponding to the objective function, modify the absolute extrinsic parameters using the optimized extrinsic parameter set, and calibrate the visual sensor using the modified absolute extrinsic parameters.

9. An electronic device / computer apparatus, characterized in that, The electronic device / computer apparatus includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images