Extrinsic calibration system, method and device for visual sensor of multi-legged robot

Abstract

The application discloses a multi-legged robot vision sensor extrinsic parameter calibration system, method and device. The technical scheme provided by the application comprises the following steps: obtaining a first transformation matrix of a foot coordinate system transformation to a body coordinate system, obtaining a second transformation matrix of a body vision coordinate system transformation to a world coordinate system, and a third transformation matrix of a foot vision coordinate system transformation to the world coordinate system; and calculating a fourth transformation matrix of the body vision coordinate system transformation to the foot vision coordinate system according to the second transformation matrix and the third transformation matrix. Then, the extrinsic parameter calibration is calculated according to the first transformation matrix, the second transformation matrix and the fourth transformation matrix. The relative state measurement between the foot vision coordinate system and the body coordinate system of the multi-legged robot is combined, the extrinsic parameter problem of the multi-legged robot is converted into a mechanical arm hand-eye calibration problem, a calibration board is loaded in the environment, and the extrinsic parameter calibration can be completed by the movement of the foot relative to the body coordinate system. The operation is simple, and the extrinsic parameter calibration precision is high.

Description

Technical Field

[0001] This application relates to the field of robot vision technology, and in particular to an external parameter calibration system, method, apparatus, device, and storage medium for a multi-legged robot vision sensor. Background Technology

[0002] Mobile robots are increasingly being deployed in outdoor and structured, complex environments. Compared to wheeled and tracked robots, multi-legged robots, with their animal-like leg structures, offer greater mobility. They can navigate or avoid obstacles with low energy consumption and achieve high-speed movement even in complex environments. Furthermore, they have advantages such as low requirements for surface conditions and minimal damage to the ground, making them widely applicable in exploration, search and rescue, and transportation scenarios. During robot movement, the robot estimates its own state and reconstructs its surroundings by integrating multiple visual sensors, inertial measurement units, and encoders from the leg motors. Accurate sensor calibration is a crucial prerequisite for sensor information fusion.

[0003] To achieve mapping, localization, and navigation based on vision sensors, it is necessary to calibrate the intrinsic and extrinsic parameters of the vision sensors mounted on the robot. Intrinsic parameter calibration of vision sensors can be relatively easily achieved by combining publicly available knowledge with different camera imaging models, such as pinhole cameras and fisheye cameras, to obtain parameters such as focal length, optical center, and distortion. However, extrinsic parameter calibration of vision sensors, which involves calculating the translation vector and rotation matrix between the robot's body coordinate system and the vision sensor coordinate system—a total of six degrees of freedom—is often referred to as "pose" or "attitude." Due to the unique kinematic model of multi-legged robots compared to wheeled and tracked robots, extrinsic parameter calibration of vision sensors is significantly more difficult, and mature solutions are scarce.

[0004] For the extrinsic parameter calibration of vision sensors on mobile robot platforms, current mainstream solutions are mostly based on the robot's ego-motion. This involves the robot moving in its environment, using the motor encoders and vision sensors mounted on the robot to generate two different trajectories, representing the robot's pose at different times. Then, mathematical methods such as nonlinear optimization are used to estimate the extrinsic parameters. However, in multi-legged robot systems, the above approach struggles to achieve accurate extrinsic parameter estimation. The main reasons include: 1. The legged odometry data obtained from the legged robot encoder has significant errors, meaning the robot's trajectory has large errors, making it difficult to use for high-precision extrinsic parameter calibration; 2. Multi-legged robots typically move in two-dimensional environments, such as walking freely on the ground, which cannot provide sufficient excitation for all six degrees of freedom, thus hindering high-precision calibration of all six degrees of freedom; 3. In addition to the above problems, the extrinsic parameter calibration of vision sensors also faces temporal deviations. Because the camera's hardware clock and the robot's motor / encoder clock cannot be strictly aligned via I / O signals, there is typically a deviation of 50-100 milliseconds on the computer host. Summary of the Invention

[0005] This application provides an extrinsic parameter calibration system, method, apparatus, device, and storage medium for a multi-legged robot's vision sensor, to accurately calibrate the extrinsic parameters between the multi-legged robot's body coordinate system and the body camera coordinate system.

[0006] In a first aspect, embodiments of this application provide an extrinsic parameter calibration system for a multi-legged robot vision sensor, comprising:

[0007] The system comprises a multi-legged robot, a calibration plate, and a processor. The multi-legged robot includes multiple legs, each equipped with a body vision sensor and a foot vision sensor. Both the body vision sensor and the foot vision sensor are connected to the processor. The body vision sensor acquires image information from the front of the multi-legged robot, while the foot vision sensors acquire image information from the front of their respective legs. The processor calculates a target transformation matrix from the body coordinate system to the body camera coordinate system based on the body vision coordinate system corresponding to the body vision sensor and the foot vision coordinate system corresponding to the foot vision sensor. The calibration plate is mounted in front of the multi-legged robot and falls within the line of sight of both the body vision sensor and the foot vision sensor.

[0008] In a second aspect, embodiments of this application provide a method for calibrating the extrinsic parameters of a vision sensor for a multi-legged robot, comprising:

[0009] Construct the foot coordinate system of the multi-legged robot, and obtain the first transformation matrix that transforms the foot coordinate system to the body coordinate system of the multi-legged robot;

[0010] The second transformation matrix for transforming the body vision coordinate system corresponding to the body vision sensor to the world coordinate system and the third transformation matrix for transforming the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system are obtained, wherein the world coordinate system is a coordinate system established with any point on the calibration plate as the origin.

[0011] A fourth transformation matrix is ​​calculated based on the second transformation matrix and the third transformation matrix to transform the body visual coordinate system to the foot visual coordinate system;

[0012] The target transformation matrix for transforming the body coordinate system to the body camera coordinate system is calculated based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

[0013] In a third aspect, embodiments of this application provide an extrinsic parameter calibration device for a multi-legged robot's vision sensor, comprising:

[0014] First matrix acquisition module: used to construct the foot coordinate system of the multi-legged robot and acquire the first transformation matrix of the foot coordinate system to the body coordinate system of the multi-legged robot;

[0015] The second matrix acquisition module is used to acquire the second transformation matrix of the body vision coordinate system corresponding to the body vision sensor to the world coordinate system and the third transformation matrix of the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system. The world coordinate system is a coordinate system established with any point on the calibration board as the origin.

[0016] The third matrix acquisition module is used to calculate a fourth transformation matrix based on the second transformation matrix and the third transformation matrix to transform the body visual coordinate system to the foot visual coordinate system.

[0017] The fourth matrix acquisition module is used to calculate the target transformation matrix from the body coordinate system to the body camera coordinate system based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

[0018] In a fourth aspect, embodiments of this application provide an external parameter calibration device, including: a memory and one or more processors;

[0019] The memory is used to store one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the extrinsic parameter calibration method for the multi-legged robot vision sensor as described in the second aspect.

[0021] In a fifth aspect, embodiments of this application provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the extrinsic parameter calibration method for a multi-legged robot vision sensor as described in the second aspect.

[0022] This application embodiment obtains a first transformation matrix for transforming the foot coordinate system to the body coordinate system of the multi-legged robot, a second transformation matrix for transforming the body visual coordinate system corresponding to the body visual sensor to the world coordinate system, and a third transformation matrix for transforming the foot visual coordinate system corresponding to the foot visual sensor to the world coordinate system. A fourth transformation matrix for transforming the body visual coordinate system to the foot visual coordinate system is calculated based on the second and third transformation matrices. Then, a target transformation matrix for transforming the body coordinate system to the body camera coordinate system is calculated based on the first, second, and fourth transformation matrices. By defining a new foot coordinate system for the multi-legged robot and fixing foot visual sensors to the feet of the multi-legged robot, and combining high-precision relative state measurements between the foot visual coordinate system and the body coordinate system of the multi-legged robot, the extrinsic parameter problem of the multi-legged robot is transformed into a robotic arm hand-eye calibration problem. During the extrinsic parameter calibration process, the multi-legged robot does not need to perform large-scale movements in the environment; only a calibration plate needs to be loaded in the environment, and the extrinsic parameter calibration can be completed by moving a single foot of the multi-legged robot relative to the body coordinate system. The operation is simple, and the extrinsic parameter calibration accuracy is high. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the external parameter calibration system for a vision sensor of a multi-legged robot provided in an embodiment of this application;

[0024] Figure 2 This is a flowchart of an extrinsic parameter calibration method for a vision sensor of a multi-legged robot provided in an embodiment of this application;

[0025] Figure 3 This is a flowchart of another method for calibrating the extrinsic parameters of a vision sensor for a multi-legged robot, provided in an embodiment of this application.

[0026] Figure 4 This is a flowchart of another method for calibrating the extrinsic parameters of a vision sensor for a multi-legged robot, provided in an embodiment of this application.

[0027] Figure 5 This is a schematic diagram of the external parameter calibration device for a multi-legged robot vision sensor provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an external parameter calibration device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0030] This application provides a method, apparatus, device, and storage medium for extrinsic parameter calibration of a vision sensor for a multi-legged robot. The extrinsic parameter calibration method can be executed by an extrinsic parameter calibration device. This extrinsic parameter calibration device can be implemented through software and / or hardware. The extrinsic parameter calibration device can consist of two or more physical entities, or it can consist of a single physical entity; this embodiment does not limit the specific implementation.

[0031] In this embodiment, the extrinsic parameter calibration device has the function of executing the complete process of the extrinsic parameter calibration method, including receiving settings for various coordinate systems and collecting posture or related data in different coordinate systems. For example, the extrinsic parameter calibration device can be a laptop computer, desktop computer, smart interactive tablet, or one or more processors integrated into the multi-legged robot, or even a server. In this embodiment, it is applied to a multi-legged robot. As the name suggests, a multi-legged robot adds a multi-legged structure to the basic structure of a regular intelligent robot, thus making it more flexible in crawling, walking, and other movements. Typically, each leg of a multi-legged robot has multiple joints, including the hip joint, thigh, calf, and ankle joint. Due to the special structure of multi-legged robots, they have a special kinematic model compared to ordinary robots, making the extrinsic parameter calibration of the body vision sensor of multi-legged robots more difficult. To achieve high-precision extrinsic parameter calibration of multi-legged robots, this embodiment innovatively installs an additional foot vision sensor on the foot of the multi-legged robot and defines a new foot coordinate system. See also Figure 1The illustrated extrinsic calibration system for a multi-legged robot's vision sensor includes a multi-legged robot 1, a calibration plate 3 external to the multi-legged robot 1, and a processor (not shown). The multi-legged robot 1 has multiple legs 2. A body vision sensor 4, which is also a camera, is mounted on the multi-legged robot 1 and is typically installed at the front or top to acquire image information from the front of the multi-legged robot 1. A foot vision sensor 5, which can also be a camera, is arbitrarily selected from among the multiple legs 2 and mounted on it to acquire image information from the front of the leg 2. Specifically, the foot vision sensor 5 is mounted at the joint of the leg 2, such as the contact point. The calibration plate 3 is mounted at the front of the multi-legged robot, and preferably uses an AprilGrid calibration plate. Specifically, in this embodiment, the calibration plate 3 falls within the line of sight of both the body vision sensor and the foot vision sensor. The processor calculates the target transformation matrix from the body coordinate system to the body camera coordinate system based on the body visual coordinate system corresponding to the body visual sensor 4 and the foot visual coordinate system corresponding to the foot visual sensor 5, which is the process of extrinsic parameter calibration. For the specific extrinsic parameter calibration process, please refer to the extrinsic parameter calibration method disclosed in the embodiments of this application. The extrinsic parameter calibration method for the multi-legged robot visual sensor provided in the embodiments of this application is applied to the above-mentioned multi-legged robot time sensor extrinsic parameter calibration system. By redefining the foot coordinate system of the multi-legged robot, fixing the foot visual sensor on the foot of the multi-legged robot, and combining high-precision relative state measurement between the foot visual coordinate system and the body coordinate system of the multi-legged robot, the extrinsic parameter problem of the multi-legged robot is transformed into a robotic arm hand-eye calibration problem. During the extrinsic parameter calibration process, the multi-legged robot does not need to perform large-scale movements in the environment; only a calibration plate needs to be loaded in the environment, and the extrinsic parameter calibration can be completed by moving a single foot of the multi-legged robot relative to the body coordinate system. The operation is simple and the extrinsic parameter calibration accuracy is high.

[0032] The following sections will provide detailed explanations of each point.

[0033] Figure 2 A flowchart of an extrinsic parameter calibration method for a multi-legged robot vision sensor provided in an embodiment of this application is given. See also... Figure 2 The extrinsic parameter calibration method for a multi-legged robot vision sensor includes:

[0034] Step 201: Construct the foot coordinate system of the multi-legged robot and obtain the first transformation matrix that transforms the foot coordinate system to the body coordinate system of the multi-legged robot.

[0035] In this embodiment, the foot coordinate system is a coordinate system constructed with the foot as the origin. The origin is preferably any point on the foot where the foot vision sensor is installed. For example, the location corresponding to the installation of the foot vision sensor can be chosen as the origin, or the bottom of the foot can be used as the origin. In other examples, other locations can be selected as the origin of the foot coordinate system according to actual needs. In coordinate system transformation, the transformation relationship between the two coordinate systems can be calculated based on the coordinate information of any point in one coordinate system and the coordinate information of any point in the other coordinate system. The transformation between the two coordinate systems can be calculated by obtaining the origins of each coordinate system. In application, the coordinate system is fixed to the rigid body corresponding to that coordinate system to represent the transformation between the coordinate systems, that is, the posture of one coordinate system relative to another.

[0036] In this embodiment, the feet of the multi-legged robot are in a special state, and their coordinate positions change during movement. To calculate the coordinate system transformed from the foot coordinate system to the multi-legged robot's body coordinate system, this embodiment obtains a pre-defined foot kinematic model of the multi-legged robot and encoded data collected by the foot motor encoder to realize the transformation matrix between the foot coordinate system and the multi-legged robot's body coordinate system. The kinematic model provides the transformation relationship from the multi-legged robot's body coordinate system to the multi-legged robot's foot hip joint coordinate system, the transformation relationship from the multi-legged robot's foot hip joint coordinate system to the multi-legged robot's knee joint coordinate system, and the transformation relationship from the multi-legged robot's knee joint coordinate system to the foot contact point coordinate system. Based on this, combined with the encoded data collected by the foot motor encoder, the first transformation matrix from the foot coordinate system to the multi-legged robot's body coordinate system can be calculated. The calculation of the first transformation matrix based on the foot kinematic model and the encoded data is a conventional method in the art, and the specific principle will not be elaborated here.

[0037] In a multi-legged robot system, inertial measurement units, motor encoders, and other components can obtain the motion posture of the corresponding rigid body at different times, and a series of motion postures constitute a trajectory. As mentioned above, after obtaining the first transformation matrix from the foot coordinate system to the body coordinate system of the multi-legged robot, one or a series of motion postures of the foot coordinate system relative to the body coordinate system can be obtained based on this first transformation matrix.

[0038] Step 202: Obtain the second transformation matrix from the body vision coordinate system corresponding to the body vision sensor to the world coordinate system, and the third transformation matrix from the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system.

[0039] In this embodiment, the body vision sensor is mounted on the multi-legged robot, and the foot vision sensor is mounted on the foot of the multi-legged robot. The world coordinate system is a coordinate system established with any point on the calibration plate as the origin. The calibration plate is a flat plate placed in front of the multi-legged robot and simultaneously within the line of sight of the body vision sensor and the foot vision sensor when calibrating the world coordinate system. In the application of this embodiment, it is used to correct lens distortion of the 3D camera, determine the conversion relationship between physical dimensions and pixels, and determine the relationship between the 3D geometric position of any point on the surface of a spatial object and its corresponding point in the image. The calibration plate is a flat plate with a fixed pattern array. In use, the camera in the system photographs the flat plate with the fixed-spacing pattern array. After algorithm calculation, the geometric model of the camera can be calculated accordingly, resulting in high-precision measurement and reconstruction.

[0040] In this embodiment, a body vision coordinate system and a foot vision coordinate system are first constructed. The body vision coordinate system is for the multi-legged robot's body vision sensor, i.e., the body camera's coordinate system, while the foot vision coordinate system is for the multi-legged robot's foot vision sensor, i.e., the foot camera's coordinate system. Specifically, the timestamps of the body vision sensor and the foot vision sensor are aligned, and a first image of the calibration board acquired by the body vision sensor at a target timestamp and a second image of the calibration board acquired by the foot vision sensor at the target timestamp are obtained. The size parameters of the calibration board are obtained. Using the PnP (Perspective-n-Points) problem-solving approach, a second transformation matrix is ​​obtained based on the size parameters and the first image to transform the body vision coordinate system to the world coordinate system. A third transformation matrix is ​​also obtained based on the size parameters and the second image to transform the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system. In step 202, strict hardware clock alignment between the body vision sensor and the foot vision sensor is not required. Software clock alignment can be achieved through a shared host and the system clock. The clock error can typically be controlled within 2-3 milliseconds, which meets the robot's accuracy requirements.

[0041] Step 203: Calculate the fourth transformation matrix based on the second transformation matrix and the third transformation matrix to transform the body visual coordinate system to the foot visual coordinate system.

[0042] In this embodiment, the ultimate goal is to obtain the target transformation matrix from the body coordinate system to the body camera coordinate system. However, since it is difficult to directly estimate the relationship between the body coordinate system and the body camera coordinate system, this embodiment uses a custom foot coordinate system to calculate the first transformation matrix, the second transformation matrix, and the third transformation matrix, and then continues to calculate the fourth transformation matrix to jointly derive the target transformation matrix.

[0043] In this step, combined with the extrinsic parameter calibration system for the multi-legged robot vision sensor disclosed in this application, since both the body vision sensor and the foot vision sensor observe the same calibration plate, the transformation relationship between the body vision coordinate system corresponding to the body vision sensor and the foot vision coordinate system corresponding to the foot vision sensor can be obtained based on the second transformation matrix and the third transformation matrix.

[0044] More specifically, the embodiment calculates the fourth transformation matrix from the body visual coordinate system to the foot visual coordinate system using the following formula:

[0045]

[0046] Where Fw represents the world coordinate system, Indicates the body camera coordinate system. Represents the visual coordinate system of the foot. This is the second transformation matrix, which is also the body camera coordinate system. Transformation of the world coordinate system Fw This is the third transformation matrix. This is the fourth transformation matrix. This is the transformation matrix from the world coordinate system to the foot visual coordinate system. In this embodiment, each transformation matrix includes a translation vector t and a rotation matrix R. Following the representation of homogeneous matrices, the transformation matrix is ​​expressed as:

[0047]

[0048] And 0T1 represents the transformation matrix from coordinate system 1 to coordinate system 0.

[0049] Step 204: Calculate the target transformation matrix from the body coordinate system to the body camera coordinate system based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

[0050] This embodiment achieves accurate extrinsic parameter calibration, which is derived using a first transformation matrix for transforming the foot coordinate system to the body coordinate system, a second transformation matrix for transforming the body visual coordinate system to the world coordinate system, a third transformation matrix for transforming the foot visual coordinate system to the world coordinate system, and a fourth transformation matrix for transforming the body visual coordinate system to the foot visual coordinate system, all of which are easy to measure accurately.

[0051] Specifically, in this step, Formula 3 is obtained by transforming Formula 2 and Formula 1. Formula 2 is used to calculate the transformation matrix from the foot coordinate system to the foot visual coordinate system. Formula 2 is as follows:

[0052]

[0053] Starting with Formula 2, the coordinate system is gradually transformed. Observations of the calibration plate by the body vision sensor and foot vision sensor at different times reveal the motion posture of the world coordinate system relative to the foot vision coordinate system obtained from the j-th measurement and calculation. Motion posture of the body's visual coordinate system relative to the world coordinate system The motion posture of the foot coordinate system relative to the body coordinate system The motion posture relative to the foot visual coordinate system obtained from the i-th measurement and calculation. Motion posture of the body's visual coordinate system relative to the world coordinate system The motion posture of the foot coordinate system relative to the body coordinate system There are differences in the results, but since the foot vision sensor is fixedly installed on the foot of the multi-legged robot, the transformation matrix from the foot coordinate system to the foot vision coordinate system will not change no matter how many times the calibration plate is observed, or in other words, the motion posture result of the foot coordinate system relative to the foot vision coordinate system remains unchanged, that is, the calculation result on the left side of the equation in Formula 2 remains unchanged.

[0054] Based on this, in this step, the calibration plate is image acquired using the body vision sensor and the foot vision sensor. This makes the formula obtained according to Formula 2 as follows:

[0055]

[0056] Transforming Formula 1, we get:

[0057]

[0058] Based on formula Based on the formula Transformed into:

[0059]

[0060] Formula Multiply both sides of the equation by and That is, multiply both sides of the equation in Formula 5 by... Also multiplied by Formula 3 is obtained:

[0061]

[0062] The target transformation matrix is ​​calculated using Formula 3. Specifically, Formula 3 can be rewritten as the classical equation AX = XB, where... Substituting the values ​​of A, B, and X from Formula 3 above into AX = XB, and solving the equation using a known method, we can obtain the target transformation matrix, which is the motion attitude of the body coordinate system relative to the body camera coordinate system.

[0063] Among them, F l Using the foot coordinate system, Let be the target transformation matrix. This is the transformation matrix from the foot coordinate system to the foot visual coordinate system. b T l Let be the first transformation matrix, where i is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the i-th acquisition, and j is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the j-th acquisition. Let j be the j-th motion posture in the world coordinate system relative to the foot visual coordinate system. The i-th motion posture in the world coordinate system relative to the foot visual coordinate system. Let j be the j-th motion pose of the body's visual coordinate system relative to the world coordinate system. The i-th motion pose of the body's visual coordinate system relative to the world coordinate system. Let be the j-th motion posture of the foot coordinate system relative to the body coordinate system. Let i be the i-th motion posture of the foot coordinate system relative to the body coordinate system. The j-th motion posture of the body visual coordinate system relative to the foot visual coordinate system.

[0064] Figure 3 Another method for calibrating the extrinsic parameters of a multi-legged robot's vision sensor, according to an embodiment of this application, is shown below. Figure 3 The extrinsic parameter calibration method for the vision sensor of this multi-legged robot includes:

[0065] Step 301: Construct the foot coordinate system of the multi-legged robot and obtain the first transformation matrix that transforms the foot coordinate system to the body coordinate system of the multi-legged robot.

[0066] Similar to the previous embodiment, the foot coordinate system in this embodiment is a coordinate system constructed with the foot as the origin. To obtain the first transformation matrix from the foot coordinate system to the body coordinate system of the multi-legged robot, the transformation from the foot coordinate system to the body coordinate system of the multi-legged robot is achieved by acquiring the preset foot kinematic model of the multi-legged robot and the encoded data collected by the foot motor encoder of the multi-legged robot. The kinematic model provides the transformation relationship from the body coordinate system of the multi-legged robot to the hip joint coordinate system of the foot of the multi-legged robot, the transformation relationship from the hip joint coordinate system of the foot of the multi-legged robot to the knee joint coordinate system of the multi-legged robot, and the transformation relationship from the knee joint coordinate system of the knee joint coordinate system to the foot contact point coordinate system. Based on this, combined with the encoded data collected by the foot motor encoder, the first transformation matrix from the foot coordinate system to the body coordinate system of the multi-legged robot can be calculated.

[0067] Step 302: Obtain the second transformation matrix from the body vision coordinate system corresponding to the body vision sensor to the world coordinate system, and the third transformation matrix from the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system.

[0068] In conjunction with the extrinsic parameter calibration system for the multi-legged robot's vision sensors, the aforementioned body vision sensor is mounted on the multi-legged robot, and the foot vision sensor is mounted on the foot of the multi-legged robot. The world coordinate system is a coordinate system established with any point on the calibration plate as the origin. The calibration plate is a flat plate placed in front of the multi-legged robot and simultaneously falling within the line of sight of the body vision sensor and the foot vision sensor when calibrating the world coordinate system.

[0069] First, a body vision coordinate system and a foot vision coordinate system are constructed. The body vision coordinate system is for the multi-legged robot's body vision sensor, i.e., the body camera's coordinate system, while the foot vision coordinate system is for the multi-legged robot's foot vision sensor, i.e., the foot camera's coordinate system. Specifically, the timestamps of the body vision sensor and the foot vision sensor are aligned, and a first image of the calibration board acquired by the body vision sensor at a target timestamp and a second image of the calibration board acquired by the foot vision sensor at the target timestamp are obtained. The size parameters of the calibration board are then obtained. Using the PnP (Perspective-n-Points) problem-solving approach, a second transformation matrix is ​​obtained based on the size parameters and the first image to transform the body vision coordinate system to the world coordinate system. Finally, a third transformation matrix is ​​obtained based on the size parameters and the second image to transform the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system.

[0070] Step 303: Calculate the fourth transformation matrix based on the second and third transformation matrices to transform the body visual coordinate system to the foot visual coordinate system.

[0071] The examples are based on formulas Complete the calculation of the fourth transformation matrix. Here, Fw represents the world coordinate system. Indicates the body camera coordinate system. Represents the visual coordinate system of the foot. This is the second transformation matrix, which is also the body camera coordinate system. Transformation of the world coordinate system Fw This is the third transformation matrix. This is the fourth transformation matrix. This is the transformation matrix from the world coordinate system to the foot visual coordinate system.

[0072] Step 304: Calculate the target transformation matrix from the body coordinate system to the body camera coordinate system based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

[0073] In this step, the target transformation matrix is ​​calculated through a series of formula derivations. First, from the formula... By starting with a gradual transformation of the coordinate system, we can derive the following: Based on this formula Transform into According to the formula Formula Can be transformed into At this time in the formula Multiply both sides simultaneously and That is, multiplying both sides of the equation by . Also multiplied by Formula Formula Rewritten as the classic equation AX = XB, where Substituting the above formula and the corresponding values ​​of A, B, and X into AX = XB, and solving the equation using a known solution method, we can obtain the target transformation matrix, which is the motion attitude of the body coordinate system relative to the body camera coordinate system.

[0074] In the above, F l Using the foot coordinate system, Let be the target transformation matrix. This is the transformation matrix from the foot coordinate system to the foot visual coordinate system. b T l Let be the first transformation matrix, where i is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the i-th acquisition, and j is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the j-th acquisition. Let j be the j-th motion posture in the world coordinate system relative to the foot visual coordinate system. The i-th motion posture in the world coordinate system relative to the foot visual coordinate system. Let j be the j-th motion pose of the body's visual coordinate system relative to the world coordinate system. The i-th motion pose of the body's visual coordinate system relative to the world coordinate system. Let be the j-th motion posture of the foot coordinate system relative to the body coordinate system. Let i be the i-th motion posture of the foot coordinate system relative to the body coordinate system. The j-th motion posture of the body visual coordinate system relative to the foot visual coordinate system.

[0075] Step 305: Obtain the fifth transformation matrix for transforming the world coordinate system to the body visual coordinate system and the sixth transformation matrix for transforming the body coordinate system to the foot coordinate system.

[0076] In step 304 above, the target transformation matrix has been calculated. Using the coordinate system transformation method provided in this embodiment, the transformation matrix from the foot coordinate system to the foot visual coordinate system can be further deduced.

[0077] The fifth transformation matrix for transforming from the world coordinate system to the ontology visual coordinate system is: The sixth transformation matrix for transforming the body coordinate system to the foot coordinate system is: l T b .

[0078] Step 306: Calculate the transformation matrix from the foot coordinate system to the foot visual coordinate system based on the target transformation matrix, the third transformation matrix, the fifth transformation matrix, and the sixth transformation matrix.

[0079] Based on the third, fifth, and sixth transformation matrices and the target transformation matrix obtained above, the transformation matrix from the foot coordinate system to the foot visual coordinate system to be calculated in this embodiment is obtained.

[0080] The specific transformation derivation process is similar to the process of obtaining the target transformation matrix. First, from the formula... Let's begin the derivation process. l T b This is the sixth transformation matrix for transforming the body coordinate system to the foot coordinate system. Similarly, by observing the calibration board at different times, it can be found that the motion posture of the world coordinate system relative to the body visual coordinate system obtained from the j-th measurement and calculation is... Movement posture of the foot visual coordinate system relative to the world coordinate system Motion posture of the body coordinate system relative to the foot coordinate system The motion posture relative to the foot visual coordinate system obtained from the i-th measurement and calculation. Motion posture of the body's visual coordinate system relative to the world coordinate system The motion posture of the foot coordinate system relative to the body coordinate system The results differ, but since the body vision sensor is fixedly mounted on the multi-legged robot, the equation can be obtained. For the formula Based on the transformation, the formula can be... Derived as Then, based on this formula, multiply both sides of the equation by... and The formula is as follows The transformation matrix from the foot coordinate system to the foot visual coordinate system can be calculated using the classical equation AX = XB. in, This is the fifth transformation matrix for transforming from the world coordinate system to the ontological visual coordinate system.

[0081] like Figure 4 As shown in the figure, this application embodiment also provides a method for calibrating the extrinsic parameters of a vision sensor for a multi-legged robot, see [link to relevant documentation]. Figure 4 The extrinsic parameter calibration method for the vision sensor of this multi-legged robot includes:

[0082] Step 401: Construct the foot coordinate system of the multi-legged robot and obtain the first transformation matrix that transforms the foot coordinate system to the body coordinate system of the multi-legged robot.

[0083] Step 402: Obtain the second transformation matrix from the body vision coordinate system corresponding to the body vision sensor to the world coordinate system, and the third transformation matrix from the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system.

[0084] The body vision sensor is mounted on the multi-legged robot, and the foot vision sensor is mounted on the foot of the multi-legged robot. The world coordinate system is a coordinate system established with any point on the calibration plate as the origin. The calibration plate is a flat plate placed in front of the multi-legged robot and simultaneously within the line of sight of the body vision sensor and the foot vision sensor.

[0085] Step 403: Calculate the fourth transformation matrix based on the second transformation matrix and the third transformation matrix to transform the body visual coordinate system to the foot visual coordinate system;

[0086] Step 404: Calculate the target transformation matrix from the body coordinate system to the body camera coordinate system based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

[0087] In this embodiment, a significant time-domain difference exists between the visual sensor and the motor encoder included in the body coordinate system. The frequency of the visual sensor is typically 20-30Hz, while the motor encoder's measurement frequency is extremely fast, with the body coordinate system's state update reaching 1000Hz. Besides the frequency difference, there is also a general difference of tens of milliseconds in their clock references, known as clock skew, leading to time alignment problems between the different coordinate systems. To address this time alignment issue, this embodiment calibrates the clock skew, estimating the clock skew between the multi-legged robot's motion trajectory and the motion trajectory of the visual sensor.

[0088] Step 405: Obtain the first motion trajectory of the foot of the multi-legged robot relative to the multi-legged robot body and the second motion trajectory of the foot vision sensor relative to the body vision sensor.

[0089] In this step, the first motion trajectory of the multi-legged robot's feet relative to the robot's body and the second motion trajectory of the foot vision sensor relative to the body vision sensor are obtained. Specifically, the first transformation matrix obtained in step 401 calculates a motion posture of the foot coordinate system relative to the body coordinate system, and a series of motion postures constitutes the motion trajectory; that is, a series of motion postures of the foot coordinate system relative to the body coordinate system constitute the first motion trajectory. Correspondingly, the second motion trajectory in this step can be obtained based on the fourth transformation matrix calculated in step 403.

[0090] Step 406: Calculate the first average time interval between any two adjacent motion postures in the first motion trajectory and the second average time interval between any two adjacent motion postures in the second motion trajectory.

[0091] The first and second motion trajectories each include multiple motion postures, with a corresponding sampling interval between each pair of adjacent motion postures. In this step, the first average time interval of all sampling intervals for the first motion trajectory and the second average time interval of all sampling intervals for the second motion trajectory are calculated respectively.

[0092] Step 407: Calculate the clock deviation between the first motion trajectory and the second sensor motion trajectory based on the first average time interval and the second average time interval.

[0093] Specifically, the smaller of the first average time interval and the second average time interval is selected as the sampling time interval. At each sampling time interval, the corresponding motion posture is extracted from the first motion trajectory to form a first motion posture set, and at each sampling time interval, the corresponding motion posture is extracted from the second motion trajectory to form a second motion posture set. For example, if the first average time interval is t1 and the second average time interval is t2, and t1 is greater than t2, then t2 is used as the sampling time interval. Based on this sampling time interval t2, the first and second motion trajectories are re-acquired to form the first motion posture set and the second motion posture set. Then, the first rotation amount of the first motion posture set and the second rotation amount of the second motion posture set are calculated; the magnitude of the first angular velocity is calculated based on the first rotation amount, and the magnitude of the second angular velocity is calculated based on the second rotation amount; the convolution between the magnitude of the first angular velocity and the magnitude of the second angular velocity is calculated; the position corresponding to the maximum response value in the convolution is recorded as the clock offset.

[0094] Step 408: Correct the first motion trajectory and the second motion trajectory according to the clock deviation, and modify the target transformation matrix based on the corrected first motion trajectory and the corrected second motion trajectory.

[0095] In practical applications, after the first transformation matrix is ​​calculated in step 401 and the fourth transformation matrix is ​​calculated in step 403, steps 405 to 408 can be executed first. That is, the first motion trajectory and the second motion trajectory are obtained based on the first transformation matrix and the fourth transformation matrix. Then, the final clock deviation is calculated through each algorithm step. The target transformation matrix is ​​calculated by combining the clock deviation with the first transformation matrix, the second transformation matrix and the fourth transformation matrix.

[0096] This application also provides an extrinsic parameter calibration device for a multi-legged robot vision sensor. This device can be integrated into the extrinsic parameter calibration equipment for the multi-legged robot vision sensor and is used to execute the extrinsic parameter calibration method for the multi-legged robot vision sensor described in this application, possessing corresponding functions and beneficial effects. Specifically, Figure 5 A schematic diagram of the extrinsic parameter calibration device for the vision sensor of a multi-legged robot provided in this embodiment of the application. See also... Figure 5 The extrinsic parameter calibration device for the multi-legged robot vision sensor includes a first matrix acquisition module 501, a second matrix acquisition module 502, a third matrix acquisition module 503, and a fourth matrix acquisition module 504.

[0097] The system comprises the following modules: a first matrix acquisition module 501, used to construct the foot coordinate system of the multi-legged robot and acquire a first transformation matrix that transforms the foot coordinate system to the body coordinate system of the multi-legged robot; a second matrix acquisition module 502, used to acquire a second transformation matrix that transforms the body visual coordinate system corresponding to the body visual sensor to the world coordinate system and a third transformation matrix that transforms the foot visual coordinate system corresponding to the foot visual sensor to the world coordinate system, wherein the body visual sensor is mounted on the multi-legged robot and the foot visual sensor is mounted on the foot of the multi-legged robot, the world coordinate system is a coordinate system established with any point on the calibration plate as the origin, and the calibration plate is a flat plate placed in front of the multi-legged robot and simultaneously falling within the line of sight of the body visual sensor and the foot visual sensor; a third matrix acquisition module 503, used to calculate a fourth transformation matrix that transforms the body visual coordinate system to the foot visual coordinate system based on the second transformation matrix and the third transformation matrix; and a fourth matrix acquisition module 504, used to calculate a target transformation matrix that transforms the body coordinate system to the body camera coordinate system based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

[0098] In this embodiment, to achieve extrinsic parameter calibration, an extrinsic parameter calibration command is first applied to enter the extrinsic parameter calibration process. This extrinsic parameter calibration command can be actively input by the user. Based on this command, the entire system enters the extrinsic parameter calibration process, calculating the final target transformation matrix according to the procedure described in the extrinsic parameter calibration method for the multi-legged robot's vision sensor provided in this embodiment. That is, precise extrinsic parameter calibration is completed sequentially through the first matrix acquisition module 501, the second matrix acquisition module 502, the third matrix acquisition module 503, and the fourth matrix acquisition module 504. By defining a new foot coordinate system for the multi-legged robot and fixing foot vision sensors to the robot's feet, combined with high-precision relative state measurements between the foot vision coordinate system and the multi-legged robot's body coordinate system, the extrinsic parameter problem of the multi-legged robot is transformed into a robotic arm hand-eye calibration problem. During the extrinsic parameter calibration process, the multi-legged robot does not need to perform large-scale movements in the environment; only a calibration plate needs to be loaded in the environment, and the movement of a single leg of the multi-legged robot relative to the body coordinate system is required to complete the extrinsic parameter calibration. The operation is simple, and the extrinsic parameter calibration accuracy is high.

[0099] As a further preferred embodiment, the first matrix acquisition module 501 acquires a preset foot kinematics model of the multi-legged robot and the encoded data collected by the foot motor encoder of the multi-legged robot; it calculates a first transformation matrix from the foot coordinate system to the body coordinate system of the multi-legged robot based on the foot kinematics model and the encoded data; the second matrix acquisition module 502 aligns the timestamp of the body vision sensor with the timestamp of the foot vision sensor, acquires a first image of the calibration plate acquired by the body vision sensor at a target timestamp and a second image of the calibration plate acquired by the foot vision sensor at the target timestamp; it acquires the size parameters of the calibration plate; it acquires a second transformation matrix from the body vision coordinate system to the world coordinate system based on the size parameters and the first image; and it acquires a third transformation matrix from the foot vision coordinate system to the world coordinate system corresponding to the foot vision sensor based on the size parameters and the second image.

[0100] In the third matrix acquisition module 503, specifically according to the formula... Obtain the fourth transformation matrix from the body visual coordinate system to the foot visual coordinate system. Wherein, This is the second transformation matrix. This is the third transformation matrix. This is the fourth transformation matrix. This is the transformation matrix from the world coordinate system to the foot visual coordinate system.

[0101] Regarding how to obtain the target transformation matrix, specifically, the formula... By starting with a gradual transformation of the coordinate system, we can derive the following: Based on this formula Transform into According to the formula Formula Can be transformed into At this time in the formula Multiply both sides simultaneously and That is, multiplying both sides of the equation by . Also multiplied by Formula Formula Rewritten as the classic equation AX = XB, where Substituting the values ​​of A, B, and X from the above formula into AX = XB, and solving the equation using well-known methods, we can obtain the target transformation matrix, which is the motion attitude of the body coordinate system relative to the body camera coordinate system.l Using the foot coordinate system, Let be the target transformation matrix. This is the transformation matrix from the foot coordinate system to the foot visual coordinate system. b T l Let be the first transformation matrix, where i is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the i-th acquisition, and j is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the j-th acquisition. Let j be the j-th motion posture in the world coordinate system relative to the foot visual coordinate system. The i-th motion posture in the world coordinate system relative to the foot visual coordinate system. Let j be the j-th motion pose of the body's visual coordinate system relative to the world coordinate system. The i-th motion pose of the body's visual coordinate system relative to the world coordinate system. Let be the j-th motion posture of the foot coordinate system relative to the body coordinate system. Let i be the i-th motion posture of the foot coordinate system relative to the body coordinate system. The j-th motion posture of the body visual coordinate system relative to the foot visual coordinate system.

[0102] Based on the above embodiments, a fifth matrix acquisition module can be further provided to acquire the fifth transformation matrix for transforming the world coordinate system to the body visual coordinate system and the sixth transformation matrix for transforming the body coordinate system to the foot coordinate system. A sixth matrix acquisition module is also included, used to calculate the transformation matrix for transforming the foot coordinate system to the foot visual coordinate system based on the target transformation matrix, the third transformation matrix, the fifth transformation matrix, and the sixth transformation matrix.

[0103] The modules included in the extrinsic parameter calibration device for the multi-legged robot vision sensor proposed in this application are divided according to functional logic, but are not limited to the above division method, as long as the corresponding function can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0104] like Figure 6 As shown, this application embodiment also provides an extrinsic parameter calibration device, including: a memory 601 and one or more processors 602; the memory 601 is used to store one or more programs; when the one or more programs are executed by the one or more processors 602, the one or more processors implement the extrinsic parameter calibration method for the multi-legged robot vision sensor as described in this application.

[0105] The memory 601 and processor 602 are connected via a communication bus to enable communication between them. Taking an intelligent interactive flat panel as an example of this external parameter calibration device, the memory 601 and processor 602 may also include a wireless communication module and a user operation port. The user operation port provides users with input for external parameter calibration commands. The wireless communication module connects the processor 602 to the main control device in the multi-legged robot body, such as a controller or microcontroller, to receive collected data and complete the external parameter calibration calculations.

[0106] Processor 602 may include one or more processing cores. Processor 602 connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in processor 602, and by calling data stored in memory 601. Optionally, processor 602 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Arrays (PLA). Processor 402 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 602 and may be implemented as a separate chip.

[0107] The memory 601 may include RAM or read-only memory. Optionally, the memory 601 may include non-transitory computer-readable storage medium. The memory 601 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 601 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 601 may also be at least one storage device located remotely from the aforementioned processor 602.

[0108] The extrinsic calibration equipment provided in the above embodiments can be used to perform any of the extrinsic calibration methods for multi-legged robot vision sensors proposed in this application, and has corresponding functions and beneficial effects.

[0109] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute the extrinsic parameter calibration method for a multi-legged robot vision sensor as provided in the above embodiments. This extrinsic parameter calibration method for the multi-legged robot vision sensor includes: defining a foot coordinate system of the multi-legged robot; obtaining a first transformation matrix transforming the foot coordinate system to the body coordinate system of the multi-legged robot; obtaining a second transformation matrix transforming the body vision coordinate system corresponding to the body vision sensor to the world coordinate system, and a third transformation matrix transforming the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system, wherein the body vision sensor… The device is mounted on a multi-legged robot, and the foot vision sensor is mounted on the foot of the multi-legged robot. The world coordinate system is a coordinate system established with any point on the calibration plate as the origin. The calibration plate is a flat plate placed in front of the multi-legged robot and simultaneously falling within the line of sight of the body vision sensor and the foot vision sensor. A fourth transformation matrix is ​​calculated based on the second transformation matrix and the third transformation matrix to transform the body vision coordinate system to the foot vision coordinate system. A target transformation matrix is ​​calculated based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix to transform the body coordinate system to the body camera coordinate system.

[0110] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0111] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0113] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0115] Note that the above are merely preferred embodiments and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the protection scope of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, and the scope of this application is determined by the scope of the claims.

Claims

1. A extrinsic parameter calibration system for a vision sensor of a multi-legged robot, characterized in that, The system includes a multi-legged robot, a calibration plate, and a processor. The multi-legged robot has multiple legs and is equipped with a body vision sensor. Each leg is equipped with a foot vision sensor, and both the body vision sensor and the foot vision sensor are connected to the processor. The body vision sensor is used to acquire image information in front of the multi-legged robot, and the foot vision sensor is used to acquire image information in front of the foot. The processor is used to calculate the target transformation matrix from the body coordinate system to the body vision coordinate system based on the body vision coordinate system corresponding to the body vision sensor and the foot vision coordinate system corresponding to the foot vision sensor. The calibration plate is installed in front of the multi-legged robot, and the calibration plate falls within the line of sight of the body vision sensor and the foot vision sensor. Specifically, the processor is used to construct the foot coordinate system of the multi-legged robot, obtain a first transformation matrix that transforms the foot coordinate system to the body coordinate system of the multi-legged robot; obtain a second transformation matrix that transforms the body visual coordinate system corresponding to the body visual sensor to the world coordinate system, and a third transformation matrix that transforms the foot visual coordinate system corresponding to the foot visual sensor to the world coordinate system, wherein the world coordinate system is a coordinate system established with any point on the calibration board as the origin; calculate a fourth transformation matrix that transforms the body visual coordinate system to the foot visual coordinate system based on the second transformation matrix and the third transformation matrix; and calculate a target transformation matrix that transforms the body coordinate system to the body visual coordinate system based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

2. A method for calibrating the extrinsic parameters of a vision sensor for a multi-legged robot, characterized in that, include: Construct the foot coordinate system of the multi-legged robot, and obtain the first transformation matrix that transforms the foot coordinate system to the body coordinate system of the multi-legged robot; The second transformation matrix for transforming the body vision coordinate system corresponding to the body vision sensor to the world coordinate system and the third transformation matrix for transforming the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system are obtained, wherein the world coordinate system is a coordinate system established with any point on the calibration plate as the origin. A fourth transformation matrix is ​​calculated based on the second transformation matrix and the third transformation matrix to transform the body visual coordinate system to the foot visual coordinate system; The target transformation matrix for transforming the body coordinate system to the body visual coordinate system is calculated based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

3. The external parameter calibration method according to claim 2, characterized in that, Obtaining the first transformation matrix from the foot coordinate system to the body coordinate system of the multi-legged robot includes: Obtain the preset foot kinematics model of the multi-legged robot, and obtain the encoded data collected by the encoder of the foot motor of the multi-legged robot; The first transformation matrix is ​​calculated based on the foot kinematics model and the encoded data to transform the foot coordinate system to the body coordinate system of the multi-legged robot.

4. The external parameter calibration method according to claim 2, characterized in that, Obtain the second transformation matrix from the body vision coordinate system corresponding to the body vision sensor to the world coordinate system, and the third transformation matrix from the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system, including: Align the timestamp of the body vision sensor with the timestamp of the foot vision sensor to obtain the first image of the calibration plate acquired by the body vision sensor at the target timestamp and the second image of the calibration plate acquired by the foot vision sensor at the target timestamp. Obtain the dimensional parameters of the calibration plate; Based on the size parameters and the first image, a second transformation matrix is ​​obtained to transform the body visual coordinate system to the world coordinate system. Based on the size parameters and the second image, a third transformation matrix is ​​obtained to transform the foot visual coordinate system corresponding to the foot visual sensor to the world coordinate system.

5. The external parameter calibration method according to claim 2, characterized in that, A fourth transformation matrix is ​​calculated based on the second transformation matrix and the third transformation matrix to transform the body visual coordinate system to the foot visual coordinate system, specifically as follows: The fourth transformation matrix from the proprioceptive visual coordinate system to the foot visual coordinate system is calculated according to Formula 1. Formula 1 is as follows: ； in, This is the second transformation matrix. This is the third transformation matrix. This is the fourth transformation matrix. This is the transformation matrix from the world coordinate system to the foot visual coordinate system.

6. The external parameter calibration method according to claim 2, characterized in that, The target transformation matrix for transforming the body coordinate system to the body visual coordinate system is calculated based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix, including: Formula 3 is obtained by transforming Formula 2 and Formula 1, and the target transformation matrix is ​​calculated according to Formula 3. Formula 2 is: ； Formula 3: ； in, Let be the target transformation matrix. This is the transformation matrix from the foot coordinate system to the foot visual coordinate system. Let be the first transformation matrix, where i is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the i-th acquisition, and j is the image of the calibration plate acquired by the body vision sensor and the foot vision sensor in the j-th acquisition. Let be the j-th motion posture of the foot coordinate system relative to the body coordinate system. Let i be the i-th motion posture of the foot coordinate system relative to the body coordinate system. The j-th motion posture of the body visual coordinate system relative to the foot visual coordinate system. The i-th motion posture of the body visual coordinate system relative to the foot visual coordinate system.

7. The external parameter calibration method according to claim 6, characterized in that, Also includes: Obtain the fifth transformation matrix from the world coordinate system to the body visual coordinate system and the sixth transformation matrix from the body coordinate system to the foot coordinate system; The transformation matrix from the foot coordinate system to the foot visual coordinate system is calculated based on the target transformation matrix, the third transformation matrix, the fifth transformation matrix, and the sixth transformation matrix.

8. The external parameter calibration method according to claim 2, characterized in that, Also includes: Acquire the first motion trajectory of the foot of the multi-legged robot relative to the multi-legged robot body and the second motion trajectory of the foot vision sensor relative to the body vision sensor; Calculate the first average time interval between any two adjacent motion postures in the first motion trajectory and the second average time interval between any two adjacent motion postures in the second motion trajectory. Calculate the clock deviation between the first motion trajectory and the second motion trajectory based on the first average time interval and the second average time interval; The first motion trajectory and the second motion trajectory are corrected according to the clock deviation, and the target transformation matrix is ​​modified based on the corrected first motion trajectory and the corrected second motion trajectory.

9. The external parameter calibration method according to claim 8, characterized in that, Calculating the clock deviation between the first motion trajectory and the second motion trajectory based on the first average time interval and the second average time interval includes: The smaller of the first average time interval and the second average time interval is selected as the sampling time interval; At each sampling time interval, the corresponding motion posture is extracted from the first motion trajectory to form a first motion posture set, and at each sampling time interval, the corresponding motion posture is extracted from the second motion trajectory to form a second motion posture set; Calculate the first rotation amount of the first motion posture set and the second rotation amount of the second motion posture set; The modulus of the first angular velocity is calculated based on the first rotation amount, the modulus of the second angular velocity is calculated based on the second rotation amount, and the convolution between the modulus of the first angular velocity and the modulus of the second angular velocity is calculated. The position corresponding to the maximum response value in the convolution is recorded as the clock offset.

10. The external parameter calibration method according to claim 9, characterized in that, The first motion trajectory and the second motion trajectory are corrected according to the clock deviation, and the target transformation matrix is ​​modified based on the corrected first motion trajectory and the corrected second motion trajectory, specifically as follows: The first motion posture set and the second motion posture set are corrected according to the clock deviation, and the target transformation matrix is ​​modified based on the corrected first motion posture set and the corrected second motion posture set.

11. A extrinsic parameter calibration device for a vision sensor of a multi-legged robot, characterized in that, include: First matrix acquisition module: used to construct the foot coordinate system of the multi-legged robot and acquire the first transformation matrix of the foot coordinate system to the body coordinate system of the multi-legged robot; The second matrix acquisition module is used to acquire the second transformation matrix of the body vision coordinate system corresponding to the body vision sensor to the world coordinate system and the third transformation matrix of the foot vision coordinate system corresponding to the foot vision sensor to the world coordinate system. The world coordinate system is a coordinate system established with any point on the calibration board as the origin. The third matrix acquisition module is used to calculate a fourth transformation matrix based on the second transformation matrix and the third transformation matrix to transform the body visual coordinate system to the foot visual coordinate system. The fourth matrix acquisition module is used to calculate the target transformation matrix from the body coordinate system to the body visual coordinate system based on the first transformation matrix, the second transformation matrix, and the fourth transformation matrix.

12. An external parameter calibration device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the extrinsic parameter calibration method for the multi-legged robot vision sensor as described in any one of claims 2-10.

13. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the extrinsic parameter calibration method for the vision sensor of the multi-legged robot as described in any one of claims 2-10.

Images