A multi-modal sensor joint calibration method and related apparatus

Abstract

The application provides a multi-modal sensor joint calibration method and related device, and relates to the technical field of robot control. In the process of assisting the calibration robot to drive the target calibration board to move according to the expected motion trajectory, for each expected panning time period involved in the expected motion trajectory, the actual body pose of the to-be-calibrated robot is adaptively adjusted according to the corresponding expected calibration board pose, so that the multi-modal environment perception sensor deployed on the to-be-calibrated robot is aligned with the target calibration board for calibration board observation, so as to perform joint calibration of perception parameters based on the multi-modal calibration board observation data corresponding to each of the plurality of expected panning time periods, to determine the actual internal and external parameter data of each multi-modal environment perception sensor on the to-be-calibrated robot, so as to realize low-cost, high-efficiency and high-precision multi-sensor autonomous calibration effect for any to-be-calibrated robot through the cooperative cooperation between multiple robots.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a multimodal sensor joint calibration method and related apparatus. Background Technology

[0002] With the continuous development of science and technology, intelligent robot technology has received widespread attention from various industries (such as home services, manufacturing, emergency rescue, and other operational fields) due to its great research and application value. In order to improve the intelligent robot's ability to accurately perceive the external environment and realize functions such as autonomous navigation, autonomous obstacle avoidance, and target recognition and tracking, multimodal environmental perception sensors (which involve various single-modal perception sensors with different perception methods, including but not limited to image perception sensors (such as pinhole cameras, fisheye cameras, panoramic cameras, etc.) and point cloud perception sensors (such as depth cameras, lidar, millimeter-wave radar, etc.)) are often deployed on intelligent robots, and multi-sensor data fusion technology is used to achieve comprehensive perception of the robot's working environment.

[0003] In this process, the reliable implementation of multi-sensor data fusion technology relies on the accurate joint calibration of the intrinsic and extrinsic parameter data of each of the multimodal environmental perception sensors. However, it is worth noting that existing multi-sensor joint calibration schemes are inherently difficult to achieve an effective balance between calibration accuracy, calibration efficiency, and calibration cost, often resulting in huge calibration costs without guaranteeing calibration accuracy and efficiency. Therefore, how to achieve low-cost, high-efficiency, and high-accuracy multi-sensor joint calibration is a crucial technical problem that urgently needs to be solved when intelligent robots undergo factory-standard calibration or rework recalibration. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a multimodal sensor joint calibration method and system, an intelligent robot, and a readable storage medium. Through the collaborative cooperation among multiple robots, the method can effectively improve the consistency of calibration board observations (i.e., require that the calibration board pose observed by the multimodal sensors at the same time remains consistent) and the accuracy of joint calibration during the multimodal sensor joint calibration process of any robot to be calibrated. At the same time, it can reduce calibration costs (including but not limited to calibration time costs, calibration manpower costs, and calibration room setup and maintenance costs) and calibration errors, making it easy to achieve low-cost, high-efficiency, and high-accuracy multi-sensor autonomous calibration for any robot to be calibrated.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, this application provides a multimodal sensor joint calibration method, applied to a robot to be calibrated that is equipped with multimodal environmental perception sensors, the method comprising: The desired motion trajectory of the target calibration board under the action of the auxiliary calibration robot is obtained, wherein the desired motion trajectory includes the desired calibration board pose achieved by the target calibration board in multiple desired shooting time periods; For each desired shooting time period, the actual body pose of the robot to be calibrated is adaptively adjusted according to the corresponding desired calibration board pose, so that the multimodal environment perception sensor is aligned with the target calibration board to observe the calibration board and obtain the corresponding multimodal calibration board observation data. Based on the observation data of the multimodal calibration board corresponding to each of the multiple expected shooting time periods, the sensing parameters of the multimodal environmental perception sensors are jointly calibrated to obtain the actual internal and external parameter data of each multimodal environmental perception sensor on the robot to be calibrated.

[0006] In an optional implementation, for each desired shooting time period, the step of adaptively adjusting the actual body pose of the robot to be calibrated according to the corresponding desired calibration board pose, so that the multimodal environmental perception sensor is aligned with the target calibration board for calibration board observation, includes: Based on the theoretical overlapping perception field space of the multimodal environment perception sensor relative to the theoretical deployment pose of the robot to be calibrated, and the relative positional relationship between the robot to be calibrated and the auxiliary calibration robot, the theoretical body pose of the robot to be calibrated that is adapted to the pose of the desired calibration board is predicted; wherein, the target calibration board that maintains the pose of the desired calibration board is located within the theoretical overlapping perception field space of the robot to be calibrated when it achieves the theoretical body pose. Adjust the actual body pose of the robot to be calibrated according to the theoretical body pose, and detect whether the multimodal environmental perception sensor is simultaneously aligned with the target calibration board; If the multimodal environment perception sensors are detected to be simultaneously aligned with the target calibration board, then the multimodal environment perception sensors are controlled to simultaneously observe the target calibration board. Otherwise, based on the actual alignment status of each multimodal environment perception sensor with respect to the target calibration board, the actual body pose of the robot to be calibrated is adjusted, and the process jumps to the step of detecting whether the multimodal environment perception sensors are simultaneously aligned with the target calibration board to continue execution.

[0007] In an optional implementation, the multimodal environment sensing sensor is composed of multiple single-modal sensing sensors. The step of detecting whether the multimodal environment sensing sensors are simultaneously aligned with the target calibration plate includes: The system iterates through the multiple single-modal sensing sensors and, for each single-modal sensing sensor encountered, detects whether the target calibration board is within the current effective environmental sensing space of that single-modal sensing sensor. When it is detected that the target calibration board is within the effective environmental perception space of each of the plurality of single-modal sensing sensors, it is determined that the multimodal environmental perception sensors are simultaneously aligned with the target calibration board. When the target calibration plate is not detected within the effective environmental perception space of at least one single-modal sensing sensor, it is determined that the multi-modal environmental sensing sensor is not simultaneously aligned with the target calibration plate.

[0008] In an optional implementation, the multimodal environment perception sensor is composed of multiple single-modal perception sensors. The step of adjusting the actual body pose of the robot to be calibrated based on the actual alignment status of each of the multimodal environment perception sensors with respect to the target calibration board includes: Among the plurality of single-modal sensing sensors, a first sensing sensor is determined to be absent from the target calibration board within the corresponding effective environmental sensing space, and a second sensing sensor is determined to be present within the corresponding effective environmental sensing space. Based on the theoretical set poses of each of the plurality of single-modal sensing sensors on the robot to be calibrated, the theoretical pose offset between the theoretical set pose of the second sensing sensor and the theoretical set pose of the first sensing sensor is calculated. The theoretical pose offset is mapped onto the body motion coordinate system of the robot to be calibrated to obtain the desired body pose offset that matches the theoretical pose offset, and the actual body pose is processed according to the desired body pose offset.

[0009] In an optional implementation, the multimodal calibration board observation data for the same desired shooting time period includes calibration board observation data of multiple single-modal sensing sensors for that desired shooting time period. The step of jointly calibrating the sensing parameters of the multimodal environmental sensing sensors based on the multimodal calibration board observation data corresponding to each of the multiple desired shooting time periods to obtain the actual intrinsic and extrinsic parameter data of each multimodal environmental sensing sensor on the robot to be calibrated includes: For each desired time period for posing, the calibration board corner point detection is performed on the calibration board observation data of each of the multiple single-modal sensing sensors during the desired time period to obtain the calibration board corner point data of each of the multiple single-modal sensing sensors during the desired time period. Corner point registration is performed on the calibration board corner point data of each of the multiple single-modal sensing sensors during the desired posing time period to obtain the corner point registration results of the multiple single-modal sensing sensors during the desired posing time period. Based on the corner registration results corresponding to each of the multiple desired shooting time periods, the internal and external parameters of the multiple single-modal sensing sensors are jointly solved to obtain the actual internal and external parameter data of each of the multiple single-modal sensing sensors.

[0010] Secondly, this application provides a multimodal sensor joint calibration method, applied to an auxiliary calibration robot and at least one robot to be calibrated, both deployed in the same calibration environment, wherein each robot to be calibrated is equipped with a multimodal environment perception sensor, and the method includes: The auxiliary calibration robot acquires the actual environmental information of the calibration environment and plans a desired motion trajectory that meets the obstacle avoidance requirements for the target calibration board based on the actual environmental information; wherein, the desired motion trajectory includes the desired calibration board pose achieved by the target calibration board in multiple desired shooting time periods; The auxiliary calibration robot sends the desired motion trajectory to each robot to be calibrated, and drives the target calibration board to move according to the desired motion trajectory; Each robot to be calibrated adaptively adjusts its actual body posture according to the corresponding desired calibration board posture during each desired shooting time period involved in the desired motion trajectory, so that the deployed multimodal environmental perception sensor is aligned with the target calibration board to observe the calibration board and obtain the corresponding multimodal calibration board observation data. The robot to be calibrated performs joint calibration of the sensing parameters of the deployed multimodal environmental perception sensors based on the observation data of the multimodal calibration boards corresponding to the multiple expected shooting time periods, thereby obtaining the actual internal and external parameter data of each of the multimodal environmental perception sensors on the robot to be calibrated.

[0011] In an optional implementation, the step of any robot to be calibrated adaptively adjusting its actual body pose according to the pose of the corresponding target calibration board within a single desired posing time period, so that the deployed multimodal environmental perception sensor is aligned with the target calibration board for calibration board observation, includes: Based on the theoretical overlapping perception field space of the multimodal environment perception sensor relative to the theoretical deployment pose of the robot to be calibrated, and the relative positional relationship between the robot to be calibrated and the auxiliary calibration robot in the calibration environment, the theoretical body pose of the robot to be calibrated that is adapted to the pose of the desired calibration board is predicted; wherein, the target calibration board that maintains the pose of the desired calibration board is located within the theoretical overlapping perception field space of the robot to be calibrated when realizing the theoretical body pose. Adjust its actual body posture according to the theoretical body posture, and detect whether the multimodal environmental perception sensor is simultaneously aligned with the target calibration plate; If the multimodal environment sensing sensors are detected to be simultaneously aligned with the target calibration board, then the multimodal environment sensing sensors are controlled to simultaneously observe the target calibration board. Otherwise, based on the actual alignment status of each multimodal environment sensing sensor with respect to the target calibration board, the actual body posture is adjusted, and the process jumps to the step of detecting whether the multimodal environment sensing sensors are simultaneously aligned with the target calibration board to continue execution.

[0012] Thirdly, this application provides a multimodal sensor joint calibration system, the system including an auxiliary calibration robot deployed in the same calibration environment and at least one robot to be calibrated, wherein the at least one robot to be calibrated is communicatively connected to the auxiliary calibration robot, and each robot to be calibrated is deployed with a multimodal environment perception sensor. The at least one robot to be calibrated and the auxiliary calibration robot each store a computer program, and can collaboratively execute the multimodal sensor joint calibration method described in any of the foregoing embodiments by running the computer program.

[0013] Fourthly, this application provides an intelligent robot, including a multimodal environmental perception sensor, a processor, and a memory. The memory stores a computer program that can be executed by the processor. The processor can execute the computer program to cooperate with an auxiliary calibration robot that drives a target calibration board to realize the multimodal sensor joint calibration method described in any of the foregoing embodiments.

[0014] Fifthly, this application provides a readable storage medium storing a computer program thereon. When the computer program is executed by the robot to be calibrated, it cooperates with the auxiliary calibration robot that drives the target calibration board to realize the multimodal sensor joint calibration method described in any of the foregoing embodiments, wherein the robot to be calibrated is equipped with multimodal environmental perception sensors.

[0015] In this case, the beneficial effects of the embodiments of this application may include the following: After the robot to be calibrated acquires the desired motion trajectory of the target calibration board under the guidance of the auxiliary calibration robot, this application adaptively adjusts the actual body pose of the robot to be calibrated for each desired shooting time period involved in the desired motion trajectory, based on the corresponding desired calibration board pose. This allows the multimodal environmental perception sensors deployed on the robot to be calibrated to align with the target calibration board for observation, obtaining corresponding multimodal calibration board observation data. Then, based on the multimodal calibration board observation data corresponding to each of the multiple desired shooting time periods, the sensing parameters are jointly processed. Calibration is used to determine the actual intrinsic and extrinsic parameter data of each of the multimodal environmental perception sensors on the robot to be calibrated. In the process of joint calibration of multimodal sensors of any robot to be calibrated, the cooperation between multiple robots can effectively reduce human intervention and eliminate the need for building and maintaining a large calibration room. This can improve the consistency of observation on the calibration board of multimodal sensors and the accuracy of joint calibration, while reducing calibration costs (including but not limited to calibration time costs, calibration manpower costs, calibration room construction and maintenance costs, etc.) and calibration errors. In this way, low-cost, high-efficiency and high-accuracy multi-sensor autonomous calibration can be achieved for any robot to be calibrated.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the system composition of the multimodal sensor joint calibration system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the device composition of the intelligent robot provided in the embodiments of this application; Figure 3 A schematic flowchart illustrating the first multimodal sensor joint calibration method provided in this application embodiment; Figure 4 for Figure 3 A flowchart illustrating the sub-steps included in step S230; Figure 5 A schematic flowchart illustrating the second multimodal sensor joint calibration method provided in this application embodiment; Figure 6for Figure 5 The flowchart of the sub-steps included in step S320 is shown below.

[0019] Icons: 10 - Multimodal sensor joint calibration system; 100 - Intelligent robot; 200 - Auxiliary calibration robot; 110 - Memory; 120 - Processor; 130 - Communication unit; 140 - Multimodal environmental perception sensor. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] In the description of this application, it should be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are used only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the equipment or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0024] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0025] Furthermore, it is understood in the description of this application that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0026] Through diligent research, the applicant discovered that existing multi-sensor joint calibration solutions primarily rely on either manually holding calibration boards or constructing a separate large calibration chamber. The former requires a staff member to hold the calibration board in different poses, while each of the multimodal environmental perception sensors observes the board in the same pose. The sensor intrinsic and extrinsic parameters are then jointly calculated from the actual observation data of the multimodal environmental perception sensors under different calibration board poses to achieve the joint calibration effect. The latter method requires constructing a separate large calibration chamber. This chamber is surrounded by calibration boards of varying orientations and heights, with a rotatable support mechanism in the center. This mechanism drives the intelligent robot to be calibrated to rotate, allowing the multimodal environmental perception sensors to observe the calibration boards from multiple angles, thus achieving the joint calibration effect.

[0027] However, it is worth noting that the first method relies on a large amount of repetitive work by staff, resulting in a huge workload and extremely low calibration efficiency. It requires a very high investment of human resources. At the same time, because it is difficult for the human body to maintain a fixed posture for a long time, the calibration board often cannot remain completely still during the actual observation of the multimodal environmental perception sensor, which seriously affects the consistency of the calibration board observations among multiple sensors and is prone to serious calibration errors. The second method requires a large investment in advance to build and continuously maintain a large calibration room. It also requires a large investment in transportation costs to transport many intelligent robots to be calibrated to this large calibration room for calibration. Overall, it suffers from high calibration costs and low calibration efficiency.

[0028] To address this, the applicant has developed a multimodal sensor joint calibration method and system, an intelligent robot, and a readable storage medium. This method enables low-cost, high-efficiency, and high-precision autonomous multimodal sensor calibration by leveraging the collaborative cooperation among multiple robots during the multimodal sensor joint calibration process of any robot to be calibrated, thereby solving the technical problems existing in the aforementioned multimodal sensor joint calibration schemes.

[0029] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the system composition of the multimodal sensor joint calibration system 10 provided in this application embodiment. In this application embodiment, the multimodal sensor joint calibration system 10 can be deployed in any calibration environment and achieves low-cost, high-efficiency, and high-precision multi-sensor autonomous calibration for the robot to be calibrated (i.e., the intelligent robot 100 that needs to calibrate sensor intrinsic and extrinsic parameter data). The multimodal sensor joint calibration system 10 may include an auxiliary calibration robot 200 deployed in the same calibration environment and at least one robot to be calibrated (i.e., the intelligent robot 100 that needs to calibrate sensor intrinsic and extrinsic parameter data).

[0031] In this embodiment, each intelligent robot 100 is communicatively connected to the auxiliary calibration robot 200 to obtain the desired motion trajectory of the target calibration board (which may be, but is not limited to, a checkerboard calibration board, a dot calibration board, a QR code calibration board, etc.) under the action of the auxiliary calibration robot 200. Each intelligent robot 100 is equipped with a multimodal environmental perception sensor (which consists of multiple single-modal perception sensors, such as a panoramic camera, a depth camera, a lidar, etc.). During the process of the auxiliary calibration robot 200 moving the target calibration board according to the desired motion trajectory, the auxiliary calibration robot 200 can continuously and stably maintain the fixed calibration board pose. Combined with the means of adaptive adjustment of robot pose, the multimodal environmental perception sensor deployed by itself can adaptively follow and quickly observe the target calibration board that is stably maintained in different calibration board pose states. In order to achieve a low-cost, high-efficiency and high-precision multi-sensor autonomous calibration effect, the sensor intrinsic and extrinsic parameters can be jointly solved based on the actual observation data obtained by the corresponding multimodal environmental perception sensor under different calibration board pose states. The auxiliary calibration robot 200 can be a mobile robot (e.g., humanoid robot, tracked robot, wheeled robot, etc.) equipped with gripping tools such as grippers, dexterous hands, and robotic arms (used to hold the target calibration plate for adjusting and / or maintaining different postures); the intelligent robot 100, which is the robot to be calibrated, can be, but is not limited to, humanoid robots, sweeping robots, service robots, delivery robots, etc.

[0032] Alternatively, please refer to Figure 2 , Figure 2 This is a schematic diagram of the device composition of an intelligent robot 100 provided in an embodiment of this application. In this embodiment, the intelligent robot 100, which is to be calibrated, may include a memory 110, a processor 120, a communication unit 130, and a multimodal environmental perception sensor 140. The memory 110, processor 120, communication unit 130, and multimodal environmental perception sensor 140 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0033] In this embodiment, the memory 110 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 110 is used to store computer programs, and the processor 120 can execute the computer programs accordingly after receiving execution instructions.

[0034] In this embodiment, the processor 120 can be an integrated circuit chip with signal processing capabilities. The processor 120 can be a general-purpose processor, including at least one of a central processing unit (CPU), graphics processing unit (GPU), network processor (NP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0035] In this embodiment, the communication unit 130 is used to establish a communication connection between the intelligent robot 100 and other electronic devices via a network, and to send and receive data through the network, wherein the network includes wired communication networks and wireless communication networks. For example, the intelligent robot 100 can obtain the desired motion trajectory planned for the target calibration board from the auxiliary calibration robot 200 through the communication unit 130.

[0036] In this embodiment, the multimodal environmental perception sensor 140 is used to realize the multidimensional perception effect of the corresponding intelligent robot 100 on the external environment, which is achieved by multiple single-modal perception sensors working together.

[0037] In this embodiment, the auxiliary calibration robot 200 can pre-store a calibration board motion trajectory planning program related to the multimodal sensor joint calibration function, while the intelligent robot 100 can pre-store a specific computer program related to the multimodal sensor joint calibration function in the memory 110. By driving the processor 120 to execute the specific computer program, the robot follows the auxiliary calibration robot 200, which moves the target calibration board, to perform multi-sensor autonomous calibration operations. This facilitates the improvement of the calibration board observation consistency and joint calibration accuracy of the multimodal environmental perception sensors, while simultaneously reducing calibration costs and calibration errors, thereby achieving a low-cost, high-efficiency, and high-accuracy multi-sensor autonomous calibration effect.

[0038] Understandable Figure 2 The block diagram shown is only a schematic diagram of one composition of the intelligent robot 100. The intelligent robot 100 may also include components such as... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown. Figure 2 The components shown can be implemented using hardware, software, or a combination thereof.

[0039] In this application, to ensure that each intelligent robot 100 (which serves as the robot to be calibrated) in the multimodal sensor joint calibration system 10 follows the auxiliary calibration robot 200 that drives the target calibration plate to perform multimodal sensor autonomous calibration operations, so as to achieve a low-cost, high-efficiency and high-precision multimodal sensor autonomous calibration effect, this application embodiment provides a multimodal sensor joint calibration method applied to the multimodal sensor joint calibration system 10 to achieve the aforementioned objective. The multimodal sensor joint calibration method provided in this application will be described in detail below.

[0040] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the first multimodal sensor joint calibration method provided in this application embodiment. In this application embodiment, the first multimodal sensor joint calibration method is applied to the aforementioned multimodal sensor joint calibration system 10, and the first multimodal sensor joint calibration method may include steps S210 to S240.

[0041] Step S210: The calibration robot is assisted in acquiring the actual environmental information of the calibration environment, and based on the actual environmental information, it plans the desired motion trajectory that meets the motion obstacle avoidance requirements for the target calibration board.

[0042] In this embodiment, after the auxiliary calibration robot 200 is deployed in the calibration environment, it will call its own assembled calibrated environment perception sensor to perceive the environmental information of the calibration environment, so as to identify the various obstacles that actually exist in the calibration environment from the actual environmental information, and mark the identified obstacles in the environmental map constructed using the actual environmental information. Then, the auxiliary calibration robot 200 will, based on the calibration board size (including calibration board height, calibration board thickness, and calibration board width) of the target calibration board, and considering the obstacle avoidance requirements (i.e., the target calibration board will not collide with any obstacles in the calibration environment during the process of being moved) and the calibration board positioning requirements (i.e., the target calibration board needs to maintain its pose unchanged during multiple specific time periods during the process of being moved, so that the robot to be calibrated can observe the target calibration board within the corresponding specific time periods), plan the calibration board motion trajectory to obtain the desired motion trajectory. At this time, the desired motion trajectory will record the desired calibration board pose achieved by the target calibration board in multiple desired positioning time periods. The desired positioning time period is used to represent the specified time period reserved for each robot to be calibrated to observe the calibration board. The target calibration board maintains the same desired calibration board pose at all time points within a single desired positioning time period, and the desired calibration board poses corresponding to different desired positioning time periods are different from each other.

[0043] In step S220, the auxiliary calibration robot sends the desired motion trajectory to each robot to be calibrated, and drives the target calibration plate to move according to the desired motion trajectory.

[0044] In this embodiment, after the auxiliary calibration robot 200 completes the motion trajectory planning operation for the target calibration board within the calibration environment, the auxiliary calibration robot 200 uses its own gripping tool to grasp the target calibration board and moves the target calibration board according to the desired motion trajectory. Simultaneously, the auxiliary calibration robot 200 also synchronously sends the desired motion trajectory to each robot to be calibrated (i.e., the intelligent robot 100) within the calibration environment, enabling each robot to adaptively follow the auxiliary calibration robot 200 for pose adjustment and achieve highly consistent multi-sensor collaborative observation of the target calibration board within each desired shooting time period.

[0045] In step S230, each robot to be calibrated adaptively adjusts its actual body pose according to the pose of the corresponding target calibration board during each desired shooting time period involved in the desired motion trajectory, so that the deployed multimodal environmental perception sensor is aligned with the target calibration board to observe the calibration board and obtain the corresponding multimodal calibration board observation data.

[0046] In this embodiment, during the process of the auxiliary calibration robot 200 moving the target calibration board according to the desired motion trajectory, each robot to be calibrated will adaptively adjust its actual body pose (i.e., the actual pose of the sensor deployment body part on the robot to be calibrated (which is used to deploy the multimodal environmental perception sensor 140, such as the head of a humanoid robot) according to the corresponding desired calibration board pose during each desired shooting time period involved in the desired motion trajectory) so that the multimodal environmental perception sensor 140 on the robot to be calibrated can align with the target calibration board to perform calibration board observation during the desired shooting time period and obtain the corresponding multimodal calibration board observation data (wherein, the multimodal calibration board observation data of the same desired shooting time period includes the calibration board observation data of each of the multiple single-modal perception sensors in the multimodal environmental perception sensor 140 during the desired shooting time period). The auxiliary calibration robot 200 continuously and stably maintains the actual calibration board pose of the target calibration board in the corresponding desired calibration board pose state during each desired shooting time period. This ensures that the multiple single-modal sensing sensors deployed on each calibration robot can have good consistency in calibration board observation, effectively reducing human intervention, avoiding calibration errors caused by human factors, and eliminating the need for additional construction and continuous maintenance of large calibration rooms. As a result, it effectively reduces calibration costs and calibration errors, and improves the accuracy and overall calibration efficiency of multi-sensor joint calibration.

[0047] Alternatively, please refer to Figure 4 , Figure 4 yes Figure 3 The flowchart of step S230 includes the sub-steps. In this embodiment, for any robot to be calibrated, the step "adaptively adjust its actual body pose according to the corresponding desired calibration board pose, so that the deployed multimodal environmental perception sensors are aligned with the target calibration board for calibration board observation" during a single desired shooting time period may include sub-steps S231 to S235, to ensure that each robot to be calibrated can adaptively follow the auxiliary calibration robot 200 to adjust its robot pose, so that the multiple deployed single-modal perception sensors have good calibration board observation consistency during the observation of the target calibration board.

[0048] Sub-step S231: Based on the theoretical overlapping perception field space of the multimodal environment perception sensor relative to the theoretical deployment pose of the robot to be calibrated, and the relative positional relationship between the robot to be calibrated and the auxiliary calibration robot in the calibration environment, predict the theoretical body pose of the robot to be calibrated that is adapted to the pose of the desired calibration board.

[0049] In this embodiment, the theoretical overlapping perception field of view space is used to represent the overlapping field of view space between the overall environmental perception field of view space of all single-modal perception sensors of the robot to be calibrated after the multimodal environmental perception sensor 140 is deployed. At this time, for each single-modal perception sensor, the effective environmental perception space range of the single-modal perception sensor is essentially used to represent the actual spatial distribution range of the theoretical overlapping perception field of view space at the corresponding overall environmental perception field of view space. The theoretical deployment pose is used to describe the relative pose distribution of the theoretical overlapping perception field of view space with respect to the origin of the body motion coordinate system of the robot to be calibrated after the multimodal environmental perception sensor 140 is deployed. The body motion coordinate system is used to describe the local coordinate system of the sensor deployment body part of the robot to be calibrated. For any desired calibration plate pose, when a robot to be calibrated adjusts its actual body pose to a theoretical body pose that matches the desired calibration plate pose, the target calibration plate is often located within the theoretical overlapping perception field of view space of the robot to be calibrated. Therefore, each robot to be calibrated within the calibration environment can, based on spatial geometry reverse calculation technology, solve for the appropriate theoretical body pose for any desired calibration board pose.

[0050] Sub-step S232: Adjust the actual body posture according to the theoretical body posture.

[0051] Sub-step S233: Detect whether the multimodal environmental perception sensors are simultaneously aligned with the target calibration board.

[0052] In this embodiment, for any robot to be calibrated, the step of "detecting whether the multimodal environment perception sensor 140 on the robot to be calibrated is simultaneously aligned with the target calibration board" may include sub-steps A to C, so as to effectively determine whether the corresponding multimodal environment perception sensor 140 can achieve a multi-sensor collaborative observation action with strong observation consistency for the target calibration board.

[0053] Sub-step A: Traverse the multiple single-modal sensing sensors included in the multimodal environmental sensing sensor 140, and for each traversed single-modal sensing sensor, detect whether the target calibration board is within the current effective environmental sensing space range of that single-modal sensing sensor.

[0054] Specifically, for each single-modal sensing sensor, the presence of feature data (e.g., calibration board image features, calibration board point cloud features) matching the target calibration board can be determined by detecting whether the sensor sensing data (e.g., two-dimensional images captured by an image sensing sensor, three-dimensional point cloud data collected by a point cloud sensing sensor) within the corresponding effective environmental sensing space range exists. If the sensor sensing data contains feature data matching the target calibration board, it indicates that the target calibration board is within the effective environmental sensing space range of the single-modal sensing sensor; otherwise, it indicates that the target calibration board is not within the effective environmental sensing space range of the single-modal sensing sensor.

[0055] Sub-step B: When it is detected that the target calibration board is within the effective environmental perception space of each of the multiple single-modal sensing sensors, it is determined that the multimodal environmental sensing sensor 140 is simultaneously aligned with the target calibration board.

[0056] Sub-step C: When the target calibration plate is not detected within the effective environmental perception space of at least one single-modal sensing sensor, it is determined that the multi-modal environmental sensing sensor 140 is not simultaneously aligned with the target calibration plate.

[0057] In this embodiment, when a robot to be calibrated achieves its theoretical body pose within a single desired shooting time period, and the multimodal environment perception sensor 140 deployed on the robot is simultaneously aligned with the target calibration plate, it indicates that the multimodal environment perception sensor 140 can achieve a multi-sensor collaborative observation action with strong observation consistency within the desired shooting time period, and the robot to be calibrated will execute sub-step S234 accordingly; however, when a robot to be calibrated achieves its theoretical body pose within a single desired shooting time period, but the multimodal environment perception sensor 140 is not simultaneously aligned with the target calibration plate, it indicates that the theoretical body pose cannot yet achieve a multi-sensor collaborative observation action with strong observation consistency, and the robot to be calibrated will execute sub-step S235 accordingly to readjust the actual body pose.

[0058] Sub-step S234: Control the multi-modal environmental perception sensor to simultaneously observe the target calibration board.

[0059] Sub-step S235: Adjust the actual body posture of the multimodal environmental perception sensors according to the actual alignment status of each sensor with the target calibration plate.

[0060] In this embodiment, when it is determined that the multimodal environment perception sensor 140 of a robot to be calibrated is not simultaneously aligned with the target calibration plate within a single desired shooting time period, the first perception sensor that does not contain the target calibration plate within the corresponding effective environment perception space and the second perception sensor that does contain the target calibration plate within the corresponding effective environment perception space can be determined from among the multiple single-modal perception sensors included in the multimodal environment perception sensor 140. Then, based on the theoretical pose of each of the multiple single-modal perception sensors on the robot to be calibrated (which can be described using the robot's base coordinate system), the theoretical pose of the second perception sensor is calculated. The theoretical pose offset between the set pose and the theoretical set pose of the first sensing sensor (which characterizes the overall pose offset of the second sensing sensor and the first sensing sensor under ideal conditions) is set. Then, the theoretical pose offset is mapped from the robot base coordinate system of the robot to be calibrated to the body motion coordinate system of the robot to be calibrated to obtain the desired body pose offset that matches the theoretical pose offset. At this time, the actual body pose of the robot to be calibrated can be processed according to the desired body pose offset to make the actual overlapping sensing field space between the multiple single-modal sensing sensors evolve towards covering the target calibration plate. On this basis, the robot to be calibrated will jump back to the above sub-step S233 to continue execution until a multi-sensor cooperative observation action with strong observation consistency is achieved within the desired shooting time period.

[0061] Therefore, by executing the above sub-steps S231 to S235, this application ensures that each robot to be calibrated can adaptively follow the auxiliary calibration robot 200 to adjust its robot pose, so that the multiple deployed single-modal sensing sensors have good consistency in observing the target calibration board during the observation process.

[0062] In step S240, the robot to be calibrated performs joint calibration of the perception parameters of the deployed multimodal environmental perception sensors based on the observation data of the multimodal calibration board corresponding to each of the multiple expected shooting time periods, thereby obtaining the actual internal and external parameter data of each multimodal environmental perception sensor on the robot to be calibrated.

[0063] In this embodiment, when a robot to be calibrated uses its own multimodal environmental perception sensor 140 to perform multi-sensor collaborative observation actions on the target calibration board under different desired calibration board pose states through robot pose adaptive adjustment, the robot to be calibrated can use a traditional multi-sensor joint calibration algorithm to perform joint calibration of perception parameters based on the observation data of the multimodal calibration board corresponding to each of the multiple desired shooting time periods, so as to obtain the actual internal and external parameter data of each of the multiple single-modal perception sensors involved in the multimodal environmental perception sensor 140 on the robot to be calibrated, thereby achieving a low-cost, high-efficiency and high-precision multi-sensor autonomous calibration effect.

[0064] Optionally, in one embodiment of this example, for any robot to be calibrated, the step "to perform joint calibration of the sensing parameters of the multimodal environment perception sensor 140 based on the observation data of the multimodal calibration board corresponding to each of the multiple expected shooting time periods, and to obtain the actual internal and external parameter data of each of the multimodal environment perception sensors on the robot to be calibrated" may include sub-steps D to F.

[0065] Sub-step D: For each desired posed shooting time period, perform calibration board corner detection on the calibration board observation data of each of the multiple single-modal sensing sensors included in the multimodal environmental sensing sensor 140 during the desired posed shooting time period, and obtain the calibration board corner data of each of the multiple single-modal sensing sensors during the desired posed shooting time period.

[0066] In this embodiment, for the image sensing sensor, the calibration board observation data within a single desired shooting time period is a two-dimensional observation image. In this case, the corner data of the calibration board of the image sensing sensor during the desired shooting time period can be determined directly by extracting image feature points from the two-dimensional observation image. For the point cloud sensing sensor, the calibration board observation data within a single desired shooting time period is a three-dimensional observation point cloud. In this case, the three-dimensional observation point cloud is converted into a two-dimensional normalized grayscale image based on the point cloud intensity, and the corner data of the calibration board of the point cloud sensing sensor during the desired shooting time period is determined by extracting image feature points from the two-dimensional normalized grayscale image.

[0067] Sub-step E: Perform corner registration on the calibration board corner data of each of the multiple single-modal sensing sensors during the desired posing time period to obtain the corner registration results of the multiple single-modal sensing sensors during the desired posing time period.

[0068] In this embodiment, for each desired time period, the corresponding corner registration result records the calibration board corner attribution status between the calibration board observation data of the multiple single-modal sensing sensors (for example, which image pixel in the two-dimensional observation image of the image sensing sensor and which discrete point in the three-dimensional observation point cloud of the point cloud sensing sensor belong to the same calibration board corner).

[0069] Sub-step F: Based on the corner point registration results corresponding to each of the multiple expected shooting time periods, perform joint internal and external parameter solving on the multiple single-modal sensing sensors to obtain the actual internal and external parameter data of each of the multiple single-modal sensing sensors.

[0070] Therefore, by executing the above sub-steps D to F, this application can make full use of the multimodal calibration board observation data of the same multimodal environment perception sensor 140 under different calibration board pose states to achieve a high-precision sensor intrinsic and extrinsic parameter solution effect.

[0071] This application can ensure that each intelligent robot 100 (as the robot to be calibrated) in the multimodal sensor joint calibration system 10 follows the auxiliary calibration robot 200 that drives the target calibration board to move to perform multi-sensor autonomous calibration operations by executing the above steps S210 to S240. This facilitates the improvement of the consistency of calibration board observation and joint calibration accuracy of multimodal environmental perception sensors, and simultaneously reduces calibration costs and calibration errors, so as to achieve a low-cost, high-efficiency and high-accuracy multi-sensor autonomous calibration effect.

[0072] In this application, to ensure that each intelligent robot 100 can follow the auxiliary calibration robot 200 to perform multi-sensor autonomous calibration operations when acting as a robot to be calibrated, so as to achieve a low-cost, high-efficiency and high-precision multi-sensor autonomous calibration effect, this application embodiment provides a multi-modal sensor joint calibration method applicable to any robot to be calibrated to achieve the aforementioned objective. The multi-modal sensor joint calibration method provided in this application will be described in detail below.

[0073] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating the second multimodal sensor joint calibration method provided in this application embodiment. In this application embodiment, the second multimodal sensor joint calibration method is applied to any intelligent robot 100 that serves as the robot to be calibrated, and the second multimodal sensor joint calibration method may include steps S310 to S330.

[0074] Step S310: Obtain the expected motion trajectory of the target calibration board under the action of the auxiliary calibration robot, wherein the expected motion trajectory includes the expected pose of the target calibration board achieved by the target calibration board in multiple expected shooting time periods.

[0075] In this embodiment, the specific planning process of the desired motion trajectory can be referred to the detailed description of steps S210 and S220 above, and will not be repeated here.

[0076] Step S320: For each desired shooting time period, the actual body pose of the robot to be calibrated is adaptively adjusted according to the corresponding desired calibration board pose, so that the multimodal environment perception sensor is aligned with the target calibration board to observe the calibration board and obtain the corresponding multimodal calibration board observation data.

[0077] In this embodiment, during the process of the auxiliary calibration robot 200 moving the target calibration board according to the desired motion trajectory, each robot to be calibrated will adaptively adjust its actual body posture according to the corresponding desired calibration board posture during each desired shooting time period involved in the desired motion trajectory. This allows the multimodal environmental perception sensor 140 on the robot to be calibrated to align with the target calibration board during the desired shooting time period and observe the calibration board, thereby obtaining the corresponding multimodal calibration board observation data (wherein, the multimodal calibration board observation data for the same desired shooting time period includes the calibration board observation data of each of the multiple single-modal perception sensors in the multimodal environmental perception sensor 140 during the desired shooting time period).

[0078] Alternatively, please refer to Figure 6 , Figure 6 yes Figure 5 The flowchart of step S320 includes the sub-steps. In this embodiment, step S320 may include sub-steps S321 to S325 to ensure that the corresponding robot to be calibrated can adaptively follow the auxiliary calibration robot 200 to adjust its robot pose, so that the multiple deployed single-modal sensing sensors have good consistency in observing the target calibration board during the observation process.

[0079] Sub-step S321: Based on the theoretical overlapping perception field space of the multimodal environmental perception sensor relative to the theoretical deployment pose of the robot to be calibrated, and the relative positional relationship between the robot to be calibrated and the auxiliary calibration robot, predict the theoretical body pose of the robot to be calibrated that is adapted to the pose of the desired calibration board.

[0080] In this embodiment, the target calibration board maintaining the desired calibration board pose is located within the theoretically overlapping perceptual field of view space of the robot to be calibrated when achieving the theoretical body pose. The specific execution steps of sub-step S321 can be referred to the detailed description of sub-step S231 above, and will not be repeated here.

[0081] Sub-step S322: Adjust the actual body pose of the robot to be calibrated according to the theoretical body pose.

[0082] Sub-step S323: Detect whether the multimodal environmental perception sensors are simultaneously aligned with the target calibration board.

[0083] In this embodiment, the specific execution steps of sub-step S323 (i.e., sub-steps A to C above) can be referred to the detailed description of sub-step S231 above, and will not be repeated here.

[0084] Specifically, when the robot to be calibrated achieves the theoretical body pose within a single desired shooting time period, and the multimodal environment perception sensor 140 deployed on the robot is simultaneously aligned with the target calibration plate, it indicates that the multimodal environment perception sensor 140 can achieve a multi-sensor collaborative observation action with strong observation consistency within the desired shooting time period, and the robot to be calibrated will execute sub-step S324 accordingly; however, when the robot to be calibrated achieves the theoretical body pose within a single desired shooting time period, but the multimodal environment perception sensor 140 is not simultaneously aligned with the target calibration plate, it indicates that the theoretical body pose cannot yet achieve a multi-sensor collaborative observation action with strong observation consistency, and the robot to be calibrated will execute sub-step S325 accordingly to readjust the actual body pose.

[0085] Sub-step S324: Control the multi-modal environmental perception sensor to simultaneously observe the target calibration board.

[0086] Sub-step S325: Adjust the actual body pose of the robot to be calibrated based on the actual alignment status of each of the multimodal environmental perception sensors with the target calibration board.

[0087] In this embodiment, the specific execution steps of sub-step S325 can be referred to the detailed description of sub-step S235 above, and will not be repeated here.

[0088] Therefore, by executing the above sub-steps S321 to S325, this application ensures that the robot to be calibrated can adaptively follow the auxiliary calibration robot 200 to adjust its robot pose, so that the multiple deployed single-modal sensing sensors have good consistency in observing the target calibration board during the observation process.

[0089] Step S330: Based on the observation data of the multimodal calibration board corresponding to each of the multiple expected shooting time periods, perform joint calibration of the sensing parameters of the multimodal environmental perception sensors to obtain the actual internal and external parameter data of each multimodal environmental perception sensor on the robot to be calibrated.

[0090] In this embodiment, the specific execution steps of step S330 (which involve the above-mentioned sub-steps D to F) can be referred to the detailed description of sub-step S240 above, and will not be repeated here.

[0091] Therefore, by executing the above steps S310 to S330, this application can ensure that each intelligent robot 100 can follow the auxiliary calibration robot 200 to perform multi-sensor autonomous calibration operations when acting as a robot to be calibrated. This facilitates the improvement of the consistency of the calibration board observations and the accuracy of joint calibration of the multimodal environmental perception sensors, while simultaneously reducing calibration costs and calibration errors, thereby achieving a low-cost, high-efficiency, and high-accuracy multi-sensor autonomous calibration effect.

[0092] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0093] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the various functions provided in this application are implemented as software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, including several instructions to enable the intelligent robot 100, which is deployed with multimodal environmental perception sensors, to cooperate with the auxiliary calibration robot 200 that drives the target calibration board to execute all or part of the steps of the methods described in the various embodiments of this application when acting as a calibration robot to be calibrated. The aforementioned readable storage medium includes: USB flash drives, mobile hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0094] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for joint calibration of multimodal sensors, characterized in that, The method, applied to a robot to be calibrated and equipped with multimodal environmental perception sensors, includes: The desired motion trajectory of the target calibration board under the action of the auxiliary calibration robot is obtained, wherein the desired motion trajectory includes the desired calibration board pose achieved by the target calibration board in multiple desired shooting time periods; For each desired shooting time period, the actual body pose of the robot to be calibrated is adaptively adjusted according to the corresponding desired calibration board pose, so that the multimodal environment perception sensor is aligned with the target calibration board to observe the calibration board and obtain the corresponding multimodal calibration board observation data. The multimodal calibration board observation data for the same desired shooting time period includes the calibration board observation data of multiple single-modal perception sensors for that desired shooting time period. Based on the observation data of the multimodal calibration board corresponding to each of the multiple expected shooting time periods, the sensing parameters of the multimodal environmental perception sensors are jointly calibrated to obtain the actual intrinsic and extrinsic parameter data of each multimodal environmental perception sensor on the robot to be calibrated. Specifically, for each expected shooting time period, the calibration board corner point detection is performed on the calibration board observation data of each of the multiple single-modal perception sensors during that expected shooting time period to obtain the calibration board corner point data of each of the multiple single-modal perception sensors during that expected shooting time period. Corner point registration is then performed on the calibration board corner point data of each of the multiple single-modal perception sensors during that expected shooting time period to obtain the corner point registration results of the multiple single-modal perception sensors during that expected shooting time period. Finally, based on the corner point registration results corresponding to each of the multiple expected shooting time periods, the intrinsic and extrinsic parameters of the multiple single-modal perception sensors are jointly solved to obtain the actual intrinsic and extrinsic parameter data of each of the multiple single-modal perception sensors.

2. The method according to claim 1, characterized in that, For each desired shooting time period, the step of adaptively adjusting the actual body pose of the robot to be calibrated according to the corresponding desired calibration board pose, so that the multimodal environment perception sensor is aligned with the target calibration board for calibration board observation, includes: Based on the theoretical overlapping perception field space of the multimodal environment perception sensor relative to the theoretical deployment pose of the robot to be calibrated, and the relative positional relationship between the robot to be calibrated and the auxiliary calibration robot, the theoretical body pose of the robot to be calibrated that is adapted to the pose of the desired calibration board is predicted; wherein, the target calibration board that maintains the pose of the desired calibration board is located within the theoretical overlapping perception field space of the robot to be calibrated when it achieves the theoretical body pose. Adjust the actual body pose of the robot to be calibrated according to the theoretical body pose, and detect whether the multimodal environmental perception sensor is simultaneously aligned with the target calibration board; If the multimodal environment perception sensors are detected to be simultaneously aligned with the target calibration board, then the multimodal environment perception sensors are controlled to simultaneously observe the target calibration board. Otherwise, based on the actual alignment status of each multimodal environment perception sensor with respect to the target calibration board, the actual body pose of the robot to be calibrated is adjusted, and the process jumps to the step of detecting whether the multimodal environment perception sensors are simultaneously aligned with the target calibration board to continue execution.

3. The method according to claim 2, characterized in that, The multimodal environment sensing sensor is composed of multiple single-modal sensing sensors. The step of detecting whether the multimodal environment sensing sensors are simultaneously aligned with the target calibration plate includes: The system iterates through the multiple single-modal sensing sensors and, for each single-modal sensing sensor encountered, detects whether the target calibration board is within the current effective environmental sensing space range of that single-modal sensing sensor. When it is detected that the target calibration board is within the effective environmental perception space of each of the plurality of single-modal sensing sensors, it is determined that the multimodal environmental perception sensors are simultaneously aligned with the target calibration board. When the target calibration plate is not detected within the effective environmental perception space of at least one single-modal sensing sensor, it is determined that the multi-modal environmental sensing sensor is not simultaneously aligned with the target calibration plate.

4. The method according to claim 2, characterized in that, The multimodal environment perception sensor is composed of multiple single-modal perception sensors. The step of adjusting the actual body pose of the robot to be calibrated based on the actual alignment status of each of the multimodal environment perception sensors with respect to the target calibration board includes: Among the plurality of single-modal sensing sensors, a first sensing sensor is determined to be absent from the target calibration board within the corresponding effective environmental sensing space, and a second sensing sensor is determined to be present within the corresponding effective environmental sensing space. Based on the theoretical set poses of each of the plurality of single-modal sensing sensors on the robot to be calibrated, the theoretical pose offset between the theoretical set pose of the second sensing sensor and the theoretical set pose of the first sensing sensor is calculated. The theoretical pose offset is mapped onto the body motion coordinate system of the robot to be calibrated to obtain the desired body pose offset that matches the theoretical pose offset, and the actual body pose is processed according to the desired body pose offset.

5. A method for joint calibration of multimodal sensors, characterized in that, The method, applicable to an auxiliary calibration robot and at least one robot to be calibrated, deployed in the same calibration environment, wherein each robot to be calibrated is equipped with a multimodal environment perception sensor, comprises: The auxiliary calibration robot acquires the actual environmental information of the calibration environment and plans a desired motion trajectory that meets the obstacle avoidance requirements for the target calibration board based on the actual environmental information; wherein, the desired motion trajectory includes the desired calibration board pose achieved by the target calibration board in multiple desired shooting time periods; The auxiliary calibration robot sends the desired motion trajectory to each robot to be calibrated, and drives the target calibration board to move according to the desired motion trajectory; Each robot to be calibrated adaptively adjusts its actual body posture according to the corresponding desired calibration board posture during each desired shooting time period involved in the desired motion trajectory, so that the deployed multimodal environmental perception sensors are aligned with the target calibration board to observe the calibration board and obtain the corresponding multimodal calibration board observation data. The multimodal calibration board observation data for the same desired shooting time period includes the calibration board observation data of multiple single-modal perception sensors in the same desired shooting time period. The robot to be calibrated performs joint calibration of the sensing parameters of the deployed multimodal environmental perception sensors based on the observation data of the multimodal calibration board corresponding to each of the multiple expected shooting time periods, thereby obtaining the actual intrinsic and extrinsic parameter data of each multimodal environmental perception sensor on the robot to be calibrated. Specifically, for each expected shooting time period, the corner points of the calibration board are detected for the observation data of the multiple single-modal perception sensors during the expected shooting time period, thereby obtaining the corner point data of the multiple single-modal perception sensors during the expected shooting time period. Then, corner point registration is performed on the corner point data of the multiple single-modal perception sensors during the expected shooting time period, thereby obtaining the corner point registration results of the multiple single-modal perception sensors during the expected shooting time period. Finally, based on the corner point registration results corresponding to each of the multiple expected shooting time periods, the intrinsic and extrinsic parameters of the multiple single-modal perception sensors are jointly solved to obtain the actual intrinsic and extrinsic parameter data of each of the multiple single-modal perception sensors.

6. The method according to claim 5, characterized in that, The steps for any robot to be calibrated to adaptively adjust its actual body pose according to the pose of the corresponding target calibration board within a single desired posing time period, so that the deployed multimodal environmental perception sensors are aligned with the target calibration board for calibration board observation, include: Based on the theoretical overlapping perception field space of the multimodal environment perception sensor relative to the theoretical deployment pose of the robot to be calibrated, and the relative positional relationship between the robot to be calibrated and the auxiliary calibration robot in the calibration environment, the theoretical body pose of the robot to be calibrated that is adapted to the pose of the desired calibration board is predicted; wherein, the target calibration board that maintains the pose of the desired calibration board is located within the theoretical overlapping perception field space of the robot to be calibrated when realizing the theoretical body pose; Adjust its actual body posture according to the theoretical body posture, and detect whether the multimodal environmental perception sensor is simultaneously aligned with the target calibration plate; If the multimodal environment sensing sensors are detected to be simultaneously aligned with the target calibration board, then the multimodal environment sensing sensors are controlled to simultaneously observe the target calibration board. Otherwise, based on the actual alignment status of each multimodal environment sensing sensor with respect to the target calibration board, the actual body posture is adjusted, and the process jumps to the step of detecting whether the multimodal environment sensing sensors are simultaneously aligned with the target calibration board to continue execution.

7. A multimodal sensor joint calibration system, characterized in that, The system includes an auxiliary calibration robot deployed in the same calibration environment and at least one robot to be calibrated, wherein each of the at least one robot to be calibrated is communicatively connected to the auxiliary calibration robot, and each robot to be calibrated is equipped with a multimodal environmental perception sensor. The at least one robot to be calibrated and the auxiliary calibration robot each store a computer program, and can collaboratively execute the multimodal sensor joint calibration method according to any one of claims 1-6 by running the computer program.

8. An intelligent robot, characterized in that, The method includes a multimodal environmental perception sensor, a processor, and a memory. The memory stores a computer program that can be executed by the processor. The processor can execute the computer program to cooperate with an auxiliary calibration robot that drives a target calibration board to implement the multimodal sensor joint calibration method according to any one of claims 1-6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the robot to be calibrated, it works in conjunction with the auxiliary calibration robot that drives the target calibration board to implement the multimodal sensor joint calibration method according to any one of claims 1-6, wherein the robot to be calibrated is equipped with multimodal environmental perception sensors.

Images