Mobile robot external parameter calibration method and system based on multi-vision technology and medium

Abstract

The invention provides a mobile robot external parameter calibration method and system based on a multi-vision technology and a medium, and the method comprises the steps: setting a plurality of collection positions in a ventilation pipeline, and controlling a mobile robot to sequentially move to each collection position in the ventilation pipeline based on the collection positions; acquiring acquisition data of each acquisition position based on a laser scanner and a stereoscopic vision system, splicing the acquisition data of each acquisition position into a TVS point cloud and a stereoscopic vision point cloud, and performing plane fitting to generate orthogonal vectors of three planes; analyzing orthogonal vectors of the three planes based on a least square method and singular values, and calculating normal vectors of the three planes; the stereoscopic vision point cloud is rotated or / and translated according to the rotation matrix, so that the stereoscopic vision point cloud is aligned with the TVS point cloud, and a calibration result is obtained; the laser scanner and the stereoscopic vision system are combined, and the plane surface of the ventilation pipeline is used as a calibration object, so that high-precision external parameter calibration is realized.

Description

Technical Field

[0001] The present application relates to the technical field of robot parameter calibration. Specifically, it relates to an external parameter calibration method, system and medium for a mobile robot based on multi-vision technology. Background Art

[0002] Mobile robots play an important role in industrial, logistics and service applications, especially in navigation tasks in narrow and dark environments such as infrastructure inspection. Existing vision systems usually rely on a single sensor, such as a laser scanner or a stereo vision system. However, due to factors such as environmental light changes and occlusion, the accuracy and robustness of a single sensor are limited. To improve the perception ability of mobile robots in complex environments, it is usually necessary to fuse the data of multiple vision systems. However, the data fusion of multiple vision systems requires accurate external parameter calibration, that is, to determine the relative positions and postures between different sensors. Existing external parameter calibration methods usually rely on calibration targets or are greatly affected by environmental light, making it difficult to perform real-time calibration in dark or complex environments. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide an external parameter calibration method, system and medium for a mobile robot based on multi-vision technology. By combining a laser scanner and a stereo vision system and using the planar surface of a ventilation duct as a calibration object, it can perform real-time external parameter calibration in dark or complex environments without relying on a calibration target, achieving high-precision external parameter calibration.

[0004] The embodiments of the present application also provide an external parameter calibration method for a mobile robot based on multi-vision technology, including: Set multiple acquisition positions in the ventilation duct, and control the mobile robot to move sequentially to each acquisition position in the ventilation duct based on the acquisition positions; the mobile robot needs to move sequentially along a preset route and stay at multiple specific "acquisition position points", that is, move one by one to "each acquisition position".

[0005] Obtain the acquisition data of each acquisition position based on the laser scanner and the stereo vision system, and splice the acquisition data of each acquisition position into a TVS point cloud (point cloud data generated by the laser scanner) and a stereo vision point cloud; the TVS point cloud represents the point cloud data generated by the laser scanner (or three-dimensional vision by scanning). The acquisition positions refer to a number of spatial points preset inside the ventilation duct for data acquisition. Each acquisition position provides a data acquisition site for the laser scanner and the stereo vision system, facilitating multi-view scanning and calibration calculation of the duct surface. These acquisition points are generally set near key geometric structures such as elbows, branches, and diameter changes of the duct to obtain high-quality plane fitting information.

[0006] Perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes. Based on the least squares method and singular value analysis of the orthogonal vectors of the three planes, calculate the normal vectors of the three planes. Calculate the rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result.

[0007] Optionally, in the external parameter calibration method of the mobile robot based on multi-vision technology described in the embodiments of the present application, multiple acquisition positions are set inside the ventilation duct, and the mobile robot is controlled to move sequentially to each acquisition position inside the ventilation duct based on the acquisition positions, specifically including: Analyze the distribution information of the ventilation duct based on the building drawings, and the distribution information includes the direction, branch situation, and pipe diameter size of the ventilation duct. Analyze the elbow, branch node, diameter change position of the pipe, and the area with abnormal air flow or pollutant accumulation based on the distribution information of the ventilation duct. Set multiple acquisition positions according to the elbow, branch node, diameter change position of the pipe, and the area with abnormal air flow or pollutant accumulation. Set the robot movement route based on multiple acquisition positions, and set the robot movement parameters, where the robot movement parameters include the robot movement speed, robot steering angle, robot acceleration, and robot deceleration. Control the robot to move sequentially to each acquisition position along the movement route according to the robot movement parameters.

[0008] Optionally, in the external parameter calibration method of the mobile robot based on multi-vision technology described in the embodiments of the present application, acquisition data of each acquisition position is obtained based on the laser scanner and the stereo vision system, and the acquisition data of each acquisition position is stitched into a TVS point cloud and a stereo vision point cloud, specifically including: Install the laser scanner on the mobile robot, calibrate the laser scanner, and obtain the scanning parameters of the laser scanner, where the scanning parameters include the scanning angle range, resolution, and scanning frequency. Obtain the laser scan data in real time based on the scanning parameters of the laser scanner, preprocess the laser scan data, and remove the noise points based on Gaussian filtering or median filtering. Outlier points in the laser scan data are removed based on statistical analysis methods to obtain optimized data. The optimized data is normalized to obtain preprocessed data, and TVS point clouds are generated based on the preprocessed data. Stereo images are collected based on a stereo vision system, the matching relationships of corresponding points in the left and right images are analyzed, the features of the stereo images are extracted, and stereo vision point clouds are generated.

[0009] Optionally, in the mobile robot extrinsic parameter calibration method based on multi-vision technology described in the embodiments of this application, plane fitting is performed on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes, specifically including: Based on minimizing the sum of the squares of the distances from the points in the point cloud to the fitting plane, the plane parameters are determined. Based on the plane parameters, a plane fitting algorithm is generated. The TVS point cloud data is input into the plane fitting algorithm to generate a TVS point cloud fitting plane. The stereo vision point cloud is input into the plane fitting algorithm to generate a stereo vision point cloud fitting plane. Based on the TVS point cloud fitting plane and the stereo vision point cloud fitting plane, three-plane fitting in the three-dimensional space is performed to obtain orthogonal vectors of the three planes.

[0010] Optionally, in the mobile robot extrinsic parameter calibration method based on multi-vision technology described in the embodiments of this application, based on the least squares method and singular value analysis of the orthogonal vectors of the three planes, the normal vectors of the three planes are calculated, specifically including: The orthogonal vectors of the three planes are obtained, and the orthogonal vectors of the three planes are fitted based on the least squares method to obtain a fitting result. A plane equation is constructed, and the preliminary coefficients of the plane equation are analyzed based on the fitting result. Based on the orthogonal vectors of the three planes, singular values are calculated, and decomposition is performed according to the singular values to generate the normal vectors of the three planes.

[0011] Optionally, in the mobile robot extrinsic parameter calibration method based on multi-vision technology described in the embodiments of this application, based on the normal vectors of the three planes, a rotation matrix is calculated. According to the rotation matrix, the stereo vision point cloud is rotated or / and translated to align the stereo vision point cloud with the TVS point cloud to obtain a calibration result, specifically including: Based on the plane normal vectors of the TVS point cloud and the stereo vision point cloud, a rotation matrix is calculated. The calculation formula of the rotation matrix is as follows: , where, , T represents the transpose operation, and R represents a 3×3 rotation matrix, which is used to describe the rotation transformation relationship between the stereo vision point cloud and the TVS point cloud; and respectively represent the plane normal vector matrices of the TVS point cloud and the stereo vision point cloud; and and respectively represent three orthogonal normal vectors of the plane fitted by the TVS point cloud; and and represent three orthogonal normal vectors of the plane fitted by the stereo vision point cloud; The function of the R rotation matrix is to rotate the coordinate system of the stereo vision system to align with the coordinate system of the laser scanner (TVS).

[0012] Specifically, and and are three mutually orthogonal plane normal vectors obtained by plane fitting and singular value decomposition of the TVS point cloud, usually corresponding to the normal directions of three orthogonal planes (such as the top surface, side surface, and end surface) inside the ventilation duct, and these three vectors form the basis vectors of the local coordinate system of the TVS point cloud.

[0013] Similar to the TVS point cloud, and and are three orthogonal plane normal vectors obtained by plane fitting and singular value decomposition of the stereo vision system, representing the basis vectors of the local coordinate system of the stereo vision system.

[0014] Rotate and translate the stereo vision point cloud to align the stereo vision point cloud with the TVS point cloud: The rotation calculation formula is as follows: ; In the formula represents the point set in the stereo vision point cloud; represents the center point in the camera coordinate system; represents the rotated point; R represents the rotation matrix; The translation calculation formula is as follows: ; In the formula: represents the point after rotation and translation; represents the center point coordinates of the TVS point cloud; represents the coordinate position of the camera centroid after rotation.

[0015] Second aspect, an embodiment of the present application provides an external parameter calibration system for a mobile robot based on multi-vision technology, the system comprising: a memory and a processor, wherein the memory includes a program of an external parameter calibration method for a mobile robot based on multi-vision technology, and when the program of the external parameter calibration method for a mobile robot based on multi-vision technology is executed by the processor, the following steps are implemented: Set a plurality of acquisition positions in the ventilation duct, and control the mobile robot to sequentially move to each acquisition position in the ventilation duct based on the acquisition positions; Obtain the acquisition data of each acquisition position based on the laser scanner and the stereo vision system, and splice the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud; Perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes; Based on the least squares method and singular value analysis of the orthogonal vectors of the three planes, calculate the normal vectors of the three planes; Calculate the rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result.

[0016] Optionally, in the external parameter calibration system for a mobile robot based on multi-vision technology described in the embodiment of the present application, setting a plurality of acquisition positions in the ventilation duct and controlling the mobile robot to sequentially move to each acquisition position in the ventilation duct based on the acquisition positions specifically includes: Analyze the distribution information of the ventilation duct based on the building drawings, and the distribution information includes the direction, branch situation and pipe diameter size of the ventilation duct; Analyze the elbow, branch node, reduced diameter position of the pipe and the areas with abnormal air flow or pollutant accumulation based on the distribution information of the ventilation duct; Set a plurality of acquisition positions according to the elbow, branch node, reduced diameter position of the pipe and the areas with abnormal air flow or pollutant accumulation; Set the robot movement route based on the plurality of acquisition positions, and set the robot movement parameters, where the robot movement parameters include the robot movement speed, the robot steering angle, the robot acceleration and the robot deceleration; Control the robot to sequentially move to each acquisition position along the movement route according to the robot movement parameters.

[0017] Optionally, in the external parameter calibration system for a mobile robot based on multi-vision technology described in the embodiment of the present application, obtaining the acquisition data of each acquisition position based on the laser scanner and the stereo vision system and splicing the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud specifically includes: Install a laser scanner on a mobile robot, calibrate the laser scanner, and obtain the scanning parameters of the laser scanner. The scanning parameters include the scanning angle range, resolution, and scanning frequency. Obtain laser scanning data in real time based on the scanning parameters of the laser scanner, preprocess the laser scanning data, and remove noise points based on Gaussian filtering or median filtering. Eliminate outliers in the laser scanning data based on statistical analysis methods to obtain optimized data, normalize the optimized data to obtain preprocessed data, and generate a TVS point cloud according to the preprocessed data. Collect stereo images based on a stereo vision system, analyze the matching relationship of corresponding points in the left and right images, extract the features of the stereo images, and generate a stereo vision point cloud.

[0018] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a program for calibrating the external parameters of a mobile robot based on multi-vision technology. When the program for calibrating the external parameters of a mobile robot based on multi-vision technology is executed by a processor, the steps of the method for calibrating the external parameters of a mobile robot based on multi-vision technology as described in any one of the above are implemented.

[0019] As can be seen from the above, a method, system, and medium for calibrating the external parameters of a mobile robot based on multi-vision technology provided by an embodiment of the present application set multiple acquisition positions in a ventilation duct, control the mobile robot to move sequentially to each acquisition position in the ventilation duct based on the acquisition positions; obtain the acquisition data of each acquisition position based on the laser scanner and the stereo vision system, and splice the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud; perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes; calculate the normal vectors of the three planes based on the least squares method and singular value analysis of the orthogonal vectors of the three planes; calculate a rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain a calibration result; by combining a laser scanner and a stereo vision system and using the planar surface of the ventilation duct as a calibration object, it is possible to perform real-time external parameter calibration in a dark or complex environment without relying on a calibration target, and achieve high-precision external parameter calibration. Description of the Drawings

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative efforts.

[0021] Figure 1 Flow chart of the external parameter calibration method for a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 2 Flow chart of the mobile robot control method for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 3 Flow chart of the TVS point cloud and stereo vision point cloud analysis method for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 4 Schematic diagram of a TVS positioner and its components for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 5 Schematic diagram of a TVS aperture and its components for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 6 Schematic diagram of the triangulation parameters of a stereo vision system for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 7 Data acquisition and external parameter calibration block diagram for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 8 Comparison diagram of the TVS point cloud and the stereo vision point cloud for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application; Figure 9 Comparison diagram of the point cloud after fitting a plane for the external parameter calibration method of a mobile robot based on multi-vision technology provided by an embodiment of the present application. Detailed implementation manners

[0022] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0023] It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first" and "second" are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0024] Please refer to Figure 1 , Figure 1 which is a flowchart of a method for calibrating the extrinsic parameters of a mobile robot based on multi-vision technology in some embodiments of the present application. The method for calibrating the extrinsic parameters of a mobile robot based on multi-vision technology is used in a terminal device. The method for calibrating the extrinsic parameters of a mobile robot based on multi-vision technology includes the following steps: S101, set a plurality of acquisition positions in the ventilation duct, and control the mobile robot to move sequentially to each acquisition position in the ventilation duct based on the acquisition positions; S102, acquire the acquisition data of each acquisition position based on the laser scanner and the stereo vision system, and splice the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud; S103, perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes; S104, calculate the normal vectors of the three planes based on the least squares method and singular value analysis of the orthogonal vectors of the three planes; S105, calculate the rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result.

[0025] It should be noted that the mobile robot is placed in the ventilation duct, the laser scanner and the stereo vision system are started, the mobile robot moves along the y-axis and stops at multiple positions k, the data of the laser scanner and the stereo vision system are acquired, the centroid of the laser spot is calculated using image processing technology, and its three-dimensional coordinates are calculated by triangulation. The acquired data are spliced into a point cloud, and plane fitting and orthogonal vector generation are performed, the rotation matrix is calculated, and the stereo vision point cloud is rotated and translated to align it with the TVS point cloud. The accuracy and robustness of the extrinsic parameter calibration are verified through experiments.

[0026] Please refer to Figure 2 , Figure 2 which is a flowchart of a control method for a mobile robot in a method for calibrating the extrinsic parameters of a mobile robot based on multi-vision technology in some embodiments of the present application. According to an embodiment of the present invention, a plurality of acquisition positions are set in the ventilation duct, and the mobile robot is controlled to move sequentially to each acquisition position in the ventilation duct based on the acquisition positions, which specifically includes: S201, analyzing distribution information of ventilation ducts based on architectural drawings, where the distribution information includes directions, branching conditions, and pipe diameters of the ventilation ducts; S202, analyzing the elbows, branch nodes, diameter change positions, and areas with abnormal airflow or accumulation of pollutants in the duct based on the distribution information of the ventilation duct; S203, setting a plurality of collection locations according to the elbows, branch nodes, diameter change locations of the pipeline, and areas with abnormal airflow or accumulation of pollutants; S204, setting a robot movement route based on the multiple acquisition positions, and setting robot movement parameters, where the robot movement parameters include robot movement speed, robot steering angle, robot acceleration, and robot deceleration; S205, controlling the robot to move to each collection position in sequence along the moving route according to the robot movement parameters.

[0027] It should be noted that the mobile robot is controlled to move to each acquisition position in turn, and the laser scanner is started at each position for scanning. The laser scanner emits a laser beam and receives the reflected laser signal. The distance from the target point to the scanner is calculated based on the principle of the signal's flight time or phase difference, thereby obtaining the three-dimensional coordinate information of the target point. During the scanning process, the scanning data of each acquisition position is recorded.

[0028] Please refer to Figure 3 , Figure 3 This is a flow chart of a TVS point cloud and stereo vision point cloud analysis method of a mobile robot external parameter calibration method based on multi-vision technology in some embodiments of the present application. According to an embodiment of the present invention, based on a laser scanner and a stereo vision system, the collected data of each collection position is obtained, and the collected data of each collection position is spliced into a TVS point cloud and a stereo vision point cloud, specifically including: S301, installing a laser scanner on the mobile robot, and calibrating the laser scanner to obtain scanning parameters of the laser scanner, where the scanning parameters include scanning angle range, resolution, and scanning frequency; S302, acquiring laser scanning data in real time based on scanning parameters of the laser scanner, preprocessing the laser scanning data, and removing noise points based on Gaussian filtering or median filtering; S303, removing outliers from the laser scanning data based on a statistical analysis method to obtain optimized data, normalizing the optimized data to obtain preprocessed data, and generating a TVS point cloud based on the preprocessed data; S304, collecting stereo images based on the stereo vision system, analyzing the matching relationship between corresponding points in the left and right images, extracting features of the stereo images, and generating a stereo vision point cloud.

[0029] It should be noted that the data collected at each position k is spliced into two point clouds, and the calculation formula is as follows: TVS point cloud: , represents the TVS point cloud, that is, the three-dimensional point cloud data set obtained by the laser scanner; u represents the union of all collection positions k, that is, the summary of the data at all collection positions; m represents the number of points obtained at the k-th collection position, that is, the number of three-dimensional points scanned at this position; k represents the index of the collection position, that is, the k-th position where the mobile robot stops in the ventilation duct; i represents the i-th point at the k-th collection position, that is, the index of the point scanned at this position; x, y, and z respectively represent the three-dimensional coordinate values of the i-th point at the k-th collection position; Stereo vision point cloud: , represents the stereo vision point cloud data, that is, the three-dimensional point cloud set obtained by the stereo vision system; N is the number of points collected at each position k, represents the cumulative displacement of the mobile robot on the y-axis.

[0030] According to the embodiments of the present invention, plane fitting is respectively performed on the TVS point cloud and the stereo vision point cloud to generate orthogonal vectors of three planes, specifically including: Based on minimizing the sum of the squares of the distances from the points in the point cloud to the fitted plane, the plane parameters are determined; Based on the plane parameters, a plane fitting algorithm is generated, and the TVS point cloud data is input into the plane fitting algorithm to generate the TVS point cloud fitting plane; The stereo vision point cloud is input into the plane fitting algorithm to generate the stereo vision point cloud fitting plane; Based on the TVS point cloud fitting plane and the stereo vision point cloud fitting plane, three-plane fitting in the three-dimensional space is performed to obtain the orthogonal vectors of the three planes.

[0031] It should be noted that data preprocessing may be required during plane fitting, such as removing outliers, noise points, etc., to improve the accuracy and reliability of fitting. At the same time, during the process of calculating the orthogonal vectors, special cases such as vector collinearity should be noted to ensure that the generated vectors are orthogonal to each other. Plane fitting is respectively performed on the TVS point cloud and the stereo vision point cloud, and the orthogonal vectors of the corresponding three planes are generated.

[0032] According to the embodiments of the present invention, based on the least squares method and singular value analysis of the orthogonal vectors of the three planes, the normal vectors of the three planes are calculated, specifically including: Obtain the orthogonal vectors of three planes, and fit the orthogonal vectors of the three planes based on the least squares method to obtain the fitting result; Construct a plane equation, and analyze the preliminary coefficients of the plane equation based on the fitting result; Calculate the singular values based on the orthogonal vectors of the three planes, perform decomposition according to the singular values, and generate the normal vectors of the three planes.

[0033] It should be noted that after singular value decomposition, the right singular vector corresponding to the smallest singular value is the normal vector of the plane.

[0034] According to the embodiments of the present invention, calculate the rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result, specifically including: Calculate the rotation matrix based on the plane normal vectors of the TVS point cloud and the stereo vision point cloud. The calculation formula of the rotation matrix is as follows: , where, , R represents a 3×3 rotation matrix, which is used to describe the rotation transformation relationship between the stereo vision point cloud and the TVS point cloud; and respectively represent the plane normal vector matrices of the TVS point cloud and the stereo vision point cloud; , and respectively represent the three orthogonal normal vectors of the plane fitted by the TVS point cloud; , and represent the three orthogonal normal vectors of the plane fitted by the stereo vision point cloud; The role of the R rotation matrix is to rotate the coordinate system of the stereo vision system to align with the coordinate system of the laser scanner (TVS).

[0035] Specifically, , and are the three mutually orthogonal plane normal vectors obtained after the TVS point cloud is subjected to plane fitting and singular value decomposition, and usually correspond to the normal directions of three orthogonal planes (such as the top surface, side surface, and end surface) inside the ventilation duct. These three vectors constitute the basis vectors of the local coordinate system of the TVS point cloud.

[0036] Similar to the TVS point cloud, , and are the three orthogonal plane normal vectors obtained after the stereo vision system is subjected to plane fitting and singular value decomposition, representing the basis vectors of the local coordinate system of the stereo vision system.

[0037] Rotate and translate the stereo vision point cloud to align the stereo vision point cloud with the TVS point cloud: The rotation calculation formula is as follows: ; In the formula represents the point set in the stereo vision point cloud; represents the center point in the camera coordinate system; represents the rotated point; R represents the rotation matrix; The translation calculation formula is as follows: ; In the formula: represents the point after rotation and translation; represents the center point coordinates of the TVS point cloud; represents the coordinate position of the camera centroid after rotation.

[0038] It should be noted that the three normal vectors of the stereo vision point cloud and the TVS point cloud are used as column vectors respectively to construct a matrix, and the singular value decomposition (SVD) is used to calculate the rotation matrix. You can make corresponding adjustments and calculations according to the actual data for the process of rotating and translating the stereo vision point cloud.

[0039] As Figures 4 - 9 shown, in the second aspect, the embodiment of the present application provides a mobile robot extrinsic parameter calibration system based on multi-vision technology. The system includes: a memory and a processor. The memory includes a program of the mobile robot extrinsic parameter calibration method based on multi-vision technology. When the program of the mobile robot extrinsic parameter calibration method based on multi-vision technology is executed by the processor, the following steps are implemented: Set multiple acquisition positions in the ventilation duct, and control the mobile robot to move to each acquisition position in the ventilation duct in sequence based on the acquisition positions; Obtain the acquisition data of each acquisition position based on the laser scanner and the stereo vision system, and splice the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud; Perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes; Calculate the normal vectors of the three planes based on the orthogonal vectors of the three planes by using the least squares method and singular value analysis; Calculate the rotation matrix based on the normal vectors of the three planes, and rotate or / and translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result.

[0040] Specifically, the system includes the following components: Laser scanner (TVS): Based on the principle of dynamic triangulation, it can accurately estimate the three-dimensional coordinates of the surface where the laser beam is projected.

[0041] Stereo vision system: Consisting of two cameras, it is used to capture images of the laser spot and calculate the three-dimensional coordinates of the laser spot through triangulation.

[0042] Mobile robot: Equipped with the above sensors, it can move inside the ventilation duct and collect data.

[0043] It should be noted that the mobile robot is controlled to move to each collection position in sequence, and the laser scanner is started for scanning at each position. The laser scanner emits a laser beam and receives the reflected laser signal, and calculates the distance from the target point to the scanner according to the principle such as the flight time or phase difference of the signal, so as to obtain the three-dimensional coordinate information of the target point. During the scanning process, the scanning data of each collection position is recorded.

[0044] Control and data processing unit: Used to control the data collection of the sensor, process the point cloud data, and execute the external parameter calibration algorithm.

[0045] According to the embodiments of the present invention, multiple collection positions are set inside the ventilation duct, and based on the collection positions, the mobile robot is controlled to move to each collection position inside the ventilation duct in sequence, specifically including: Analyze the distribution information of the ventilation duct based on the building drawings, and the distribution information includes the orientation, branch situation and pipe diameter size of the ventilation duct; Analyze the elbow, branch node, reduced diameter position of the pipe and the areas with abnormal air flow or pollutant accumulation based on the distribution information of the ventilation duct; Set multiple collection positions according to the elbow, branch node, reduced diameter position of the pipe and the areas with abnormal air flow or pollutant accumulation; Set the robot movement route based on multiple collection positions, and set the robot movement parameters. The robot movement parameters include the robot movement speed, robot steering angle, robot acceleration and robot deceleration; Control the robot to move to each collection position along the movement route in sequence according to the robot movement parameters.

[0046] It should be noted that the mobile robot is controlled to move to each collection position in sequence, and the laser scanner is started for scanning at each position. The laser scanner emits a laser beam and receives the reflected laser signal, and calculates the distance from the target point to the scanner according to the principle such as the flight time or phase difference of the signal, so as to obtain the three-dimensional coordinate information of the target point. During the scanning process, the scanning data of each collection position is recorded.

[0047] According to an embodiment of the present invention, acquisition data at each acquisition position is obtained based on a laser scanner and a stereo vision system, and the acquisition data at each acquisition position is stitched into a TVS point cloud and a stereo vision point cloud, specifically including: Install the laser scanner on the mobile robot, calibrate the laser scanner, and obtain the scanning parameters of the laser scanner. The scanning parameters include the scanning angle range, resolution, and scanning frequency. Obtain laser scan data in real time based on the scanning parameters of the laser scanner, preprocess the laser scan data, and remove noise points based on Gaussian filtering or median filtering. Eliminate outliers in the laser scan data based on a statistical analysis method to obtain optimized data, normalize the optimized data to obtain preprocessed data, and generate a TVS point cloud according to the preprocessed data. Collect stereo images based on the stereo vision system, analyze the matching relationship of corresponding points in the left and right images, extract the features of the stereo images, and generate a stereo vision point cloud.

[0048] It should be noted that the data collected at each position k is stitched into two point clouds, and the calculation formula is as follows: TVS point cloud: , Stereo vision point cloud: , where N is the number of points collected at each position k, represents the cumulative displacement of the mobile robot on the y-axis.

[0049] According to an embodiment of the present invention, plane fitting is respectively performed on the TVS point cloud and the stereo vision point cloud to generate orthogonal vectors of three planes, specifically including: Determine plane parameters based on minimizing the sum of the squares of the distances from the points in the point cloud to the fitted plane. Generate a plane fitting algorithm based on the plane parameters, input the TVS point cloud data into the plane fitting algorithm, and generate a TVS point cloud fitted plane. Input the stereo vision point cloud into the plane fitting algorithm to generate a stereo vision point cloud fitted plane. Perform three-plane fitting in the three-dimensional space based on the TVS point cloud fitted plane and the stereo vision point cloud fitted plane to obtain orthogonal vectors of three planes.

[0050] It should be noted that data preprocessing such as removing outliers and noise points may be required during plane fitting to improve the accuracy and reliability of fitting. At the same time, during the process of calculating orthogonal vectors, special cases such as vector collinearity should be noted to ensure that the generated vectors are orthogonal to each other. Plane fitting is respectively performed on the TVS point cloud and the stereo vision point cloud, and orthogonal vectors of the corresponding three planes are generated.

[0051] According to an embodiment of the present invention, based on the least squares method and singular value analysis of the orthogonal vectors of three planes, the normal vectors of the three planes are calculated, specifically including: Obtain the orthogonal vectors of the three planes, and fit the orthogonal vectors of the three planes based on the least squares method to obtain a fitting result; Construct a plane equation, and analyze the preliminary coefficients of the plane equation based on the fitting result; Calculate the singular values based on the orthogonal vectors of the three planes, and decompose according to the singular values to generate the normal vectors of the three planes.

[0052] It should be noted that after singular value decomposition, the right singular vector corresponding to the minimum singular value is the normal vector of the plane.

[0053] According to an embodiment of the present invention, based on the normal vectors of the three planes, a rotation matrix is calculated, and the stereo vision point cloud is rotated or / and translated according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain a calibration result, specifically including: Based on the plane normal vectors of the TVS point cloud and the stereo vision point cloud, calculate the rotation matrix. The calculation formula of the rotation matrix is as follows: , Wherein, , Rotate and translate the stereo vision point cloud to align the stereo vision point cloud with the TVS point cloud: The rotation calculation formula is as follows: ; In the formula represents the point set in the stereo vision point cloud; represents the center point in the camera coordinate system; represents the rotated point; R represents the rotation matrix; The translation calculation formula is as follows: ; In the formula: represents the point after rotation and translation; represents the center point coordinate of the TVS point cloud; represents the coordinate position of the camera centroid after rotation.

[0054] It should be noted that the three normal vectors of the stereo vision point cloud and the TVS point cloud are respectively used as column vectors to construct a matrix, and the singular value decomposition (SVD) is used to calculate the rotation matrix. In the process of rotating and translating the stereo vision point cloud, you can make corresponding adjustments and calculations according to the actual data.

[0055] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for the external parameter calibration method of a mobile robot based on multi-vision technology. When the program for the external parameter calibration method of the mobile robot based on multi-vision technology is executed by a processor, the steps of the external parameter calibration method of the mobile robot based on multi-vision technology as described in any one of the above are implemented.

[0056] An external parameter calibration method, system and medium for a mobile robot based on multi-vision technology disclosed in the present invention, by setting a plurality of acquisition positions in a ventilation duct, controlling the mobile robot to sequentially move to each acquisition position in the ventilation duct based on the acquisition positions; acquiring the acquisition data of each acquisition position based on a laser scanner and a stereo vision system, and splicing the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud; respectively performing plane fitting on the TVS point cloud and the stereo vision point cloud to generate orthogonal vectors of three planes; calculating the normal vectors of the three planes based on the least squares method and singular value analysis of the orthogonal vectors of the three planes; calculating a rotation matrix based on the normal vectors of the three planes, and rotating and / or translating the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain a calibration result; by combining a laser scanner and a stereo vision system and using the flat surface of the ventilation duct as a calibration object, it is possible to perform external parameter calibration in real time in a dark or complex environment and without relying on a calibration target, achieving high-precision external parameter calibration.

[0057] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0058] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they may be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0059] In addition, in each embodiment of the present invention, each functional unit can be entirely integrated into one processing unit, or each unit can be separately regarded as one unit alone, or two or more units can be integrated into one unit; the above-mentioned integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0060] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other various media that can store program codes.

[0061] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence, or the part that makes a contribution to the prior art can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. And the foregoing storage medium includes: removable storage devices, ROM, RAM, magnetic disks or optical disks, and other various media that can store program codes.

Claims

1. A method for calibrating the external parameters of a mobile robot based on multi-vision technology, characterized in that, Including: Set multiple collection positions in the ventilation duct, and based on the collection positions, control the mobile robot to sequentially move to each collection position in the ventilation duct; Obtain the collection data of each collection position based on the laser scanner and the stereo vision system, and splice the collection data of each collection position into a TVS point cloud and a stereo vision point cloud; Perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes; Based on the least squares method and singular value analysis of the orthogonal vectors of the three planes, calculate the normal vectors of the three planes; Calculate the rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result.

2. The method for calibrating the external parameters of a mobile robot based on multi-vision technology according to claim 1, wherein Set multiple collection positions in the ventilation duct, and based on the collection positions, control the mobile robot to sequentially move to each collection position in the ventilation duct. Specifically, it includes: Analyze the distribution information of the ventilation duct based on the building drawings, and the distribution information includes the direction, branch situation, and pipe diameter size of the ventilation duct; Analyze the elbow, branch node, diameter-changing position of the pipe, and the area with abnormal air flow or pollutant accumulation based on the distribution information of the ventilation duct; Set multiple collection positions according to the elbow, branch node, diameter-changing position of the pipe, and the area with abnormal air flow or pollutant accumulation; Set the robot movement route based on multiple collection positions, and set the robot movement parameters, where the robot movement parameters include the robot movement speed, robot steering angle, robot acceleration, and robot deceleration; Control the robot to sequentially move to each collection position along the movement route according to the robot movement parameters.

3. The method for calibrating the external parameters of a mobile robot based on multi-vision technology according to claim 2, wherein Obtain the collection data of each collection position based on the laser scanner and the stereo vision system, and splice the collection data of each collection position into a TVS point cloud and a stereo vision point cloud. Specifically, it includes: Install the laser scanner on the mobile robot, calibrate the laser scanner, and obtain the scanning parameters of the laser scanner, where the scanning parameters include the scanning angle range, resolution, and scanning frequency; Obtain the laser scan data in real time based on the scanning parameters of the laser scanner, preprocess the laser scan data, and remove the noise points based on Gaussian filtering or median filtering; Eliminate the outlier points of the laser scan data based on the statistical analysis method to obtain the optimized data, normalize the optimized data to obtain the preprocessed data, and generate the TVS point cloud according to the preprocessed data; Collect stereo images based on the stereo vision system, analyze the matching relationship of corresponding points in the left and right images, extract the features of the stereo images, and generate a stereo vision point cloud.

4. The method for calibrating the external parameters of a mobile robot based on multi-vision technology according to claim 3, wherein Perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate orthogonal vectors of three planes. Specifically, it includes: Determine the plane parameters based on minimizing the sum of the squares of the distances from the points in the point cloud to the fitted plane; Generate a plane fitting algorithm based on the plane parameters, input the TVS point cloud data into the plane fitting algorithm, and generate a TVS point cloud fitted plane; Input the stereo vision point cloud into the plane fitting algorithm to generate a stereo vision point cloud fitted plane; Perform three - plane fitting in a three - dimensional space based on the planes fitted from the TVS point cloud and the planes fitted from the stereo vision point cloud to obtain the orthogonal vectors of the three planes.

5. The method for calibrating the extrinsic parameters of a mobile robot based on multi-vision technology according to claim 4, wherein Based on the least - squares method and singular - value analysis of the orthogonal vectors of the three planes, calculate the normal vectors of the three planes, specifically including: Obtain the orthogonal vectors of the three planes, perform fitting on the orthogonal vectors of the three planes based on the least - squares method to obtain the fitting result; Construct a plane equation, and analyze the preliminary coefficients of the plane equation based on the fitting result; Calculate the singular values based on the orthogonal vectors of the three planes, perform decomposition according to the singular values to generate the normal vectors of the three planes.

6. The method for calibrating the external parameters of a mobile robot based on multi-vision technology according to claim 5, wherein Calculate the rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result, specifically including: Based on the plane normal vectors of the TVS point cloud and the stereo vision point cloud, calculate the rotation matrix. The formula for the rotation matrix is as follows: ， Among them, , $R$ represents a 3×3 rotation matrix used to describe the rotation transformation relationship between the stereo vision point cloud and the TVS point cloud; The represents the TVS point cloud; The represents the planar normal vector matrix of the stereo vision point cloud; $T$ represents the transpose operation, and respectively represent the three orthogonal normal vectors of the plane fitted by the TVS point cloud; , and represent three orthogonal normal vectors for fitting a plane to a stereoscopic vision point cloud. Rotate and translate the stereo vision point cloud to align the stereo vision point cloud with the TVS point cloud: The rotation calculation formula is as follows: ； In the formula represents the set of points in the stereo vision point cloud; represents the center point in the camera coordinate system; represents the rotated point; R represents the rotation matrix; The translation calculation formula is as follows: ； In the formula: represents the point after rotation and translation; represents the center point coordinates of the TVS point cloud; represents the coordinate position of the camera centroid after rotation.

7. A mobile robot external parameter calibration system based on multi-vision technology, characterized in that, The system includes: a memory and a processor. The memory includes a program for the external - parameter calibration method of a mobile robot based on multi - vision technology. When the program for the external - parameter calibration method of the mobile robot based on multi - vision technology is executed by the processor, the following steps are implemented: Set multiple acquisition positions in the ventilation duct, and control the mobile robot to move sequentially to each acquisition position in the ventilation duct based on the acquisition positions; Obtain the acquisition data of each acquisition position based on the laser scanner and the stereo vision system, and splice the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud; Perform plane fitting on the TVS point cloud and the stereo vision point cloud respectively to generate the orthogonal vectors of the three planes; Based on the least - squares method and singular - value analysis of the orthogonal vectors of the three planes, calculate the normal vectors of the three planes; Calculate the rotation matrix based on the normal vectors of the three planes, and rotate and / or translate the stereo vision point cloud according to the rotation matrix to align the stereo vision point cloud with the TVS point cloud to obtain the calibration result.

8. The external parameter calibration system for a mobile robot based on multi-vision technology according to claim 7, wherein Set multiple acquisition positions in the ventilation duct, and control the mobile robot to move sequentially to each acquisition position in the ventilation duct based on the acquisition positions, specifically including: Analyze the distribution information of the ventilation duct based on the architectural drawings. The distribution information includes the orientation, branch situation, and pipe diameter size of the ventilation duct; Analyze the elbow joints, branch nodes, diameter - changing positions, and areas with abnormal airflows or pollutant accumulations in the pipe based on the distribution information of the ventilation duct; Set multiple acquisition positions according to the elbow joints, branch nodes, diameter - changing positions, and areas with abnormal airflows or pollutant accumulations in the pipe; Set the robot movement route based on the multiple acquisition positions, and set the robot movement parameters. The robot movement parameters include the robot movement speed, robot steering angle, robot acceleration, and robot deceleration; Control the robot to move sequentially to each acquisition position along the movement route according to the robot movement parameters.

9. The external parameter calibration system for a mobile robot based on multi-vision technology according to claim 8, characterized in that, Obtain the acquisition data of each acquisition position based on the laser scanner and the stereo vision system, and splice the acquisition data of each acquisition position into a TVS point cloud and a stereo vision point cloud, specifically including: Install a laser scanner on a mobile robot, calibrate the laser scanner, and obtain the scanning parameters of the laser scanner, where the scanning parameters include the scanning angle range, resolution, and scanning frequency; Obtain laser scan data in real time based on the scanning parameters of the laser scanner, preprocess the laser scan data, and remove noise points based on Gaussian filtering or median filtering; Eliminate outliers in the laser scan data based on a statistical analysis method to obtain optimized data, normalize the optimized data to obtain preprocessed data, and generate a TVS point cloud based on the preprocessed data; Collect stereo images based on a stereo vision system, analyze the matching relationship of corresponding points in the left and right images, extract the features of the stereo images, and generate a stereo vision point cloud.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a program for the external parameter calibration method of a mobile robot based on multi-vision technology. When the program for the external parameter calibration method of a mobile robot based on multi-vision technology is executed by a processor, the steps of the external parameter calibration method of a mobile robot based on multi-vision technology as described in any one of claims 1 to 6 are implemented.

Images