A Localization Method for Bridge Inspection Robots Based on Multi-Sensor Information Fusion

Abstract

The present invention relates to the technical field of data processing, and specifically relates to a positioning method for a bridge detection robot with multi-sensor information fusion. The method includes: respectively constructing a building point cloud coordinate system and a carrier point cloud coordinate system according to the point cloud data of the target bridge and the point cloud data of the objects in the environment where the target bridge is located; rotating and aligning the building point cloud coordinate system and the carrier point cloud coordinate system according to the similarity of the relative distances between each point cloud in the carrier point cloud coordinate system and other point clouds and the relative distances between each point cloud of the bridge pier in the building point cloud coordinate system and other point clouds; screening the real shooting area according to the similarity of the positions of all the real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each bridge pier of the target bridge, and the similarity of the position distribution of the corner points in the image and the corner points in the rotated and aligned coordinate system, and positioning the robot in combination with the orientation of the robot. The present invention improves the positioning accuracy of the bridge detection robot.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a positioning method for a bridge inspection robot with multi-sensor information fusion. Background Art

[0002] A bridge inspection robot is an intelligent device specifically used for bridge structure inspection. It can move flexibly at various structural parts of the bridge and is equipped with a variety of sensors and detection devices to achieve comprehensive, efficient, and accurate inspection of the bridge.

[0003] Due to the large and complex bridge structure and the need to accurately locate abnormal positions during inspection, such as cracks on the surface of bridge piers and fractures on the surface of ropes, etc., it is necessary to accurately position the bridge inspection robot during its working process to evaluate the impact of abnormal points on the bridge structure and facilitate planning the inspection path of the robot to avoid colliding with the bridge structure and causing unnecessary losses.

[0004] Existing positioning methods for bridge inspection robots are mainly achieved through methods such as GPS, visual positioning, and lidar positioning. However, these positioning methods are often affected by insufficient signal strength and dim and dusty environments, resulting in a large impact on the positioning accuracy of the robot. Summary of the Invention

[0005] In order to solve the problem of low positioning accuracy when the existing method positions the robot in the bridge inspection environment, the purpose of the present invention is to provide a positioning method for a bridge inspection robot with multi-sensor information fusion, and the specific technical solutions adopted are as follows:

[0006] The present invention provides a positioning method for a bridge inspection robot with multi-sensor information fusion, and the method includes the following steps:

[0007] Obtain the point cloud data of the target bridge, the point cloud data of the objects in the environment where the target bridge is located, and the images captured by the target bridge inspection robot; construct a building point cloud coordinate system according to the point cloud data of the target bridge, and construct a carrier point cloud coordinate system according to the point cloud data of the objects in the environment where the target bridge is located;

[0008] Align the building point cloud coordinate system and the carrier point cloud coordinate system by rotation according to the similarity between the relative distances of each point cloud in the carrier point cloud coordinate system and other point clouds and the relative distances of each point cloud of the bridge pier in the building point cloud coordinate system and other point clouds; screen the suspected shooting areas according to the similarity between the positions of all the real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each bridge pier of the target bridge;

[0009] Screen the real shooting areas from all the suspected shooting areas according to the similarity of the position distribution of the corner points in the image and the corner points in each suspected shooting area;

[0010] Locate the robot according to the position distribution of the point cloud in the real shooting area and the orientation of the robot.

[0011] Preferably, the rotation alignment of the building point cloud coordinate system and the carrier point cloud coordinate system according to the similarity between the relative distances between each point cloud in the carrier point cloud coordinate system and other point clouds and the relative distances between each point cloud of the pier in the building point cloud coordinate system and other point clouds includes:

[0012] Calculate the first average distance between each point cloud in the carrier point cloud coordinate system and all other point clouds in the carrier point cloud coordinate system respectively;

[0013] For any point cloud on any pier in the building point cloud coordinate system: calculate the second average distance between the any point cloud on the any pier and all other point clouds on the any pier; obtain the first similarity between each point cloud in the carrier point cloud coordinate system and the any point cloud on the any pier according to the first difference between the first average distance and the second average distance, and the first difference is negatively correlated with the first similarity;

[0014] Match the point cloud in the carrier point cloud coordinate system with the point cloud on any pier of the target bridge according to the magnitude relationship of the first similarity between each point cloud in the carrier point cloud coordinate system and all the point clouds on the any pier, and determine the pier in the carrier point cloud coordinate system;

[0015] Rotate and align the building point cloud coordinate system and the carrier point cloud coordinate system according to the positions of the piers in the carrier point cloud coordinate system and the positions of the piers in the building point cloud coordinate system.

[0016] Preferably, the matching of the point cloud in the carrier point cloud coordinate system with the point cloud on any pier of the target bridge according to the magnitude relationship of the first similarity between each point cloud in the carrier point cloud coordinate system and all the point clouds on the any pier to determine the pier in the carrier point cloud coordinate system includes:

[0017] For any point cloud in the building point cloud coordinate system: if the maximum value of the first similarity between the any point cloud in the building point cloud coordinate system and all the point clouds in the carrier point cloud coordinate system is greater than the preset first similarity threshold, then use the any point cloud in the building point cloud coordinate system as the matching point of the point cloud in the carrier point cloud coordinate system corresponding to the maximum value;

[0018] All the matching points in the carrier point cloud coordinate system form the pier in the carrier point cloud coordinate system.

[0019] Preferably, screening the suspected shooting areas according to the similarity between the positions of all the real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each pier of the target bridge includes:

[0020] The coordinate information of all the real-time point clouds collected by the robot at the current moment in the carrier point cloud coordinate system after rotation and alignment constitutes a first position vector; all the point clouds on each pier in the building point cloud coordinate system after rotation and alignment constitute a second position vector for each pier; calculate the second similarity between the first position vector and each second position vector;

[0021] Screen the suspected shooting areas by using the second similarity.

[0022] Preferably, screening the suspected shooting areas by using the second similarity includes:

[0023] Determine the area where the pier is located when the second similarity is greater than a preset second similarity threshold as the suspected shooting area.

[0024] Preferably, screening the real shooting areas from all the suspected shooting areas according to the similarity of the position distributions of the corner points in the image and the corner points in each suspected shooting area includes:

[0025] The coordinate information of all the corner points in the image constitutes a third position vector;

[0026] All the corner points in each suspected shooting area constitute a reference point set for each suspected shooting area; project all the points in each reference point set in the camera shooting direction to obtain the corner points of the projection area of each reference point set, and the coordinate information of the corner points of the projection area of each reference point set constitutes a fourth position vector for each reference point set;

[0027] Take the similarity between the third position vector and each fourth position vector as the matching coefficient between the corner points in the image and the reference point set of each suspected shooting area;

[0028] Screen the real shooting areas by using the matching coefficient.

[0029] Preferably, screening the real shooting areas by using the matching coefficient includes:

[0030] Determine the suspected shooting area corresponding to the maximum value of the matching coefficient as the real shooting area.

[0031] Preferably, positioning the robot according to the position distribution of the point cloud in the real shooting area and the orientation of the robot includes:

[0032] Obtain the radial distance, polar angle, and azimuth angle of each point cloud within the actual shooting area respectively; wherein, the radial distance is the distance from the point cloud to the sensor;

[0033] Based on the minimum and maximum values of the radial distances, the minimum and maximum values of the polar angles, and the minimum and maximum values of the azimuth angles of all the point clouds within the actual shooting area, determine the suspected area where the robot is located;

[0034] Locate the robot according to the position distribution of the point clouds in the suspected area where the robot is located and the unit vector of the robot's direction.

[0035] Preferably, the locating the robot according to the position distribution of the point clouds in the suspected area where the robot is located and the unit vector of the robot's direction includes:

[0036] Obtain the vector pointing from each point cloud in the suspected area where the robot is located to the polar coordinate origin as the feature vector of each point cloud;

[0037] Calculate the cosine similarity between the unit vector of the robot's direction and the feature vector of each point cloud; determine the point cloud corresponding to the maximum value of the cosine similarity as the position of the robot.

[0038] Preferably, the determining the suspected area where the robot is located based on the minimum and maximum values of the radial distances, the minimum and maximum values of the polar angles, and the minimum and maximum values of the azimuth angles of all the point clouds within the actual shooting area includes:

[0039] Obtain the radius range of the suspected area where the robot is located based on the minimum and maximum values of the radial distances of all the point clouds within the actual shooting area;

[0040] Obtain the polar angle range of the suspected area where the robot is located based on the minimum and maximum values of the polar angles of all the point clouds within the actual shooting area;

[0041] Obtain the azimuth angle range of the suspected area where the robot is located based on the minimum and maximum values of the azimuth angles of all the point clouds within the actual shooting area;

[0042] Determine the suspected area where the robot is located according to the radius range, the polar angle range, and the azimuth angle range.

[0043] The present invention has at least the following beneficial effects:

[0044] The present invention uses different sensors to collect the point cloud data of the target bridge and the point cloud data of the objects in the environment where the target bridge is located. A building point cloud coordinate system is constructed based on the point cloud data of the target bridge, and a carrier point cloud coordinate system is constructed based on the point cloud data of the objects in the environment where the target bridge is located. The building point cloud coordinate system only contains the point cloud data of the structure of the target bridge itself, while the carrier point cloud coordinate system contains not only the point cloud data of the structure of the target bridge itself but also the point cloud data of other sundries in the environment where the target bridge is located. It is necessary to perform an alignment operation on the positions in the two coordinate systems. According to the similarity between the positions of all the real-time point clouds collected by the robot at the current moment in the rotation-aligned coordinate system and the positions of the point clouds of each pier of the target bridge, the suspected shooting areas are screened. The similarity of the position distributions of the corner points in the images captured by the detection robot and the corner points in each suspected shooting area is evaluated, and the real shooting areas are screened from all the suspected shooting areas. Furthermore, the robot is accurately positioned in combination with the orientation of the robot. The method provided in this embodiment combines the data information collected by multiple sensors, so as to complement the data between different sensors in environments such as dusty and dim, avoid the situation where the positioning is inaccurate due to environmental influence of a single sensor, and improve the positioning accuracy of the bridge detection robot while ensuring the anti-interference ability of the positioning. Brief Description of the Drawings

[0045] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0046] Figure 1 It is a flowchart of a positioning method for a bridge detection robot with multi-sensor information fusion provided by an embodiment of the present invention;

[0047] Figure 2 It is a structural block diagram of a positioning system for a bridge detection robot with multi-sensor information fusion provided by an embodiment of the present invention. Detailed Embodiments

[0048] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following provides a detailed description of a positioning method for a bridge detection robot with multi-sensor information fusion proposed according to the present invention in combination with the accompanying drawings and preferred embodiments.

[0049] Unless otherwise defined, all the technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0050] The following specifically describes the specific solution of a positioning method for a bridge inspection robot with multi-sensor information fusion in conjunction with the accompanying drawings.

[0051] An embodiment of a positioning method for a bridge inspection robot with multi-sensor information fusion:

[0052] The specific scenario targeted by this embodiment is as follows: The pier is an important part in the bridge design process. Its main function is to support the upper structure of the bridge, including the bridge span structure (such as beams, arches, suspension cables, etc.) and the deck system (such as bridge deck pavement, railings, etc.). Since usually a large part of the pier is underwater and is affected by water flow impact and corrosion all year round, the inspection of the pier is usually relatively frequent to ensure the safety of the bridge. During the process of using a bridge inspection robot to inspect the bridge, it is often necessary to plan the inspection path. Before planning it, it is necessary to accurately position the bridge inspection robot. This embodiment provides a positioning method for a bridge inspection robot with multi-sensor information fusion to achieve accurate positioning of the bridge inspection robot to ensure the accurate accuracy of subsequent path planning.

[0053] This embodiment proposes a positioning method for a bridge inspection robot with multi-sensor information fusion. As Figure 1 shown, a positioning method for a bridge inspection robot with multi-sensor information fusion in this embodiment includes the following steps:

[0054] Step S1, obtain the point cloud data of the target bridge, the point cloud data of the objects in the environment where the target bridge is located, and the images captured by the target bridge inspection robot; construct a building point cloud coordinate system according to the point cloud data of the target bridge, and construct a carrier point cloud coordinate system according to the point cloud data of the objects in the environment where the target bridge is located.

[0055] A variety of sensors are installed on the bridge inspection robot, namely: a vision sensor, a lidar, and an inertial measurement unit. Among them, the vision sensor uses a camera to obtain environmental information for identifying the feature points of the pier and obstacles; the lidar emits laser beams and receives the reflected signals to obtain the three-dimensional point cloud data of the objects in the environment where the bridge is located for constructing an environmental map and precise ranging; the inertial measurement unit includes an accelerometer and a gyroscope, which measure the acceleration and angular velocity of the robot for attitude estimation and short-term position calculation.

[0056] Before the bridge inspection robot starts working, first load the structural map of the target bridge into the robot's system and pre-associate the point cloud data of the target bridge. Subsequently, activate all the sensors on the bridge inspection robot and ensure that all sensors can work properly. Further, deploy the bridge inspection robot on the pier and start inspecting the appearance of the pier. During the working process of the bridge inspection robot, continuously collect the surrounding environment information of the robot through the above-mentioned multiple sensors. The surrounding environment information includes the point cloud data of the objects in the environment where the target bridge is located, and use the camera to capture the environmental images in real time. At the same time, upload the collected data to the control terminal of the inspection personnel through wireless / wired communication methods.

[0057] Further, map all the point cloud data of the target bridge collected to the three-dimensional rectangular coordinate system, and denote the mapped coordinate system as the building point cloud coordinate system. The building point cloud coordinate system only includes the point cloud of the bridge itself. Among them, the point cloud of the bridge itself contains the accurate three-dimensional coordinate information of each part of the bridge (such as piers, bridge decks, etc.). When constructing the building point cloud coordinate system, the implementer can determine the origin position and axis direction of the building point cloud coordinates according to the structural characteristics and inspection requirements of the bridge. For example, a certain key point of the pier can be selected as the origin, the length direction of the bridge as the X-axis, the width direction as the Y-axis, and the height direction as the Z-axis; map all the point cloud data of the target bridge to this coordinate system, and the building point cloud coordinate system is obtained. In specific applications, the implementer sets the coordinate axes and the coordinate origin of this three-dimensional rectangular coordinate system according to the specific situation. Since there may be sundries such as stones and tree trunks underwater, and there is no such information in the building point cloud coordinate system, and the point cloud of the objects in the environment where the target bridge is located collected contains the information of such objects, the current position and attitude information of the bridge inspection robot are obtained through an inertial measurement unit or other positioning devices. The attitude information includes the orientation of the robot (such as pitch angle, yaw angle, roll angle), etc. Taking the geometric center of the robot or the installation position of the lidar as the coordinate origin, taking the orientation of the robot as the positive direction of the X-axis, taking the vertical direction as the Z-axis direction, and taking the direction perpendicular to both the X-axis and the Z-axis as the Y-axis, map all the point cloud data of the objects in the environment where the target bridge is located collected by the robot to the new three-dimensional rectangular coordinate system, and denote this coordinate system as the carrier point cloud coordinate system. The point cloud data of the objects in the environment where the target bridge is located exists in the carrier point cloud coordinate system. In specific applications, the implementer sets the coordinate axes and the coordinate origin of the carrier point cloud coordinate system according to the specific situation.

[0058] Step S2: Rotate and align the building point cloud coordinate system and the carrier point cloud coordinate system according to the similarity between the relative distances of each point cloud in the carrier point cloud coordinate system from other point clouds and the relative distances of each point cloud of the bridge pier in the building point cloud coordinate system from other point clouds; screen the suspected shooting areas according to the similarity between the positions of all the real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each bridge pier of the target bridge.

[0059] Since the actual positions represented by the same coordinates in the carrier point cloud coordinate system and the building point cloud coordinate system may be different, the acquired point cloud data and the constructed point cloud coordinate system are both affected by the robot's pose, which may lead to the misalignment of the directions of the carrier coordinate system and the building point cloud coordinate system. At the same time, in the underwater environment, the data collected by the robot in real time is affected by the underwater sediment and floating objects different from the pre-loaded map, and directly comparing the data of the two coordinate systems is also inaccurate. Therefore, it is first necessary to perform a matching operation on the two coordinate systems to align the building point cloud coordinate system and the carrier point cloud coordinate system for subsequent positioning. The existing method is to directly compare the data collected by the lidar with the pre-loaded map data, that is, to directly compare the two point cloud coordinate systems, usually for all data (3D point clouds) comparison, but this method has a large amount of calculation and a slow processing speed. In this embodiment, a partial feature area of the pre-loaded map will be intercepted, that is, a small area will be matched, so as to realize the overall alignment of the two coordinate systems, with less processing volume and better calculation efficiency compared with the existing method.

[0060] Specifically, calculate the average distance between each point cloud in the carrier point cloud coordinate system and all other point clouds in the carrier point cloud coordinate system, and record this average distance as the first average distance. Each point cloud in the carrier point cloud coordinate system has a corresponding first average distance.

[0061] For any point cloud on any bridge pier in the building point cloud coordinate system: calculate the average distance between this point cloud and all other point clouds on this bridge pier, and record this average distance as the second average distance; obtain the first similarity between each point cloud in the carrier point cloud coordinate system and this point cloud on this bridge pier in the building point cloud coordinate system according to the first difference between the first average distance and the second average distance, and the first difference and the first similarity are negatively correlated.

[0062] In this embodiment, a specific calculation formula for the first similarity is given. The first similarity between the k-th point cloud in the carrier point cloud coordinate system and the j-th point cloud of the i-th bridge pier in the building point cloud coordinate system can be expressed as:

[0063]

[0064] Where Represents the first similarity between the k-th point cloud in the carrier point cloud coordinate system and the j-th point cloud of the i-th pier in the building point cloud coordinate system. Represents the second average distance between the j-th point cloud of the i-th pier in the building point cloud coordinate system and all other point clouds on the i-th pier. Represents the first average distance between the k-th point cloud in the carrier point cloud coordinate system and all other point clouds in the carrier point cloud coordinate system. Represents a preset first adjustment parameter. Represents the absolute value symbol. Represents a normalization function.

[0065] In this embodiment, introducing a preset first adjustment parameter in the calculation formula of the first similarity is to prevent the denominator from being 0. In this embodiment, the preset first adjustment parameter is 0.01. In specific applications, the implementer can set it according to specific circumstances. Used to represent the first difference. The greater the first difference, the lower the similarity between the two point clouds, that is, the smaller the first similarity between the k-th point cloud in the carrier point cloud coordinate system and the j-th point cloud of the i-th pier in the building point cloud coordinate system.

[0066] By adopting the above method, the first similarity between each point cloud in the carrier point cloud coordinate system and each point cloud on the pier in the building point cloud coordinate system can be obtained. The greater the first similarity, the higher the similarity degree between the corresponding two point clouds.

[0067] For any point cloud in the building point cloud coordinate system: If the maximum value of the first similarity between this point cloud and all point clouds in the carrier point cloud coordinate system is greater than the preset first similarity threshold, then the point cloud in the building point cloud coordinate system is used as the matching point of the point cloud in the carrier point cloud coordinate system corresponding to the maximum value. By adopting this method, the point clouds in the building point cloud coordinate system can be matched, and the matching points of each point cloud in the building point cloud coordinate system can be obtained in the carrier point cloud coordinate system. All the matching points in the carrier point cloud coordinate system form the pier in the carrier point cloud coordinate system. In this embodiment, the preset first similarity threshold is 0.8. In specific applications, the implementer can set it according to specific circumstances.

[0068] Further, according to the position of the pier in the carrier point cloud coordinate system and the position of the pier in the carrier point cloud coordinate system, determine the included angle of the pier in the carrier point cloud coordinate system and the pier in the building point cloud coordinate system in the X-axis direction 、the included angle in the Y-axis direction <Q 、the included angle in the Z-axis direction , and then rotate and align the building point cloud coordinate system and the carrier point cloud coordinate system. After rotation and alignment, each same coordinate information in the two coordinate systems represents the same position of the pier. The specific rotation and alignment method is an existing technology and will not be elaborated here.

[0069] Next, combine the distance and angle information feedback by the real-time point cloud data to screen for suspected shooting areas.

[0070] Specifically, in accordance with a preset order, splice the coordinate information of all the real-time point clouds collected by the robot at the current moment in the carrier point cloud coordinate system after rotation alignment to form a first position vector. For example, the coordinate information of the real-time point clouds in the carrier point cloud coordinate system is respectively , , …, , then the first position vector formed by them is , where represents the coordinate of the first real-time point cloud in the carrier point cloud coordinate system, represents the coordinate of the second real-time point cloud in the carrier point cloud coordinate system, represents the coordinate of the nth real-time point cloud in the carrier point cloud coordinate system. Similarly, in accordance with a preset order, splice all the point clouds on each pier in the rotated and aligned building point cloud coordinate system to form a second position vector for each pier. Each pier has a corresponding second position vector. The construction method of the second position vector is similar to that of the first position vector, and will not be elaborated here too much; since the number of all the real-time point clouds collected by the robot at the current moment may not be equal to the number of point clouds on each pier in the building point cloud coordinate system, when the two numbers are not equal, fill the vacant positions with 0 to make the length of the first position vector equal to the length of the second position vector. It should be noted that: the preset order can be from top to bottom, from left to right. In specific applications, the implementer can set according to the specific situation.

[0071] Calculate the cosine similarity between the first position vector and each second position vector respectively, and record this cosine similarity as the second similarity, that is, there is a second similarity between the first position vector and each second position vector. The calculation method of the cosine similarity is a prior art and will not be elaborated here too much.

[0072] The greater the second similarity, the higher the position similarity between the real-time point cloud collected by the robot at the current moment and the point cloud on the corresponding pier in the rotated and aligned building point cloud coordinate system, and the more likely the area where the corresponding pier is located in the building point cloud coordinate system is a suspected shooting area. Based on this feature, next, use the second similarity to screen for suspected shooting areas. Specifically, determine the area where the pier is located when the second similarity is greater than the preset second similarity threshold as the suspected shooting area. In this embodiment, the preset second similarity threshold is 0.6. In specific applications, the implementer sets according to the specific situation.

[0073] So far, using the above method, multiple suspected shooting areas have been screened out.

[0074] Step S3: Filter the real shooting areas from all the suspected shooting areas according to the similarity of the position distribution of the corner points in the image and the corner points in each suspected shooting area.

[0075] Perform corner detection on the image captured by the target bridge detection robot at the current moment through a corner detection algorithm (Harris corner detection algorithm). At the same time, perform corner detection on each suspected shooting area through a corner detection algorithm (Harris corner detection algorithm). The Harris corner detection algorithm is a prior art and will not be elaborated here.

[0076] Splice the coordinate information of all the corner points in the image in a preset order to form a third position vector. Splice all the corner points in each suspected shooting area in a preset order to form a reference point set for each suspected shooting area. Project all the points in each reference point set in the camera shooting direction to obtain the corner points of the projection area of each reference point set. The coordinate information of the corner points of the projection area of each reference point set forms a fourth position vector for each reference point set. It should be noted that the construction methods of the third position vector and the fourth position vector are similar to those of the first position vector and the second position vector, except that the coordinate information of each point is two-dimensional information, and the construction methods of the third position vector and the fourth position vector will not be elaborated here.

[0077] Calculate the cosine similarity between the third position vector and each fourth position vector respectively, and record this cosine similarity as the similarity between the third position vector and each fourth position vector. Then, use this similarity as the matching coefficient between the corner points in the image and the reference point set of each suspected shooting area. Since in this embodiment, the cosine similarity between the third position vector and the fourth position vector is used as the similarity between the two, if the lengths of the third position vector and the fourth position vector are not equal, they are padded with 0 to make the lengths of the supplemented third position vector and the fourth position vector equal.

[0078] By using the above method, the matching coefficient between the corner points in the image and the reference point set of each suspected shooting area can be obtained. The larger the matching coefficient, the more likely the corresponding suspected shooting area is the real shooting area. Therefore, next, use the matching coefficient to filter the real shooting areas from all the suspected shooting areas. Specifically, determine the suspected shooting area corresponding to the maximum value of the matching coefficient as the real shooting area.

[0079] Step S4: Locate the robot according to the position distribution of the point cloud in the real shooting area and the orientation of the robot.

[0080] In this embodiment, the actual shooting area of the camera is determined in step S4. Next, the robot will be accurately positioned based on the position distribution of the point cloud in the actual shooting area and the current orientation of the robot.

[0081] Since the data collected by the lidar is affected by the underwater environment, resulting in certain distortion of the data, it is impossible to determine the position of the robot through the intersection of the polar coordinates of all point cloud data. Therefore, the actual shooting area is determined through the polar coordinates of all point clouds, thereby narrowing the area for robot positioning. Place the robot at the position of each point cloud for direction comparison. If the direction vector of the shooting angle of the robot has the highest correlation with the shooting angle direction of a certain point cloud position, then the above-mentioned image is most likely taken at this position, and the relative position of the point cloud is the position of the robot. It should be noted that: the origin of the polar coordinates is the central position of the image captured by the robot.

[0082] Specifically, the radial distance, polar angle, and azimuth angle of each point cloud in the actual shooting area are obtained respectively; among them, the radial distance is the distance from the point cloud to the sensor, the polar angle is the angle with the positive Z-axis, and the azimuth angle is the angle between the projection on the XOY plane and the positive X-axis.

[0083] Take the minimum value of the radial distances of all point clouds in the actual shooting area as the lower limit value of the radius range of the suspected area, and take the maximum value of the radial distances of all point clouds in the actual shooting area as the upper limit value of the radius range of the suspected area, thereby obtaining the radius range of the suspected area of the robot.

[0084] Take the minimum value of the polar angles of all point clouds in the actual shooting area as the lower limit value of the polar angle range of the suspected area, and take the maximum value of the polar angles of all point clouds in the actual shooting area as the upper limit value of the polar angle range of the suspected area, thereby obtaining the polar angle range of the suspected area of the robot.

[0085] Take the minimum value of the azimuth angles of all point clouds in the actual shooting area as the lower limit value of the azimuth angle range of the suspected area, and take the maximum value of the azimuth angles of all point clouds in the actual shooting area as the upper limit value of the azimuth angle range of the suspected area, thereby obtaining the azimuth angle range of the suspected area of the robot.

[0086] Take the origin of the coordinate system as the center point of the suspected area of the robot, and determine the suspected area of the robot according to the radius range, the polar angle range, and the azimuth angle range.

[0087] Further, the real-time pose of the robot can be calculated based on the real-time data collected by the inertial measurement unit, where the pose mainly includes the current orientation of the robot, that is, the direction of the robot. Obtain the vector pointing from each point cloud in the suspected area where the robot is located to the polar coordinate origin as the feature vector of each point cloud in the suspected area where the robot is located. Calculate the cosine similarity between the unit vector of the robot's direction and the feature vector of each point cloud; determine the point cloud corresponding to the maximum cosine similarity as the position of the robot.

[0088] Thus, the accurate positioning of the bridge inspection robot is completed by using the method provided in this embodiment.

[0089] In this embodiment, the point cloud data of the target bridge and the point cloud data of the objects in the environment where the target bridge is located are collected by different sensors. The building point cloud coordinate system is constructed according to the point cloud data of the target bridge, and the carrier point cloud coordinate system is constructed according to the point cloud data of the objects in the environment where the target bridge is located. The building point cloud coordinate system only contains the point cloud data of the structure of the target bridge itself, while the carrier point cloud coordinate system contains the point cloud data of other sundries in the environment where the target bridge is located in addition to the point cloud data of the structure of the target bridge itself. It is necessary to perform an alignment operation on the positions in the two coordinate systems. According to the similarity between the positions of all the real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each pier of the target bridge, the suspected shooting areas are screened. Evaluate the similarity of the position distribution of the corner points in the images captured by the inspection robot and the corner points in each suspected shooting area, and screen the real shooting areas from all the suspected shooting areas, and then accurately locate the robot in combination with the orientation of the robot. The method provided in this embodiment fuses the data information collected by multiple sensors, so as to complement the data between different sensors in dusty, dim and other environments, avoid the situation that the positioning is inaccurate due to environmental influence of a single sensor, ensure the anti-interference ability of the positioning and improve the positioning accuracy of the bridge inspection robot.

[0090] An embodiment of a multi-sensor information fusion bridge inspection robot positioning system:

[0091] Refer to Figure 2 , which shows the structural block diagram of a multi-sensor information fusion bridge inspection robot positioning system provided by an embodiment of the present invention. The system includes a data acquisition module, a first screening module, a second screening module, and a positioning module;

[0092] Among them, the data acquisition module is used to obtain the point cloud data of the target bridge, the point cloud data of the objects in the environment where the target bridge is located, and the images captured by the target bridge inspection robot; construct a building point cloud coordinate system according to the point cloud data of the target bridge, and construct a carrier point cloud coordinate system according to the point cloud data of the objects in the environment where the target bridge is located;

[0093] The first screening module is used to rotate and align the building point cloud coordinate system and the carrier point cloud coordinate system according to the similarity between the relative distances of each point cloud in the carrier point cloud coordinate system and other point clouds and the relative distances of each point cloud of the bridge pier in the building point cloud coordinate system and other point clouds; and screen the suspected shooting areas according to the similarity between the positions of all the real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each bridge pier of the target bridge.

[0094] The second screening module is used to screen the real shooting areas from all the suspected shooting areas according to the similarity of the position distribution of the corner points in the image and the corner points in each suspected shooting area.

[0095] The positioning module is used to position the robot according to the position distribution of the point clouds in the real shooting area and the orientation of the robot.

[0096] It should be understood that Figure 2 The structural block diagram and modules of a bridge detection robot positioning system for multi-sensor information fusion shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented through hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art can understand that the above methods and systems can be implemented using computer-executable instructions and / or included in the processor control code, for example, such code is provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of this specification can not only be implemented by hardware circuits of programmable hardware devices such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips and transistors, or field programmable gate arrays and programmable logic devices, but also be implemented by software executed by various types of processors, or be implemented by a combination of the above hardware circuits and software (for example, firmware).

[0097] For more details about the above various modules, reference can be made to other positions in this specification, and details will not be elaborated here.

[0098] In other embodiments, a medium is further provided, and the medium stores at least one program that can be run by a computer. When the at least one program is run by the computer, the computer is made to execute the steps in the bridge detection robot positioning method for multi-sensor information fusion in the above embodiments, and the medium can be a computer-readable storage medium.

[0099] Among them, the provided system and medium are both used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0100] It should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A positioning method for a bridge inspection robot with multi-sensor information fusion, characterized in that, The method includes the following steps: Obtain the point cloud data of the target bridge, the point cloud data of the objects in the environment where the target bridge is located, and the images captured by the target bridge detection robot; construct a building point cloud coordinate system based on the point cloud data of the target bridge, and construct a carrier point cloud coordinate system based on the point cloud data of the objects in the environment where the target bridge is located; Rotate and align the building point cloud coordinate system and the carrier point cloud coordinate system according to the similarity between the relative distances of each point cloud in the carrier point cloud coordinate system from other point clouds and the relative distances of each point cloud of the bridge pier in the building point cloud coordinate system from other point clouds; screen the suspected shooting areas according to the similarity between the positions of all the real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each bridge pier of the target bridge; Screen the real shooting areas from all the suspected shooting areas according to the similarity of the position distribution of the corner points in the image and the corner points in each suspected shooting area; Locate the robot according to the position distribution of the point clouds in the real shooting area and the orientation of the robot.

2. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 1, characterized in that, The step of rotating and aligning the building point cloud coordinate system and the carrier point cloud coordinate system according to the similarity between the relative distances of each point cloud in the carrier point cloud coordinate system from other point clouds and the relative distances of each point cloud of the bridge pier in the building point cloud coordinate system from other point clouds includes: Calculate the first average distance between each point cloud in the carrier point cloud coordinate system and all other point clouds in the carrier point cloud coordinate system respectively; For any point cloud on any bridge pier in the building point cloud coordinate system: calculate the second average distance between any point cloud on any bridge pier and all other point clouds on the same bridge pier; obtain the first similarity between each point cloud in the carrier point cloud coordinate system and any point cloud on any bridge pier according to the first difference between the first average distance and the second average distance, and the first difference and the first similarity are negatively correlated; Match the point clouds in the carrier point cloud coordinate system with the point clouds on any bridge pier of the target bridge according to the magnitude relationship of the first similarity between each point cloud in the carrier point cloud coordinate system and all the point clouds on any bridge pier, and determine the bridge pier in the carrier point cloud coordinate system; Rotate and align the building point cloud coordinate system and the carrier point cloud coordinate system according to the positions of the bridge piers in the carrier point cloud coordinate system and the positions of the bridge piers in the building point cloud coordinate system.

3. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 2, characterized in that The step of matching the point clouds in the carrier point cloud coordinate system with the point clouds on any bridge pier of the target bridge according to the magnitude relationship of the first similarity between each point cloud in the carrier point cloud coordinate system and all the point clouds on any bridge pier, and determining the bridge pier in the carrier point cloud coordinate system includes: For any point cloud in the building point cloud coordinate system: if the maximum value of the first similarity between any point cloud in the building point cloud coordinate system and all the point clouds in the carrier point cloud coordinate system is greater than a preset first similarity threshold, then use any point cloud in the building point cloud coordinate system as the matching point of the point cloud in the carrier point cloud coordinate system corresponding to the maximum value; All the matching points in the carrier point cloud coordinate system form the bridge pier in the carrier point cloud coordinate system.

4. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 1, characterized in that, Screening the suspected shooting areas according to the similarity between the positions of all real-time point clouds collected by the robot at the current moment in the rotated and aligned coordinate system and the positions of the point clouds of each pier of the target bridge, including: The coordinate information of all real-time point clouds collected by the robot at the current moment in the carrier point cloud coordinate system after rotation and alignment constitutes the first position vector; all the point clouds on each pier in the building point cloud coordinate system after rotation and alignment constitute the second position vector of each pier; calculate the second similarity between the first position vector and each second position vector; Use the second similarity to screen the suspected shooting areas.

5. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 4, characterized in that, The using the second similarity to screen the suspected shooting areas includes: Determine the area where the pier is located when the second similarity is greater than the preset second similarity threshold as the suspected shooting area.

6. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 1, characterized in that, Screening the real shooting areas from all the suspected shooting areas according to the similarity of the position distribution of the corner points in the image and the corner points in each suspected shooting area, including: The coordinate information of all the corner points in the image constitutes the third position vector; All the corner points in each suspected shooting area constitute the reference point set of each suspected shooting area; project all the points in each reference point set in the camera shooting direction to obtain the corner points of the projection area of each reference point set, and the coordinate information of the corner points of the projection area of each reference point set constitutes the fourth position vector of each reference point set; Take the similarity between the third position vector and each fourth position vector as the matching coefficient between the corner points in the image and the reference point set of each suspected shooting area; Use the matching coefficient to screen the real shooting areas.

7. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 6, characterized in that The using the matching coefficient to screen the real shooting areas includes: Determine the suspected shooting area corresponding to the maximum value of the matching coefficient as the real shooting area.

8. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 1, characterized in that, Positioning the robot according to the position distribution of the point clouds in the real shooting area and the orientation of the robot, including: Obtain the radial distance, polar angle, and azimuth angle of each point cloud in the real shooting area respectively; where the radial distance is the distance from the point cloud to the sensor; Based on the minimum and maximum values of the radial distances, the minimum and maximum values of the polar angles, and the minimum and maximum values of the azimuth angles of all the point clouds in the real shooting area, determine the suspected area where the robot is located; Position the robot according to the position distribution of the point clouds in the suspected area where the robot is located and the unit vector of the robot direction.

9. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 8, characterized in that, The positioning the robot according to the position distribution of the point clouds in the suspected area where the robot is located and the unit vector of the robot direction includes: Obtain the vector pointing from each point cloud in the suspected area where the robot is located to the polar coordinate origin as the feature vector of each point cloud; Calculate the cosine similarity between the unit vector of the robot direction and the feature vector of each point cloud; determine the point cloud corresponding to the maximum value of the cosine similarity as the position of the robot.

10. A positioning method for a bridge inspection robot with multi-sensor information fusion according to claim 8, characterized in that, The based on the minimum and maximum values of the radial distances, the minimum and maximum values of the polar angles, and the minimum and maximum values of the azimuth angles of all the point clouds in the real shooting area, determining the suspected area where the robot is located, includes: Obtain the radius range of the suspected area where the robot is located based on the minimum and maximum values of the radial distances of all the point clouds within the real shooting area; Obtain the polar angle range of the suspected area where the robot is located based on the minimum and maximum values of the polar angles of all the point clouds within the real shooting area; Obtain the azimuth angle range of the suspected area where the robot is located based on the minimum and maximum values of the azimuth angles of all the point clouds within the real shooting area; Determine the suspected area where the robot is located according to the radius range, the polar angle range, and the azimuth angle range.

Images