Seamless positioning method and device for bridge full-scene cruise

Abstract

The invention relates to the technical field of positioning, in particular to a seamless positioning method and device for bridge full-scene cruise, and the method comprises the steps: obtaining satellite measurement data of a global navigation satellite system and inertial measurement data of an inertial measurement unit when a signal of the global navigation satellite system is detected; obtaining a first cruise track of the bridge cruise device by using the satellite measurement data and the inertia measurement data; when the signal of the global navigation satellite system is not detected, acquiring visual image data of the camera, laser radar data of the laser radar and inertial measurement data of the inertial measurement unit; obtaining a pre-constructed factor graph and a factor graph optimization model; performing incremental updating on the factor graph by using the visual image data, the laser radar data, the inertial measurement data and the factor graph optimization model to obtain a second cruise trajectory of the bridge cruise device; and fusing the first cruise track and the second cruise track to obtain the cruise track of the bridge cruise device.

Description

Technical Field

[0001] The present disclosure relates to the field of positioning technology, and in particular to a seamless positioning method and device for full-scene cruising of a bridge. Background Art

[0002] Positioning is the key in the control closed loop that controls the drone to fly according to the planned trajectory.

[0003] Among the related technologies, drone positioning mainly relies on the Global Navigation Satellite System (GNSS), which can adapt to most application scenarios. However, for some key shielded or enclosed spaces, such as the bottom of bridges and other key areas, satellite signals are blocked and positioning cannot be achieved, resulting in flight loopholes. Summary of the invention

[0004] The present invention provides a seamless positioning method and device for full-scene cruising on a bridge to solve the problems existing in the related art.

[0005] In a first aspect, the present disclosure provides a seamless positioning method for full-scene cruise of a bridge, which is applied to a seamless positioning system for full-scene cruise of a bridge, wherein the seamless positioning system includes a global navigation satellite system and an inertial measurement unit, a camera, and a laser radar installed on a bridge cruiser; the method includes:

[0006] When the signal of the global navigation satellite system is detected, satellite measurement data of the global navigation satellite system and inertial measurement data of the inertial measurement unit are acquired; and a first cruise trajectory of the bridge cruiser is obtained by using the satellite measurement data and the inertial measurement data;

[0007] When the signal of the global navigation satellite system is not detected, the visual image data of the camera, the lidar data of the lidar and the inertial measurement data of the inertial measurement unit are obtained; and a pre-constructed factor graph and a factor graph optimization model are obtained; wherein the factors of the factor graph include an inertial odometry factor, a laser inertial odometry factor and a visual inertial odometry factor;

[0008] Using the visual image data, the laser radar data, the inertial measurement data and the factor graph optimization model, incrementally updating the factor graph to obtain a second cruise trajectory of the bridge cruiser;

[0009] The first cruise trajectory and the second cruise trajectory are merged to obtain the cruise trajectory of the bridge cruiser.

[0010] In some embodiments, the method further comprises:

[0011] defining the inertial odometry factor, the laser inertial odometry factor and the visual inertial odometry factor;

[0012] Determining a constraint relationship between the inertial odometry factor, the laser inertial odometry factor, and the visual inertial odometry factor;

[0013] The factor graph is constructed using the constraint relationship.

[0014] In some embodiments, the inertial odometry factor is defined as:

[0015]

[0016] in, represents the residual between the i-th and j-th inertial measurement unit observations, represents the measurement value observed by the inertial measurement unit, represents the predicted value of the inertial measurement unit observation;

[0017] The laser inertial odometry factor is defined as:

[0018]

[0019] Among them, e imu represents the inertial odometer factor, represents the predicted pose constraint provided by the inertial odometer, W imu Represents the weight matrix corresponding to the inertial measurement unit, W l represents the weight matrix corresponding to the laser radar, Represents the distance between point and point, point and line, and point and surface;

[0020] The visual inertial odometry factor is defined as:

[0021]

[0022] Among them, e reproj represents the visual reprojection factor, e imu represents the inertial odometer factor, E p represents the marginalized prior residual, represents the pose prior from the inertial odometry, obs(i) represents the set of visual features i tracked in other frames, and the set C contains the visual node pairs (a, b) connected by the inertial odometry factor. reproj represents the covariance matrix of the visual reprojection factor, W imu Represents the covariance matrix of the inertial odometry factors.

[0023] In some embodiments, the factor graph optimization model is represented as:

[0024]

[0025] in, represents the inertial re-integrated residual calculated by the laser inertial odometer, represents the inertial re-integrated residual calculated by the visual inertial odometry, represents the weight coefficient, E p represents the marginalized prior residual.

[0026] In some embodiments, the factors of the factor graph further include pre-constructed measurement models of the global navigation satellite system and the inertial measurement unit; wherein the measurement model is used to characterize the relationship between the observation value of the global navigation satellite system and the navigation state of the inertial measurement unit; and the method further includes:

[0027] Acquire the pre-built measurement model;

[0028] The inertial measurement unit is calibrated using the measurement model and the satellite measurement data.

[0029] In some embodiments, the measurement model is represented as:

[0030]

[0031] in, represents the residuals of GNSS observations, represents the measurement value observed by the global navigation satellite system, h i (x i ) represents the predicted value from GNSS observations.

[0032] In a second aspect, the present disclosure provides a seamless positioning device for full-scene cruise on a bridge, which is applied to a seamless positioning system for full-scene cruise on a bridge. The seamless positioning system includes a global navigation satellite system and an inertial measurement unit, a camera, and a laser radar installed on a bridge cruiser; the device includes:

[0033] an acquisition module, configured to acquire satellite measurement data of the global navigation satellite system and inertial measurement data of the inertial measurement unit when a signal of the global navigation satellite system is detected;

[0034] A processing module, used for obtaining a first cruise trajectory of the bridge cruiser by using the satellite measurement data and the inertial measurement data;

[0035] The acquisition module is also used to acquire visual image data of the camera, laser radar data of the laser radar, and inertial measurement data of the inertial measurement unit when the signal of the global navigation satellite system is not detected; and acquire a pre-constructed factor graph and a factor graph optimization model; wherein the factors of the factor graph include an inertial odometry factor, a laser inertial odometry factor, and a visual inertial odometry factor;

[0036] The processing module is also used to incrementally update the factor graph using the visual image data, the laser radar data, the inertial measurement data and the factor graph optimization model to obtain a second cruise trajectory of the bridge cruiser;

[0037] The processing module is further used to merge the first cruise trajectory and the second cruise trajectory to obtain the cruise trajectory of the bridge cruiser.

[0038] In a third aspect, the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above aspects.

[0039] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the above aspects when executed by a processor.

[0040] In a fifth aspect, the present disclosure provides a computer program product, including a computer program / instructions, which implements the steps of the method described in the above aspects when the computer program is executed by a processor.

[0041] The present invention provides a seamless positioning method and device for full-scene cruise on a bridge. When a signal of a global navigation satellite system is detected, satellite measurement data of a global navigation satellite system and inertial measurement data of an inertial measurement unit are obtained; and the first cruise trajectory of a bridge cruiser is obtained by using the satellite measurement data and the inertial measurement data; when no signal of the global navigation satellite system is detected, visual image data of a camera, laser radar data of a laser radar and inertial measurement data of an inertial measurement unit are obtained; and a pre-constructed factor graph and a factor graph optimization model are obtained; wherein the factors of the factor graph include an inertial odometer factor, a laser inertial odometer factor and a visual inertial odometer factor; and the first cruise trajectory of a bridge cruiser is obtained by using the satellite measurement data and the inertial measurement data. Visual image data, lidar data, inertial measurement data and factor graph optimization model are used to incrementally update the factor graph to obtain the second cruise trajectory of the bridge cruiser; the first cruise trajectory and the second cruise trajectory are fused to obtain the cruise trajectory of the bridge cruiser, which can achieve absolute positioning and attitude determination in the external space of the bridge through the combination of global navigation satellite system and inertial measurement unit when there is a signal from the global navigation satellite system; when there is no signal from the global navigation satellite system, high-precision relative positioning of the bottom of the bridge is achieved through the fusion of inertial measurement unit, camera and lidar, and the results of absolute positioning and attitude determination are fused with the results of relative positioning to achieve seamless positioning of the entire bridge inspection scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present disclosure will be described in more detail below based on embodiments and with reference to the accompanying drawings:

[0043] Figure 1 A seamless positioning system for full-scene cruising on a bridge provided by an embodiment of the present disclosure;

[0044] Figure 2 A schematic flow chart of a seamless positioning method for full-scenario cruising on a bridge provided by an embodiment of the present disclosure;

[0045] Figure 3 A technical flow chart of a seamless positioning method for full-scene cruising on a bridge provided by an embodiment of the present disclosure;

[0046] Figure 4 A schematic structural diagram of a seamless positioning device for full-scene cruising on a bridge provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0049] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] Seamless positioning technology for the entire bridge scene is the key to solving bridge inspection and positioning problems.

[0051] Among the related technologies, ultra-wideband technology (UWB) and pseudo-satellite positioning technology both achieve UAV positioning through radio ranging intersection. This positioning method requires the deployment of a certain number and spatial distribution of base stations around the bridge in advance.

[0052] By shooting the target sphere mounted on the drone with a fixed array camera, the drone can be accurately positioned through intersection measurement, thereby controlling the movement of the drone for indoor positioning, but it is not suitable for large outdoor scenes.

[0053] Laser tracking is used to track and measure the omnidirectional reflection prism installed on the drone, and the position information is transmitted to the drone, so as to achieve high-precision positioning of the drone. However, this method may have occlusion problems.

[0054] Autonomous positioning can be achieved by fusing vision, laser and inertial information for bridge positioning, but there is a drift problem for a long time.

[0055] Relative positioning can be achieved by using a camera installed on the drone to shoot the control point marks on the ground station, which has good flexibility. However, there are problems with the system complexity and low reliability.

[0056] In summary, these UAV positioning methods cannot simultaneously meet the accuracy, efficiency and cost requirements of bridge inspection UAV positioning. They solve the UAV positioning problem in some scenarios, but it is still difficult to achieve positioning in the entire bridge inspection scenario.

[0057] Based on this, the embodiments of the present disclosure provide a seamless positioning method and device for full-scene cruising of a bridge, which can achieve absolute positioning and attitude determination in the external space of the bridge through a combination of a satellite navigation positioning system and an inertial measurement unit (IMU) when there is a satellite signal; when there is no satellite signal, high-precision relative positioning of the bottom of the bridge is achieved through the fusion of the inertial measurement unit, camera and lidar, and the results of absolute positioning and attitude determination are fused with the results of relative positioning to achieve seamless positioning of the entire bridge inspection scene.

[0058] Embodiment 1

[0059] The embodiment of the present disclosure provides a seamless positioning method for full-scene cruising on a bridge, which is applied to a seamless positioning system for full-scene cruising on a bridge. Figure 1 A seamless positioning system for full-scene cruising on a bridge is provided in an embodiment of the present disclosure. Figure 1 As shown, the seamless positioning system includes a global navigation satellite system 101 and an inertial measurement unit 102, a camera 103 and a laser radar 104 installed on the bridge cruiser.

[0060] Figure 2 The flowchart of a seamless positioning method for full-scene cruising of a bridge provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, a seamless positioning method for full-scenario cruising of a bridge includes:

[0061] S201, when a signal of a global navigation satellite system is detected, obtaining satellite measurement data of the global navigation satellite system and inertial measurement data of an inertial measurement unit; and obtaining a first cruise trajectory of the bridge cruiser by using the satellite measurement data and the inertial measurement data;

[0062] S202, when no signal of the global navigation satellite system is detected, obtaining visual image data of the camera, laser radar data of the laser radar, and inertial measurement data of the inertial measurement unit; and obtaining a pre-built factor graph and a factor graph optimization model; wherein the factors of the factor graph include an inertial odometry factor, a laser inertial odometry factor, and a visual inertial odometry factor;

[0063] S203, using visual image data, laser radar data, inertial measurement data and the factor graph optimization model, incrementally updating the factor graph to obtain a second cruise trajectory of the bridge cruiser;

[0064] S204: The first cruise trajectory and the second cruise trajectory are merged to obtain a cruise trajectory of the bridge cruiser.

[0065] Specifically, the above-mentioned bridge cruiser can be a cruising tool for performing full-scene cruising of the bridge. The cruising tool can be a drone or other tools, and the embodiments of the present disclosure do not specifically limit this.

[0066] During the full-scene cruising of the bridge, the bridge cruiser can receive the global navigation satellite system signal in some parts of the bridge, such as the external space of the bridge, but may not receive the global navigation satellite system signal in some parts of the bridge, such as the bottom of the bridge, resulting in the problem of inability to achieve seamless positioning in the parts of the bridge that cannot receive the global navigation satellite system signal.

[0067] Based on this, when the signal of the global navigation satellite system is detected, the satellite measurement data of the global navigation satellite system and the inertial measurement data of the inertial measurement unit are obtained; and the first cruise track of the bridge cruiser is obtained by using the satellite measurement data and the inertial measurement data. Here, the first cruise track can be understood as the absolute positioning of the bridge cruiser.

[0068] When no signal from the global navigation satellite system is detected, the visual image data of the camera, the lidar data of the lidar, and the inertial measurement data of the inertial measurement unit are obtained; and a pre-built factor graph and a factor graph optimization model are obtained. Among them, the factors of the factor graph include inertial odometry factors, laser inertial odometry factors, and visual inertial odometry factors;

[0069] Then, the factor graph is incrementally updated using visual image data, lidar data, inertial measurement data and the factor graph optimization model to obtain the second cruise trajectory of the bridge cruiser. The first cruise trajectory and the second cruise trajectory are fused to obtain the cruise trajectory of the bridge cruiser.

[0070] According to the technical solution of the embodiment of the present disclosure, when the signal of the global navigation satellite system is detected, the satellite measurement data of the global navigation satellite system and the inertial measurement data of the inertial measurement unit are obtained; and the first cruise trajectory of the bridge cruiser is obtained by using the satellite measurement data and the inertial measurement data; when the signal of the global navigation satellite system is not detected, the visual image data of the camera, the lidar data of the lidar and the inertial measurement data of the inertial measurement unit are obtained; and a pre-constructed factor graph and a factor graph optimization model are obtained; wherein the factors of the factor graph include an inertial odometer factor, a laser inertial odometer factor and a visual inertial odometer factor; and the visual image data are used to obtain the first cruise trajectory of the bridge cruiser. , lidar data, inertial measurement data and factor graph optimization model, incrementally update the factor graph to obtain the second cruise trajectory of the bridge cruiser; fuse the first cruise trajectory and the second cruise trajectory to obtain the cruise trajectory of the bridge cruiser, which can achieve absolute positioning and attitude determination in the external space of the bridge through the combination of global navigation satellite system and inertial measurement unit when there is a signal from the global navigation satellite system; when there is no signal from the global navigation satellite system, high-precision relative positioning of the bottom of the bridge can be achieved through the fusion of inertial measurement unit, camera and lidar, and the results of absolute positioning and attitude determination are fused with the results of relative positioning to achieve seamless positioning of the entire scene of bridge inspection.

[0071] Embodiment 2

[0072] Based on the above embodiment, the method may further include:

[0073] Define inertial odometry factor, laser inertial odometry factor and visual inertial odometry factor;

[0074] Determine the constraint relationship between the inertial odometry factor, the laser inertial odometry factor and the visual inertial odometry factor;

[0075] Construct factor graphs using constraint relationships.

[0076] Specifically, in factor graph optimization, each node in the pose graph is associated with a pose state. The edge between two consecutive nodes represents the relative carrier motion obtained by the inertial measurement unit pre-integration method. Based on this, the disclosed embodiment can use the residual between two inertial measurement unit observations, the measured value of the inertial measurement unit observation, and the predicted value of the inertial measurement unit observation to define the inertial odometer factor.

[0077] When a new frame of lidar data is received, feature extraction will be performed on the frame of lidar data, and the point, line and plane features will be extracted using the principal component analysis method. After extracting reliable geometric features, the motion provided by the inertial odometer is used for prediction, and the dynamic octree is used to find the correspondence between its points, lines and planes in the map. The optimal relative pose transformation is solved by minimizing the distance between point clouds of different frames, thereby defining the laser inertial odometer factor.

[0078] For each new keyframe in the visual image data, a nonlinear optimization problem consisting of the visual reprojection factor, the inertial odometry factor, the marginalized prior residual, and the pose prior from the inertial odometry is minimized to define the visual inertial odometry factor.

[0079] Then, the constraint relationship among the inertial odometry factor, the laser inertial odometry factor and the visual inertial odometry factor is determined; and a factor graph is constructed using the constraint relationship.

[0080] Embodiment 3

[0081] Based on the above embodiment, the inertial odometer factor is defined as:

[0082]

[0083] in, represents the residual between the i-th and j-th inertial measurement unit observations, represents the measurement value observed by the inertial measurement unit, represents the predicted value of the inertial measurement unit observation;

[0084] The laser inertial odometry factor is defined as:

[0085]

[0086] Among them, e imu represents the inertial odometer factor, represents the predicted pose constraint provided by the inertial odometer, W imu Represents the weight matrix corresponding to the inertial measurement unit, W l represents the weight matrix corresponding to the laser radar, Represents the distance between point and point, point and line, and point and surface;

[0087] The visual inertial odometry factor is defined as:

[0088]

[0089] Among them, e reproj represents the visual reprojection factor, e imu represents the inertial odometer factor, Ep represents the marginalized prior residual, represents the pose prior from the inertial odometry, obs(i) represents the set of visual features i tracked in other frames, and the set C contains the visual node pairs (a, b) connected by the inertial odometry factor. reproj represents the covariance matrix of the visual reprojection factor, W imu Represents the covariance matrix of the inertial odometry factors.

[0090] Embodiment 4

[0091] Based on the above embodiment, the factor graph optimization model is expressed as:

[0092]

[0093] in, represents the inertial re-integrated residual calculated by the laser inertial odometer, represents the inertial re-integrated residual calculated by the visual inertial odometry, represents the weight coefficient, E p represents the marginalized prior residual.

[0094] Embodiment 5

[0095] On the basis of the above embodiment, the factors of the factor graph further include pre-constructed measurement models of the global navigation satellite system and the inertial measurement unit; wherein the measurement model is used to characterize the relationship between the observation value of the global navigation satellite system and the navigation state of the inertial measurement unit; the method further includes:

[0096] Get pre-built measurement models;

[0097] The inertial measurement unit is calibrated using the measurement model and satellite measurement data.

[0098] Specifically, inertial odometer is a technology that uses an inertial measurement unit to measure the displacement, velocity, and posture of an object, thereby estimating and recording the object's trajectory. Inertial odometer is mainly based on Newton's laws of mechanics and the principle of inertia. Inertial measurement units usually contain accelerometers and gyroscopes. Accelerometers can measure the acceleration of an object in three axes, and gyroscopes can measure the angular velocity of an object in three axes.

[0099] By integrating the acceleration measured by the accelerometer, the velocity of the object can be obtained; by integrating the velocity, the displacement of the object can be obtained. At the same time, the angular velocity measured by the gyroscope can be used to update and estimate the attitude of the object.

[0100] In a short period of time, the inertial odometer can obtain high-precision displacement, velocity, and attitude estimation by accurately measuring and integrating acceleration and angular velocity. Especially in high dynamic environments, its short-term accuracy advantage is more obvious.

[0101] Since inertial measurement is based on integration, measurement errors will accumulate over time, resulting in long-term accuracy degradation. During long-term motion, even if the initial error is small, after a long period of integration, the errors in displacement, velocity, and attitude will become large, thus affecting the accuracy of the inertial odometer.

[0102] The factors of the factor graph in the disclosed embodiment may also include pre-built measurement models of the global navigation satellite system and the inertial measurement unit, wherein the measurement model is used to characterize the relationship between the observation value of the global navigation satellite system and the navigation state of the inertial measurement unit. In order to improve the performance of the inertial odometer, the disclosed embodiment may use the measurement model and satellite measurement data to calibrate the inertial measurement unit.

[0103] Embodiment 6

[0104] Based on the above embodiment, the measurement model is expressed as:

[0105]

[0106] in, represents the residuals of GNSS observations, represents the measurement value observed by the global navigation satellite system, h i (x i ) represents the predicted value from GNSS observations.

[0107] Embodiment 7

[0108] Based on the above embodiments, this embodiment provides an application example.

[0109] Figure 3 This is a technical flow chart of a seamless positioning method for full-scene cruising on a bridge provided by an embodiment of the present disclosure. Figure 3 As shown in the figure, in the bridge inspection scenario, the UAV seamless positioning system performs positioning in the following two modes according to whether it can receive the signal of the global navigation satellite system: the first positioning method: when the signal of the global navigation satellite system can be received, all sensor data are integrated for positioning, and the global navigation satellite system-assisted inertial measurement unit positioning method is mainly used; the second positioning method: when the signal of the global navigation satellite system cannot be received, the camera and lidar are used to assist the inertial measurement unit positioning method.

[0110] In the second positioning method, the positioning module consists of three parts: inertial odometer, visual inertial odometer and lidar inertial odometer. The inertial measurement unit provides posture priors to assist the lidar in pre-registration and the camera in image matching, thereby improving the reliability of motion estimation; conversely, the motion estimation feedback of the camera and lidar corrects the error of the inertial measurement unit, thereby restoring motion in a bidirectional iterative manner.

[0111] Specifically, the disclosed embodiment adopts a factor graph framework to perform fusion optimization on the data of the global navigation satellite system, inertial measurement unit, camera, and lidar, uses the results of the visual inertial odometry and laser inertial odometry as factors, performs incremental factor graph optimization on the inertial odometry, and finally outputs the optimal pose estimate from the inertial odometry.

[0112] The disclosed embodiment uses a drone equipped with multi-source sensors (global navigation satellite system, inertial measurement unit, camera, lidar) to obtain measurement data, and adopts a multi-source fusion positioning method to provide high-precision positioning services for drones working at the bottom of the bridge.

[0113] When performing multi-source fusion positioning, factor graph optimization factors are first constructed according to different measurement values, and then factor graph optimization is performed. Specifically, inertial odometry factors, global navigation satellite system factors, laser inertial odometry factors, and visual inertial odometry factors are constructed.

[0114] (1) Inertial odometer factor

[0115] In factor graph optimization, each node in the pose graph is associated with a pose state. The edge between two consecutive nodes represents the relative carrier motion obtained by the inertial measurement unit pre-integration (also known as IMU pre-integration) method. The inertial odometry factor can be defined as:

[0116]

[0117] in, represents the residual between the i-th and j-th inertial measurement unit observations, represents the measurement value observed by the inertial measurement unit, Represents the predicted value of the inertial measurement unit observation.

[0118] (2) Global Navigation Satellite System Factor

[0119] When the satellite signal of the global navigation satellite system can be received, the error of the inertial measurement unit can be updated according to the measurement model between the measurement value of the global navigation satellite system and the navigation state of the inertial measurement unit. The above measurement model can be expressed as:

[0120]

[0121] in, represents the residuals of GNSS observations, represents the measurement value observed by the global navigation satellite system, h i (x i ) represents the predicted value from GNSS observations.

[0122] (3) Laser inertial odometry factor

[0123] When a new LiDAR scan is received, feature extraction is performed, using principal component analysis to extract point, line, and plane features. After extracting reliable geometric features, the motion provided by the inertial odometer is used for prediction, and the dynamic octree is used to find its point, line, and plane correspondence in the map. The optimal relative pose transformation is solved by minimizing the distance between point clouds in different frames:

[0124]

[0125] Among them, e imu represents the inertial odometer factor, represents the predicted pose constraint provided by the inertial odometer, W imu Represents the weight matrix corresponding to the inertial measurement unit, W l represents the weight matrix corresponding to the laser radar, Represents the distance between point and point, point and line, and point and area.

[0126] (4) Visual inertial odometry factor

[0127] For each new keyframe, a nonlinear optimization problem consisting of the visual reprojection factor, the inertial odometry factor, the marginalized prior residual, and the pose prior from the inertial odometry is minimized.

[0128]

[0129] Among them, e reproj represents the visual reprojection factor, e imu represents the inertial odometer factor, E p represents the marginalized prior residual, represents the pose prior from the inertial odometry, obs(i) represents the set of visual features i tracked in other frames, and the set C contains the visual node pairs (a, b) connected by the inertial odometry factor. reproj represents the covariance matrix of the visual reprojection factor, W imu Represents the covariance matrix of the inertial odometry factors.

[0130] When the environment visually degrades, The visual factor will dominate the nonlinear optimization problem, and unreliable visual factors will be rejected by analyzing the information matrix. However, when the environment has good lighting conditions, the visual factor will dominate the nonlinear optimization problem, and the inertial odometry factor only provides an initial guess for visual feature tracking.

[0131] After the multi-source data factor graph is constructed, the factor graph optimization framework is used to perform incremental updates to obtain the real-time flight trajectory.

[0132] It is important to note that since both the visual inertial odometry and the laser inertial odometry are used to constrain the pre-integrated measurements of the IMU, multiple inertial re-integrated residuals are obtained simultaneously. These inertial re-integrated residuals must be weighted with an appropriate covariance matrix, which can be calculated based on the reliability of the observations.

[0133] For example, in a visually degraded environment, the inertia re-integrated residual computed by the visual inertial odometry will have a lower weight. In contrast, in a geometrically degraded environment, the inertia re-integrated residual computed by the laser inertial odometry will have a lower weight. For each new keyframe, a factor graph optimization problem consisting of the inertia re-integrated residual computed by the laser inertial odometry, the inertia re-integrated residual computed by the visual inertial odometry, and the marginalized prior residual is minimized.

[0134]

[0135] in, represents the inertial re-integrated residual calculated by the laser inertial odometer, represents the inertial re-integrated residual calculated by the visual inertial odometry, Represents the weight coefficient, F p represents the marginalized prior residual.

[0136] Based on this, the disclosed embodiments can use laser radar and camera matching to assist the inertial measurement unit for relative positioning, thereby improving the relative accuracy of the inertial measurement unit positioning; it can also be combined with the global navigation satellite system to achieve seamless positioning of the bridge inspection drone in all scenarios.

[0137] The above mainly introduces the scheme provided by the embodiment of the present disclosure. It is understandable that in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed in this article, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0138] The disclosed embodiment can divide the electronic device into functional units according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the disclosed embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0139] By dividing each functional module according to each function, an embodiment of the present disclosure provides a seamless positioning device for full-scene cruising of a bridge, which is applied to a seamless positioning system for full-scene cruising of a bridge. The seamless positioning system includes a global navigation satellite system and an inertial measurement unit, a camera and a lidar installed on a bridge cruiser. Figure 4 This is a schematic diagram of the structure of a seamless positioning device for full-scene cruising on a bridge provided by an embodiment of the present disclosure. Figure 4 As shown, the device 400 includes:

[0140] An acquisition module 401 is used to acquire satellite measurement data of the global navigation satellite system and inertial measurement data of the inertial measurement unit when a signal of the global navigation satellite system is detected;

[0141] A processing module 402 is used to obtain a first cruise trajectory of the bridge cruiser using the satellite measurement data and the inertial measurement data;

[0142] The acquisition module 401 is also used to acquire the visual image data of the camera, the laser radar data of the laser radar and the inertial measurement data of the inertial measurement unit when the signal of the global navigation satellite system is not detected; and acquire a pre-built factor graph and a factor graph optimization model; wherein the factors of the factor graph include an inertial odometry factor, a laser inertial odometry factor and a visual inertial odometry factor;

[0143] The processing module 402 is also used to use the visual image data, the laser radar data, the inertial measurement data and the factor graph optimization model to incrementally update the factor graph to obtain a second cruise trajectory of the bridge cruiser;

[0144] The processing module 402 is further configured to merge the first cruising trajectory and the second cruising trajectory to obtain the cruising trajectory of the bridge cruiser.

[0145] In some embodiments, the processing module 402 is further used to define the inertial odometry factor, the laser inertial odometry factor and the visual inertial odometry factor; determine the constraint relationship between the inertial odometry factor, the laser inertial odometry factor and the visual inertial odometry factor; and construct the factor graph using the constraint relationship.

[0146] In some embodiments, the inertial odometry factor is defined as:

[0147]

[0148] in, represents the residual between the i-th and j-th inertial measurement unit observations, represents the measurement value observed by the inertial measurement unit, represents the predicted value of the inertial measurement unit observation;

[0149] The laser inertial odometry factor is defined as:

[0150]

[0151] Among them, e imu represents the inertial odometer factor, represents the predicted pose constraint provided by the inertial odometer, W imu Represents the weight matrix corresponding to the inertial measurement unit, W l represents the weight matrix corresponding to the laser radar, Represents the distance between point and point, point and line, and point and surface;

[0152] The visual inertial odometry factor is defined as:

[0153]

[0154] Among them, e rproj represents the visual reprojection factor, e imu represents the inertial odometer factor, E p represents the marginalized prior residual, represents the pose prior from the inertial odometry, obs represents the set of visual features i tracked by other frames, and the set C contains the visual node pairs (a, b) connected by the inertial odometry factor. reproj represents the covariance matrix of the visual reprojection factor, W imu Represents the covariance matrix of the inertial odometry factors.

[0155] In some embodiments, the factor graph optimization model is represented as:

[0156]

[0157] in, represents the inertial re-integrated residual calculated by the laser inertial odometer, represents the inertial re-integrated residual calculated by the visual inertial odometry, represents the weight coefficient, E p represents the marginalized prior residual.

[0158] In some embodiments, the factors of the factor graph further include pre-constructed measurement models of the global navigation satellite system and the inertial measurement unit; wherein the measurement model is used to characterize the relationship between the observation value of the global navigation satellite system and the navigation state of the inertial measurement unit;

[0159] The acquisition module 401 is also used to acquire the pre-built measurement model;

[0160] The processing module 402 is further configured to calibrate the inertial measurement unit using the measurement model and the satellite measurement data.

[0161] In some embodiments, the measurement model is represented as:

[0162]

[0163] in, represents the residuals of GNSS observations, represents the measurement value observed by the global navigation satellite system, h i (x i ) represents the predicted value from GNSS observations.

[0164] On the basis of the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

[0165] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0166] In some implementations of this embodiment, a computer program product is provided, including a computer program / instructions, and when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0167] The processor may include, but is not limited to, one or more processors or microprocessors, etc. Each processor may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0168] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the computer-readable storage medium may include but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0169] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0170] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0171] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0172] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0173] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0175] Although the embodiments disclosed in the present disclosure are as above, the above contents are only embodiments adopted for facilitating the understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the technical field to which the present disclosure belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A seamless positioning method for full-scene cruising on a bridge, characterized in that: A seamless positioning system for full-scenario cruising on a bridge, the seamless positioning system comprising a global navigation satellite system and an inertial measurement unit, a camera and a laser radar installed on a bridge cruiser; the method comprising: When the signal of the global navigation satellite system is detected, satellite measurement data of the global navigation satellite system and inertial measurement data of the inertial measurement unit are acquired; and a first cruise trajectory of the bridge cruiser is obtained by using the satellite measurement data and the inertial measurement data; When the signal of the global navigation satellite system is not detected, the visual image data of the camera, the lidar data of the lidar and the inertial measurement data of the inertial measurement unit are obtained; and a pre-constructed factor graph and a factor graph optimization model are obtained; wherein the factors of the factor graph include an inertial odometry factor, a laser inertial odometry factor and a visual inertial odometry factor; Using the visual image data, the laser radar data, the inertial measurement data and the factor graph optimization model, incrementally updating the factor graph to obtain a second cruise trajectory of the bridge cruiser; The first cruise trajectory and the second cruise trajectory are merged to obtain the cruise trajectory of the bridge cruiser.

2. The method according to claim 1, characterized in that The method further comprises: defining the inertial odometry factor, the laser inertial odometry factor and the visual inertial odometry factor; Determining a constraint relationship between the inertial odometry factor, the laser inertial odometry factor, and the visual inertial odometry factor; The factor graph is constructed using the constraint relationship.

3. The method according to claim 2, characterized in that The inertial odometry factor is defined as: in, represents the residual between the i-th and j-th inertial measurement unit observations, represents the measurement value observed by the inertial measurement unit, represents the predicted value of the inertial measurement unit observation; The laser inertial odometry factor is defined as: Among them, e imu represents the inertial odometer factor, represents the predicted pose constraint provided by the inertial odometer, W imu Represents the weight matrix corresponding to the inertial measurement unit, W l represents the weight matrix corresponding to the laser radar, Represents the distance between point and point, point and line, and point and surface; The visual inertial odometry factor is defined as: Among them, e reproj represents the visual reprojection factor, e imu represents the inertial odometer factor, E p represents the marginalized prior residual, represents the pose prior from the inertial odometry, obs(i) represents the set of visual features i tracked in other frames, and the set C contains the visual node pairs (a, b) connected by the inertial odometry factor. reproj represents the covariance matrix of the visual reprojection factor, W imu Represents the covariance matrix of the inertial odometry factors.

4. The method according to claim 1, characterized in that: The factor graph optimization model is expressed as: in, represents the inertial re-integrated residual calculated by the laser inertial odometer, represents the inertial re-integrated residual calculated by the visual inertial odometry, represents the weight coefficient, E p represents the marginalized prior residual.

5. The method according to any one of claims 1 to 4, characterized in that: The factors of the factor graph also include pre-constructed measurement models of the global navigation satellite system and the inertial measurement unit; wherein the measurement model is used to characterize the relationship between the observation value of the global navigation satellite system and the navigation state of the inertial measurement unit; the method also includes: Acquire the pre-built measurement model; The inertial measurement unit is calibrated using the measurement model and the satellite measurement data.

6. The method according to claim 5, characterized in that The measurement model is expressed as: in, represents the residuals of GNSS observations, represents the measurement value observed by the global navigation satellite system, h i (x i ) represents the predicted value from GNSS observations.

7. A seamless positioning device for full-scene cruising on a bridge, characterized in that: A seamless positioning system for full-scenario cruising on a bridge, the seamless positioning system comprising a global navigation satellite system and an inertial measurement unit, a camera and a laser radar installed on a bridge cruiser; the device comprises: an acquisition module, configured to acquire satellite measurement data of the global navigation satellite system and inertial measurement data of the inertial measurement unit when a signal of the global navigation satellite system is detected; A processing module, used for obtaining a first cruise trajectory of the bridge cruiser by using the satellite measurement data and the inertial measurement data; The acquisition module is also used to acquire visual image data of the camera, laser radar data of the laser radar, and inertial measurement data of the inertial measurement unit when the signal of the global navigation satellite system is not detected; and acquire a pre-constructed factor graph and a factor graph optimization model; wherein the factors of the factor graph include an inertial odometry factor, a laser inertial odometry factor, and a visual inertial odometry factor; The processing module is also used to incrementally update the factor graph using the visual image data, the laser radar data, the inertial measurement data and the factor graph optimization model to obtain a second cruise trajectory of the bridge cruiser; The processing module is further used to merge the first cruise trajectory and the second cruise trajectory to obtain the cruise trajectory of the bridge cruiser.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images