High-altitude platform multi-view laser vision-inertial fusion positioning and mapping device and method

Abstract

The application provides a high-altitude platform multi-view laser vision inertial fusion positioning and mapping device and method, the device comprises a fixed holder, a monocular camera, a stereo camera, an IMU inertial sensor, a plurality of laser radars, a time synchronization system and a computing main control unit; the device is rigidly fixed to the end of the lifting platform of the arm truck and moves with the working bucket, the rotation and lifting are controlled by operating the hydraulic arm, the device is started synchronously by the time synchronization system at the starting moment of lifting; the laser radar scans the environment, the IMU inertial sensor performs pre-integration to obtain rotation and displacement increments, the point cloud data and the IMU inertial sensor data are fed back to the computing main control unit, adjacent frame matching is performed respectively to obtain a laser odometer, then global factor graph optimization is performed, finally, loop detection is performed to fuse and construct a map, the overall motion information is obtained through back-end nonlinear optimization, and the real-time pose of the working bucket is determined.

Description

Technical Field

[0001] This invention relates to the field of intelligent positioning technology for aerial work machinery, specifically to a multi-view laser vision-inertial fusion positioning and mapping device and method for aerial platforms. Background Technology

[0002] SLAM (simultaneous localization and mapping) is a technology that enables autonomous systems such as robots to perform localization, mapping, and path planning in unknown environments. Currently, SLAM technology is widely used in robotics, autonomous driving, AR, and VR, relying on multiple sensors to achieve autonomous localization, mapping, and path planning. Mainstream SLAM technologies include laser SLAM (based on lidar for mapping and navigation), visual SLAM (VSLAM, based on single / multi-view camera vision for mapping and navigation), and fusion SLAM combining laser, vision, and inertial sensors. LiDAR, cameras, and inertial sensing units are common sensors in SLAM hardware platforms.

[0003] Although SLAM technology has been applied in many fields, it has not been applied to the field of high-altitude operations before. High-altitude operations currently have many pain points. High-altitude operation equipment has insufficient perception capabilities and relies on human experience, which brings high risks and low efficiency. If SLAM technology can be applied to the field of high-altitude operations, it will be very valuable, as intelligent information technology can be used to significantly improve its safety and operational efficiency.

[0004] Compared to ground robots or autonomous vehicles, aerial work platforms have the additional vertical movement in the Z-direction. Therefore, it is not only necessary to focus on the surrounding environment in all directions for positioning and mapping, but also to be aware of the environment above and below the work platform. If the perception is inadequate, the platform will not be able to determine and avoid obstacles in various directions in time. Therefore, this invention designs a multi-view, multi-sensor fusion positioning and mapping device for the special characteristics of aerial work scenarios. It uses a variety of lidar, cameras, and inertial sensing units to enable the aerial work platform to perceive the working environment in all directions without blind spots, accurately locate itself, and build a three-dimensional map of the working environment, which greatly improves the intelligence level of the aerial work platform. Summary of the Invention

[0005] To address the limitations of existing laser SLAM or visual SLAM methods when applied to aerial work platforms, which restrict the field of view and prevent sufficient perception and reconstruction of the environment, this invention provides a multi-view laser vision-inertial fusion positioning and mapping device and method for aerial work platforms, thereby improving the robustness of aerial work systems in scenes lacking texture or features.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A multi-view laser vision-inertial fusion positioning and mapping device for high-altitude platforms includes a fixed gimbal support, a monocular camera, a stereo camera, an IMU inertial sensor, multiple lidars, a time synchronization system, and a computing main control unit. The lidars include a horizontal 360-degree lidar, a vertical 360-degree lidar, and a side-looking solid-state lidar. The monocular camera includes a first surround-view fisheye camera and a second surround-view fisheye camera. The stereo cameras include forward-looking, rear-looking, and upward-looking and downward-looking stereo cameras. The horizontal 360-degree lidar, the first surround-view fisheye camera, and the IMU inertial sensor are fixed to the gimbal at the center of the positioning and mapping device's base platform. The forward-looking and rear-looking stereo cameras are fixed to the positioning and mapping device's base platform and installed at the edge of the system. The second surround-view fisheye camera and the side-looking solid-state lidar are also fixed to the gimbal. The mapping device is mounted on the base platform at the edge of the system, with the second surround-view fisheye camera adjacent to the side-view solid-state lidar. The vertical 360-degree lidar is placed perpendicular to the base platform. The device is rigidly fixed to the end of the bucket truck's lifting platform and moves with the bucket. When the bucket truck starts working, the hydraulic arm is manipulated to control rotation and lifting. The device is synchronously powered on via the time synchronization system at the start of lifting. The lidar scans the environment, and the IMU inertial sensor performs pre-integration to obtain rotation and displacement increments. The point cloud data and IMU inertial sensor data are fed back to the main control unit for computation. Adjacent frame matching is performed to obtain the laser odometry, followed by global factor map optimization. Finally, loop closure detection is performed to fuse the calculation results and construct a map. The overall motion information is obtained through back-end nonlinear optimization to determine the real-time position and attitude of the bucket.

[0008] Furthermore, the lidar is a mechanically rotating lidar or a solid-state area array lidar.

[0009] This invention also provides a multi-view laser vision-inertial fusion positioning and mapping method for high-altitude platforms, which is implemented through an equipment installation and calibration module, a time synchronization module, and a high-altitude positioning and mapping algorithm module;

[0010] The equipment installation and calibration module is used to design the gimbal bracket, which rigidly connects the lidar, monocular camera, stereo camera and IMU inertial sensor to ensure that the relative positions of the lidar, monocular camera, stereo camera and IMU inertial sensor do not change during the lifting and lowering of the aerial work platform. At the same time, it rigidly connects the gimbal bracket to the lifting platform work bucket to ensure that the IMU inertial sensor accurately reflects the motion posture of the lifting platform work bucket.

[0011] The positioning and mapping device is equipped with a sensing device at its front end, which is a 360-degree horizontal lidar at the center position, sensing the horizontal environment as the aerial work platform rises. Forward-looking and rear-looking stereo cameras are installed on the horizontal axis, performing image-point cloud modal fusion with the multi-line horizontal lidar to provide redundant information for environmental reconstruction using multi-view parallax depth measurement and laser reflection depth measurement. Simultaneously, for the boom lift aerial work platform, a rear-looking stereo camera observes the moving boom for attitude determination and dynamic object removal. A vertical 360-degree lidar is placed on the side of the sensing device to supplement the horizontal sensing when a denser point cloud is needed.

[0012] The time synchronization module performs time synchronization for the monocular camera, stereo camera, lidar, IMU inertial sensor, and main control computing module.

[0013] The high-altitude positioning and mapping algorithm module is used to enable the horizontal sensor group to perceive environmental information in a 360-degree horizontal range. It includes a horizontal sensor group visual inertial system, a horizontal sensor group laser inertial system, and a vertical laser radar and IMU inertial sensor system.

[0014] Furthermore, the first stage of the time synchronization module is a GNSS receiving module, which acquires UTC true time data with nanosecond-level accuracy via satellite. The FPGA module processes the timing information using low-latency parallel logic circuits, converting the GNSS signal into a PPS signal and a NEMA signal. The lidar is connected to both signals for time synchronization. This synchronization signal synchronizes the horizontal 360-degree lidar, the solid-state array lidar, and the vertical 360-degree lidar. Simultaneously, the PPS signal is connected to the IMU inertial sensor and the frequency divider module. The PPS signal keeps the timestamp of the inertial data of the IMU inertial sensor consistent with the UTC true time. The PPS signal connected to the frequency divider module triggers the camera according to the required frame rate. The camera trigger signal is aligned with the PPS signal at the whole second edge. The delay between the two signals is within tens of nanoseconds, so the camera exposure time is synchronized with the IMU inertial sensor data acquisition time.

[0015] Furthermore, the time synchronization of each camera is completed during the equipment manufacturing process, so the trigger signal simultaneously triggers all cameras on the serial trigger line; the main control computing module receives NDT network data packets converted and sent by the FPGA module and performs time synchronization through the NDT protocol.

[0016] Furthermore, the horizontal sensor group visual inertial system is used for initialization and positioning; each surround-view camera and IMU inertial sensor form a small visual inertial positioning system, and the working state is determined by a reliability program. Each small visual inertial positioning system is weighted with a reliability coefficient, and the overall visual inertial system positioning result is finally output; the vision part obtains the initial state and sensor parameters from the radar, associates the radar frame with the image key frame according to the image timestamp, and performs interpolation to complete the initialization.

[0017] Furthermore, the horizontal sensor group laser inertial system is used to perform higher-precision secondary positioning and dense mapping based on the initial positioning of the horizontal sensor group visual inertial system. It uses the horizontal 360-degree lidar point cloud as the main input to the positioning module and the solid-state lidar as the main input to the mapping module, outputting the positioning trajectory, pose, and 3D dense point cloud environment map of the aerial work platform. First, the initial pose of the radar is roughly estimated using motion data obtained from the IMU inertial sensors fixed on the aerial work platform and the initialization information of the horizontal sensor group visual inertial system. The pose increment is used to transform the current laser point to the coordinate system of the starting laser point, relative to the starting point of the laser frame, to correct the laser frame rate. Feature points are extracted for each frame of the horizontal 360-degree lidar, and the curvature of each point is calculated from the point cloud data collected by the high-altitude lidar, with two corresponding thresholds set. After comparing the thresholds and curvature, corner points and planar points are extracted. Curvature greater than the upper limit of the threshold is considered a corner point, and curvature less than the upper limit is considered a corner point. The lower threshold is a planar point. After completing laser distortion correction and point cloud feature point extraction, the localization and mapping section begins. First, based on the feature points obtained by the feature extraction module, inter-frame matching is performed between adjacent data to obtain the positioning information of the laser odometry. Then, based on the calibration information before sensor installation, the coordinate transformation relationship between the side-looking solid-state lidar, the horizontal 360-degree lidar, and the IMU inertial sensor is obtained. The solid-state lidar frame is transformed to the overall system coordinate system as a key frame. A backend nonlinear localization and mapping optimization factor map is constructed. Key frames are added to the factor map, along with laser ranging factors, loop closure factors, and visual initialization factors. The factor map is optimized, and the pose of all key frames is updated. Finally, loop closure detection is performed, searching for frames with similar distances and large distances in historical key frames. To match frames, local key frames are extracted around the matching frames. At the same time, a single-frame scan is performed on the global map to obtain the pose transformation, constructing loop closure factor data and adding factor map optimization. Finally, accurate positioning information and a point cloud map of the working environment are output.

[0018] Furthermore, the vertical lidar and IMU inertial sensor system can work independently as a separate system or rely on the horizontal sensor group laser inertial system as a sub-module. The vertical lidar is used to sense objects in the field of view above and below the high-altitude work platform to avoid collisions and compression. Its panoramic field of view is 180 degrees above and 180 degrees below the platform. A binary option is set in the vertical lidar. When there are enough feature points, it independently performs localization and mapping. The final output result is weighted and estimated with the output trajectory of the horizontal sensor group laser inertial system after maximum likelihood weight estimation. When there are insufficient feature points, it does not work independently. The horizontal sensor group laser inertial system is used as a sub-module to supplement the blind spots in the field of view. Its sensing frame is used as the map frame of the localization and mapping factor map of the horizontal sensor group laser inertial system. It performs joint optimization synchronously with the horizontal sensor group laser inertial system. The final output localization and mapping result is the result of the horizontal sensor group laser inertial system.

[0019] Furthermore, the horizontal sensor group visual inertial system, the horizontal sensor group laser inertial system, and the vertical laser radar and IMU inertial sensor system are all equipped with failure detection. When one of the systems fails, its mapping and positioning mapping results are not used or its independence is canceled, and the positioning mapping results of the acceptable system are used for joint optimization.

[0020] Beneficial effects:

[0021] This invention comprises multiple laser-vision-inertial systems with different observation directions. Even if some of the laser-inertial navigation and vision-inertial navigation subsystems fail, the overall system can still function, significantly improving the robustness of the high-altitude work system in scenes lacking texture or features. This multi-view, multi-source sensor fusion positioning and mapping system has been tested in a real-world working environment and achieved excellent results, effectively solving the problem of omnidirectional, blind-spot-free scene perception required for three-dimensional motion in high-altitude work scenarios. Attached Figure Description

[0022] Figure 1 This is an engineering design drawing of the high-altitude platform multi-view laser vision-inertial fusion positioning and mapping device of the present invention; wherein, 1-horizontal 360-degree laser radar, 2-first surround-view fisheye camera, 3-IMU inertial sensor, 4-forward-viewing stereo camera, 5-rear-viewing stereo camera, 6-second surround-viewing fisheye camera, 7-side-viewing solid-state laser radar, 8-vertical 360-degree laser radar.

[0023] Figure 2-1 The images show a side view and a top view of the installation of the multi-view laser vision-inertial fusion positioning and mapping device for high-altitude platforms and high-altitude work machinery according to the present invention.

[0024] Figure 2-2 For corresponding 3D engineering design drawings; among them, 9-high-altitude platform multi-view laser vision inertial fusion positioning and mapping device, 10-high-altitude work platform, 11-high-altitude work platform lifting device, 12-chassis;

[0025] Figure 3 This is an engineering implementation diagram of the multi-view laser vision-inertial fusion positioning and mapping device for high-altitude platforms of the present invention, fully installed on a high-altitude work platform.

[0026] Figure 4 A 3D point cloud map is constructed using images of the high-altitude working environment perceived by panoramic and stereo cameras from all angles, as well as LiDAR.

[0027] Figure 5 When the laser component fails, only the surround-view camera and IMU inertial module are used to locate and determine the pose of the high-altitude platform. The lines represent the historical trajectory, and the rectangles represent the real-time pose of the platform.

[0028] Figure 6 A schematic diagram of the positioning of an aerial platform for simultaneous operation of multi-view laser, vision, and inertial systems;

[0029] Figure 7 This is a data flow diagram after clock synchronization of multi-view sensors;

[0030] Figure 8 This is a diagram of the sensor front-end clock synchronization trigger signal. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] The high-altitude platform multi-view laser vision-inertial fusion positioning and mapping method of the present invention is implemented through an equipment installation and calibration module, a time synchronization module, and a high-altitude positioning and mapping algorithm module.

[0033] The equipment installation and calibration module is used to design the gimbal bracket, rigidly connecting the lidar, camera, and IMU inertial sensor. This ensures that the relative positions of the lidar, camera, and IMU remain unchanged during the lifting and lowering of the aerial work platform. Simultaneously, the gimbal bracket is rigidly connected to the lifting platform's work bucket, ensuring that the IMU accurately reflects the movement and posture of the work bucket. The front-end of the multi-view laser vision-inertial fusion positioning and mapping device for aerial work platforms of this invention features a sensing device: a horizontally positioned, rotating 360-degree multi-line lidar that senses the horizontal environment as the aerial work platform rises. Forward-looking and rear-looking stereo cameras are installed on the horizontal axis, performing image-point cloud modal fusion with the horizontal multi-line lidar to provide redundant information for environmental reconstruction using multi-view parallax depth measurement and laser reflection depth measurement. Simultaneously, for the boom-lift aerial work platform, the rear-looking stereo camera observes the moving boom for attitude determination and dynamic object removal. A third 360-degree rotating multi-line lidar and upward-looking and downward-looking cameras are vertically mounted on the side of the sensing device to fill in the blind spots of the horizontal radar and measure the height of the aerial work platform above the ground and the distance to obstacles above. A monocular fisheye camera and a solid-state area array lidar are installed in the horizontal axis direction perpendicular to the platform where the gimbal support is located, to supplement the horizontal sensing when a denser point cloud is needed. Due to the scanning characteristics of the solid-state area array lidar, the point cloud density can accumulate over time when the aerial work platform is hovering and working.

[0034] The time synchronization module synchronizes the time of the camera, lidar, IMU inertial sensor, and main control computing module. The first stage of the time synchronization module is a GNSS receiving module, which acquires UTC true time data with nanosecond precision via satellite. The FPGA module processes the timing information using low-latency parallel logic circuits, converting the GNSS signal into PPS and NEMA signals. The lidar receives both signals for time synchronization. This synchronization signal synchronizes the horizontal 360-degree lidar, the solid-state array lidar, and the vertical 360-degree lidar. Simultaneously, the PPS signal is connected to the IMU inertial sensor and the frequency divider module. The PPS signal ensures that the timestamp of the IMU inertial sensor's inertial data is consistent with the UTC true time. The PPS signal connected to the frequency divider module triggers the camera according to the required frame rate. The camera trigger signal is aligned with the PPS signal at the whole-second edge, and the delay between the two signals is within tens of nanoseconds. Therefore, the camera's exposure time is synchronized with the IMU inertial sensor's data acquisition time. The time synchronization of each camera is completed during equipment manufacturing, so the trigger signal will simultaneously trigger all cameras on the serial trigger line. The main control computing module receives NDT network data packets converted and sent by the FPGA module, and performs time synchronization through the NDT protocol.

[0035] This invention designs photoelectric trigger sensors for horizontally oriented lidar and axially aligned stereo cameras. These trigger sensors include photoelectric, Hall effect, and mechanical reed trigger sensors. When the lidar's field of view coincides with the camera's field of view, a switch is activated, causing the camera to expose and acquire an image, ensuring the spatiotemporal synchronization of the point cloud and the image.

[0036] The high-altitude positioning and mapping algorithm module is used to enable the horizontal sensor group to perceive environmental information in a 360-degree horizontal range. It includes a horizontal sensor group visual inertial system, a horizontal sensor group laser inertial system, and a vertical laser radar and IMU inertial sensor system.

[0037] I. The horizontal sensor array of the visual-inertial system is mainly used for initialization and localization. The three-dimensional LiDAR, multiple cameras, and IMU inertial sensors complement each other, each with its own strengths. However, due to the lack of scale information in traditional monocular visual-inertial systems, the initialization effect depends heavily on the system's initial motion, resulting in insufficient success rate. To avoid the significant influence of individual cameras on random factors, each surround-view camera and IMU inertial sensor forms a small visual-inertial localization system. The operating state is determined through a reliability program, and each visual-inertial localization system is weighted with a reliability coefficient, ultimately outputting the overall visual-inertial system localization result. Simultaneously, using a robust radar with clear scale aids in the initialization of the visual-inertial system. The vision component obtains the initial state and sensor parameters from the radar. Since time synchronization between the radar and cameras has been established, radar frames can be associated with image keyframes based on image timestamps, and interpolation can be performed to complete the initialization.

[0038] II. The horizontal sensor-group laser inertial system is mainly used for higher-precision secondary positioning and dense mapping based on the initial positioning of the horizontal sensor-group visual inertial system. There are two main types of horizontal lidar: the first is 360-degree multi-line lidar, characterized by its wide detection range and ability to perceive three-dimensional environmental information in the horizontal direction without blind spots; its drawback is that its scanning method is repetitive, resulting in relatively sparse point clouds per frame. Therefore, this invention supplements the horizontal direction with a second type of solid-state lidar to jointly complete the mapping task. Solid-state lidar is characterized by non-repetitive scanning, and the density of the environmental point cloud accumulates over time to achieve a dense and uniform distribution. The algorithm of the horizontal sensor-group laser inertial system uses the 360-degree lidar point cloud as the main input to the positioning module and the solid-state lidar as the main input to the mapping module, outputting the positioning trajectory, pose, and three-dimensional dense point cloud environmental map of the aerial work platform. Because the laser beam distorts during the movement of the lidar, the initial pose of the lidar is first roughly estimated using motion data obtained from the IMU inertial sensor fixed on the high-altitude work platform and the initialization information of the visual inertial system of the horizontal sensor group. The pose increment is then used to transform the current laser point to the coordinate system of the starting laser point, relative to the starting point of the laser frame, thus correcting the laser frame rate. To ensure the real-time performance of the positioning and mapping method, feature points are extracted for each frame of the 360-degree multi-line lidar. The curvature of each point is calculated from the point cloud data collected by the high-altitude lidar, and two corresponding thresholds are set. After comparing the thresholds and curvature, corner points and planar points are extracted. Curvature greater than the upper threshold is classified as a corner point, and curvature less than the lower threshold is classified as a planar point. After completing the laser distortion correction and point cloud feature point extraction algorithms, the localization and mapping phase begins. First, based on the feature points obtained from the feature extraction module, inter-frame matching is performed between adjacent data to obtain the laser odometry's positioning information. Then, based on the calibration information before sensor installation, the coordinate transformation relationship between the solid-state lidar, the 360-degree multi-line lidar, and the IMU inertial sensor is obtained. The solid-state lidar frame is transformed to the overall system coordinate system as a keyframe, and a backend nonlinear localization and mapping optimization factor map is constructed. Keyframes are added to the factor map, along with laser ranging factors, loop closure factors, and visual initialization factors. Factor map optimization is performed, and the pose of all keyframes is updated. Finally, loop closure detection is performed: frames with similar distances but relatively far apart are searched in the historical keyframes. To match frames, local keyframes are extracted around the matching frame. Simultaneously, a single-frame scan of the global map is performed to obtain the pose transformation, constructing loop closure factor data, and adding factor map optimization. The final output is precise positioning information and a point cloud map of the operating environment.

[0039] III. Vertical LiDAR and IMU Inertial Sensor System: The vertical LiDAR and IMU inertial sensor system can work collaboratively as an independent system or rely on the horizontal sensor group LiDAR inertial system as a sub-module. The algorithm's operating mode determination conditions are as follows: The vertical LiDAR is mainly used to sense objects in the field of view above and below the high-altitude work platform to avoid collisions and compression. Its panoramic field of view is 180 degrees above and 180 degrees below the platform. Because the LiDAR is vertically placed, the sensed objects are small objects on the ground or in the air, often without obvious features. When extracting key feature points from the radar frames, it is often impossible to extract enough key points for inter-frame matching. This vertical LiDAR system has a binary option. When there are enough feature points, it performs localization and mapping independently, with the same algorithm as II. The final output is weighted and estimated by maximum likelihood weighting with the output trajectory of the horizontal sensor group LiDAR system. When there are insufficient feature points, it does not work independently. The horizontal sensor group LiDAR system is used as a sub-module to supplement blind spots. Its perception frames serve as map frames for the localization and mapping factor map of the horizontal sensor group LiDAR system and are jointly optimized synchronously with the horizontal sensor group LiDAR system. Finally, the overall localization and mapping result output by the entire system is the result of the horizontal sensor group LiDAR system.

[0040] The horizontal sensor group visual inertial system, the horizontal sensor group laser inertial system, and the vertical LiDAR and IMU inertial sensor system are all equipped with failure detection. When a sensor in one of the systems fails, its mapping and localization mapping results are not used or its independence is canceled, and joint optimization is performed using the localization mapping of the acceptable sensor group.

[0041] The high-altitude work platform laser vision and inertial navigation fusion positioning and mapping device of the present invention includes a fixed gimbal support, a monocular camera, a stereo camera, an IMU inertial sensor, multiple lidars, a power supply, a computing main control unit, a time synchronization system, and a support frame. The lidars can be either mechanically rotating lidars or solid-state area array lidars. The device is rigidly fixed to the end of the boom lift platform and moves with the work bucket. When the boom lift starts working, the operator manipulates the hydraulic arm to control rotation and lifting, and the device is simultaneously activated at the start of lifting. The LiDAR scans the environment, and the IMU inertial sensor performs pre-integration to obtain rotation and displacement increments. The point cloud data and IMU inertial sensor data are fed back to the main control unit. At this time, two threads perform adjacent frame matching to obtain the LiDAR odometry, followed by global factor graph optimization. Finally, loop closure detection is performed to fuse the calculation results to build a map. The overall motion information is obtained through backend nonlinear optimization to determine the real-time position and attitude of the working bucket. The surround-view camera perceives high-resolution images and color information of the environment, which, together with the inertial sensor, assists in the initialization of the positioning and mapping system. The stereo camera obtains the environmental depth through the parallax method, which complements the depth information of the LiDAR.

[0042] Specifically, such as Figure 1 , Figure 2-1 , Figure 2-2 , Figure 3As shown, the sensing devices at the front end of the high-altitude work platform laser vision and inertial navigation fusion positioning and mapping device of the present invention include a horizontal 360-degree lidar 1, a vertical 360-degree lidar 8, a forward-looking stereo camera 4, a rear-looking stereo camera 5, a first surround-view fisheye camera 2, a second surround-view fisheye camera 6, a side-view solid-state lidar 7, and an IMU inertial sensor 3. The horizontal 360-degree lidar 1, the first surround-view fisheye camera 2, and the IMU inertial sensor 3 are fixed on a gimbal at the center of the positioning and mapping device base platform. Only the forward-facing portion of the surround-view fisheye camera is shown, and its number and viewing angle can be increased or decreased as needed. The forward-facing stereo camera 4 and the rear-facing stereo camera 5 are fixed on the positioning and mapping device base platform and installed at the edge of the system. The second surround-view fisheye camera 6 and the side-view solid-state lidar 7 are also fixed on the positioning and mapping device base platform and installed at the edge of the system. The second surround-view fisheye camera 6 is adjacent to the side-view solid-state lidar 7, and only the right-facing portion is shown. Its number and viewing angle can be increased or decreased as needed. The vertical 360-degree lidar 8 is placed perpendicular to the base platform, and its center is a certain distance from the base platform during installation to keep the radar suspended as much as possible to avoid obstruction. The multi-view laser vision inertial fusion positioning and mapping device 9 for the aerial work platform is rigidly mounted on the aerial work platform 10; the aerial work platform 10 is hinged and fixed to the aerial work platform lifting device 11, which drives the aerial work platform 10 to move in the air; the aerial work platform lifting device 11 is fixed to the chassis 12, which is used to support the upper aerial work mechanical structure.

[0043] Figure 4 The image shows a 3D point cloud map constructed from images of the high-altitude working environment perceived by a panoramic camera and a stereo camera, as well as from a lidar system. The image shown is the mapping result of the horizontal laser-inertial-vision coupled mapping system of the present invention. Figure 5 When the laser component fails, the high-altitude platform is positioned and its pose is determined using only the surround-view camera and the IMU inertial module. The lines represent the historical trajectory, and the rectangles represent the real-time pose of the platform. The results of the visual-inertial fusion positioning using the surround-view fisheye camera are shown. Figure 6 A schematic diagram of the positioning of a high-altitude platform for simultaneous operation of multi-view laser, vision and inertial systems; showing the positioning and mapping results of side-view solid-state array lidar-inertial fusion.

[0044] Figure 7 The diagram shows the data flow after clock synchronization of the multi-view sensors according to the present invention. The data streams from the multi-view surround-view camera, stereo camera, multiple types of multi-view LiDAR, and IMU inertial sensor are synchronized in time and then input to the main control unit via a switching network. Figure 7As shown, the clock source of the time synchronization system is a GPS-disciplined clock. The clock system generates two sets of signals, PPS and GPRMC, as well as NTP clock synchronization protocol data packets. The PPS signal is used to synchronize the time reference of all inertial sensing units in the system. The PPS and GPRMC signals jointly synchronize the clock reference of all LiDARs in the system. The NTP protocol is used to synchronize the time of each computing master unit in the system. Furthermore, the exposure of the stereo camera is completed by the LiDAR through a rotating trigger sensor, and all cameras within the stereo camera are synchronously triggered on the same serial line. The exposure of the surround-view camera is triggered by a frequency divider within the IMU inertial sensor, which divides the PPS signal. The time reference of the IMU inertial sensor and LiDAR, after synchronization with the GPS-disciplined clock, serves as the secondary time reference for each camera image. Finally, the synchronized IMU motion information serial data, LiDAR UDP point cloud data, and camera RGB or grayscale image data are input to the corresponding computing master unit.

[0045] Figure 8 The diagram shown illustrates the clock synchronization trigger signal at the sensor front end, reflecting the timing of the data streams from the multi-view panoramic camera, stereo camera, various types of LiDAR, and IMU inertial sensor.

[0046] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-view laser vision-inertial fusion positioning and mapping device for high-altitude platforms, characterized in that: The device comprises a fixed gimbal support, a monocular camera, a stereo camera, an IMU inertial sensor, a plurality of laser radars, a time synchronization system and a computing master unit; the laser radars comprise a horizontal 360-degree laser radar, a vertical 360-degree laser radar and a side-view solid-state laser radar; the monocular camera comprises a first ring-view fisheye camera and a second ring-view fisheye camera; the stereo camera comprises front-view and rear-view stereo cameras and upper-view and lower-view stereo cameras; the horizontal 360-degree laser radar and the first ring-view fisheye camera are fixed on the gimbal at the center of a positioning and mapping device base platform; the front-view and rear-view stereo cameras are fixed on the positioning and mapping device base platform and are installed at the edge of the system; the second ring-view fisheye camera and the side-view solid-state laser radar are also fixed on the positioning and mapping device base platform and are installed at the edge of the system, wherein the second ring-view fisheye camera is adjacent to the side-view solid-state laser radar; the vertical 360-degree laser radar is arranged perpendicularly to the base platform; the device is rigidly fixed to the end of a boom truck lifting platform and moves with the working bucket; when the boom truck starts to work, the rotation and lifting are controlled by the hydraulic arm; the device is started synchronously by the time synchronization system at the start time of lifting; the laser radars scan the environment; the IMU inertial sensor performs pre-integration to obtain rotation and displacement increments; the point cloud data and the IMU inertial sensor data are fed back to the computing master unit; adjacent frame matching is performed to obtain a laser odometry; then global factor graph optimization is performed; finally, the operation results are fused to construct a map; through a back-end nonlinear optimization, overall motion information is obtained to determine the real-time pose of the working bucket; A binary option is arranged in the vertical direction laser radar; when there are enough feature points, positioning and mapping are independently performed; finally, the output results are weighted and output after maximum likelihood weight estimation with the output trajectory of the horizontal sensor group laser inertial system; when there are not enough feature points, the horizontal sensor group laser inertial system is not independently used as a sub-module to supplement the visual field blind area; the sensing frame is used as a map frame of the positioning and mapping factor graph of the horizontal sensor group laser inertial system; the horizontal sensor group laser inertial system is synchronously executed for joint optimization; finally, the positioning and mapping results are output.

2. The high-altitude platform multi-view laser vision-inertial fusion positioning and mapping device according to claim 1, characterized in that: The laser radar is a mechanical rotating laser radar or a solid-state area array laser radar.

3. The positioning and mapping method of the high-altitude platform multi-view laser vision-inertial fusion positioning and mapping device according to claim 1 or 2, characterized in that: The device is realized through a device installation and calibration module, a time synchronization module and a high-altitude positioning and mapping algorithm module; The device installation and calibration module is used for designing a gimbal support, rigidly connecting the laser radars, the monocular camera, the stereo camera and the IMU inertial sensor, ensuring that the relative positions among the laser radars, the monocular camera, the stereo camera and the IMU inertial sensor do not change during the lifting of the high-altitude working platform, rigidly connecting the gimbal support and the lifting platform working bucket, and ensuring that the IMU inertial sensor truly reflects the motion and pose of the lifting platform working bucket. The front end of the positioning mapping device is provided with a perception device, which is a horizontal 360-degree laser radar with a central position, and the horizontal environment is perceived during the lifting process of the aerial work platform; the front-looking and rear-looking stereo cameras are installed on the axis in the horizontal direction, and image-point cloud modal fusion is performed with the horizontal multi-line laser radar to provide multi-view parallax method depth measurement and laser reflection depth measurement redundant information for environment reconstruction; at the same time, for the aerial work platform of the arm truck, the rear-looking stereo camera observes the moving arm for posture determination and dynamic object removal; the perception device is provided with a vertical 360-degree laser radar placed vertically on the side, which is used to supplement when the horizontal perception needs more dense point clouds; The time synchronization module synchronizes the time of the monocular camera, stereo camera, laser radar, IMU inertial sensor and main control calculation module; The aerial positioning mapping algorithm module is used to enable the horizontal sensor group to perceive the environment information in a horizontal 360-degree range, which includes a horizontal sensor group visual inertial system, a horizontal sensor group laser inertial system and a vertical laser radar and IMU inertial sensor system.

4. The method of claim 3, wherein: The first stage of the time synchronization module is a GNSS receiving module, which obtains nanosecond-level precision UTC true value time data through satellites, and a FPGA module processes the time information in a low-delay logic parallel circuit to convert the GNSS signal into a PPS signal and a NEMA signal, and the laser radar accesses the two signals for time synchronization. This synchronization signal synchronizes the horizontal 360-degree laser radar, the solid-state area array laser radar and the vertical 360-degree laser radar, and at the same time, the PPS signal is connected to the IMU inertial sensor and the frequency divider module, so that the time stamp of the inertial data of the IMU inertial sensor is consistent with the UTC true value time. The PPS signal connected to the frequency divider module triggers the camera according to the required frame rate, and the camera trigger signal is aligned with the PPS signal at the whole second edge, and the time delay of the two signals is within tens of ns, so that the camera exposure image time is synchronized with the IMU inertial sensor data acquisition time.

5. The method of Claim 3, wherein: The time of each camera is synchronized during the manufacture of the device, so that the trigger signal simultaneously triggers all the cameras on the serial trigger line; the main control calculation module receives the NDT network data packet converted and sent by the FPGA module, and performs time synchronization through the NDT protocol.

6. The method of Claim 3, wherein: The horizontal sensor group visual inertial system is used for initialization and positioning; each ring-looking camera and the IMU inertial sensor form a small visual inertial positioning system, the working state is determined through a reliability program, and the final output is the positioning result of the whole visual inertial system weighted by the reliability coefficient of each small visual inertial positioning system; the visual part obtains the initial state and sensor parameters from the radar, associates the radar frame with the image key frame according to the image time stamp, and performs interpolation to complete the initialization.

7. The localization mapping method of claim 6, wherein: The horizontal sensor group laser inertial system is used for secondary positioning with higher precision and dense mapping on the basis of initial positioning of the horizontal sensor group visual inertial system, uses horizontal 360-degree laser radar point cloud as the main input of the positioning module, uses solid-state laser radar as the main input of the mapping module, and outputs the positioning trajectory, pose and three-dimensional dense point cloud environment map of the aerial work platform; firstly, the initial pose of the radar is roughly estimated by using the motion data obtained by the IMU inertial sensor fixed on the aerial work platform and the initialization information of the horizontal sensor group visual inertial system, the current laser point is transformed to the coordinate system of the starting laser point by using the pose increment, the laser frame rate is corrected relative to the starting point of the laser frame; feature points are extracted from each frame of the horizontal 360-degree laser radar, the curvature of each point of the point cloud data collected by the aerial laser radar is calculated, and two corresponding threshold values are set; after comparing the threshold value and the curvature, the corner points and plane points are extracted, the curvature greater than the upper threshold value is the corner point, and the curvature less than the lower threshold value is the plane point; after the laser distortion correction and point cloud feature point extraction are completed, the positioning and mapping part is entered, firstly, the feature points obtained by the feature extraction module are matched between adjacent data to obtain the positioning information of the laser odometry, then the coordinate transformation relationship between the side-view solid-state laser radar and the horizontal 360-degree laser radar and the IMU inertial sensor is obtained according to the calibration information before the sensor installation, the solid-state laser radar frame is transformed to the overall system coordinate system as the key frame, the back-end nonlinear positioning and mapping optimization factor graph is constructed, the key frame is added to the factor graph, the laser ranging factor, the closed-loop factor and the visual initialization factor are added, the factor graph is optimized, and the pose of all key frames is updated; finally, the closed-loop detection is performed, and the frames with similar distance and far apart are searched in the historical key frames; in order to match the frames, the local key frames around the matching frames are extracted, the single-frame scanning is performed on the global graph matching to obtain the pose transformation, the closed-loop factor data are constructed, and the factor graph optimization is added; finally, the accurate positioning information and the work environment point cloud map are output.

8. The localization mapping method of claim 7, wherein: The vertical direction laser radar and IMU inertial sensor system in the vertical direction laser radar and IMU inertial sensor system can work independently as an independent system or rely on the horizontal sensor group laser inertial system as a sub-module, and the vertical direction laser radar is used for sensing the objects in the field of view above and below the aerial work platform to avoid collision and extrusion, and the all-around field of view angle is 180 degrees above the platform and 180 degrees below the platform.

9. The method of Claim 3, wherein: The horizontal sensor group visual inertial system, the horizontal sensor group laser inertial system and the vertical direction laser radar and IMU inertial sensor system are all provided with failure detection, when one of the systems fails, the mapping and positioning mapping results thereof are not used or the independence thereof is cancelled, and the positioning and mapping results of the adoptable system are used for joint optimization.

Images