An intelligent vehicle based on panoramic vision and lidar fusion SLAM system
Through the SLAM system that integrates panoramic cameras and lidars, the problem of poor robustness of traditional SLAM systems during cornering and rapid movement is solved, and the precise positioning of smart cars and the construction of complex scene maps is realized, ensuring the safe driving and equipment of smart cars The rationality of the layout.
Patent Information
- Application Number
- CN202111178550.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-10
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-10-10
AI Technical Summary
Traditional visual SLAM systems are poorly robust when turning and moving rapidly, and cannot effectively realize the precise positioning of smart cars and the construction of complex scene maps, especially in terms of rationality of equipment layout and functional guarantees in limited spaces.
The SLAM system that integrates panoramic cameras and lidars is used to collect 360° scene images and point cloud data generated by lidars to fuse them. The mask is used to extract ring feature points, inter-frame matching, local beam adjustment and closed-loop detection. , realize the precise positioning and map construction of smart cars, and avoid obstacles through motor drive.
It improves the positioning accuracy and robustness of smart cars in complex scenarios, can identify reverse closed loops, correct scale drifts, and achieve more accurate spatial point coordinate acquisition and obstacle detection, ensuring the safe driving of smart cars.
Smart Images

Figure CN114092551B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent driving, and in particular relates to an intelligent car based on a SLAM system integrating panoramic vision and laser radar. Background Art
[0002] With the rise of intelligent driving technology, simultaneous localization and mapping (SLAM) has become a research hotspot in the industry. Among them, visual SLAM uses camera sensors to extract rich features in the environment, thereby estimating the position and posture of the vehicle and reconstructing a sparse map of the surrounding scene. However, due to the limited field of view of traditional pinhole cameras, the robustness of the system is poor when the vehicle turns and moves quickly. Panoramic cameras can image the surrounding 360° scenes, so they can provide rich scene features more efficiently, which ensures the accuracy and robustness of visual SLAM in complex scenes. The panoramic vision and laser radar fusion SLAM system can realize the positioning of smart cars. The surrounding scenes are imaged in high definition using a PAL-based panoramic camera, and then input into the computer for a series of processing. The collected image information is fused with the point cloud generated by the laser radar for more accurate positioning and mapping, while realizing functions such as ranging and obstacle avoidance. To install these devices on smart cars, they need to be reasonably laid out in a limited space to better ensure the normal operation of these functions. Summary of the invention
[0003] The purpose of the present invention is to provide an intelligent vehicle based on a panoramic vision and laser radar fusion SLAM system that can realize vehicle positioning, scene map construction, ranging and obstacle avoidance functions.
[0004] The smart car based on the panoramic vision and laser radar fusion SLAM system provided by the present invention includes an upper aluminum alloy plate and a lower aluminum alloy plate, a panoramic camera, a laser radar, a camera hood, a camera bracket, a computer, a lithium battery, a motor drive module, a motor and wheels, etc.; wherein:
[0005] The upper aluminum alloy plate and the lower aluminum alloy plate are fixedly connected by a plurality of aluminum profiles, and the height of the aluminum profiles can be selected according to actual needs;
[0006] There are four motors, which are respectively installed under the four corners of the lower aluminum alloy plate. The output shafts of the motors are connected to the four wheels of the smart car. By controlling the rotation of the four wheels, the smart car can be driven in different directions.
[0007] The lithium battery and the motor drive module are placed in a storage box, which is placed in the middle of the upper surface of the lower aluminum alloy plate; the lithium battery is used to power the smart car to ensure the normal operation of each module; the motor drive module is equipped with a wireless network and a Bluetooth module, etc., which can be connected to a computer or a mobile phone APP, as well as a wireless remote control, etc., to realize the control of the four motors.
[0008] The camera bracket is installed in the middle of the upper surface of the upper aluminum alloy plate, and the camera hood and the panoramic camera are placed above the camera bracket and fixed to prevent shaking and blurring of the captured image; the camera bracket can adjust its height according to the field of view of the panoramic camera to avoid other devices on the smart car blocking its field of view. The camera hood is used to reduce the interference of stray light and improve the imaging effect.
[0009] The computer is arranged behind the upper surface of the upper aluminum alloy plate, and is used to receive and process the image sequence taken by the panoramic camera. A platform such as Raspberry Pi can also be used, and there is no limitation here.
[0010] The laser radar is installed in front of the upper surface of the upper aluminum alloy plate and connected to a computer to detect objects in a certain field of view in front of the smart car and generate a point cloud map.
[0011] The computer is equipped with a panoramic vision and laser radar fusion SLAM system. The panoramic image sequence collected by the panoramic camera and the point cloud map generated by the laser radar are fused in the SLAM system, and the vehicle's posture is estimated and the scene map is reconstructed; at the same time, functions such as distance measurement and obstacle avoidance are realized. The operation process of the panoramic vision and laser radar fusion SLAM system is as follows:
[0012] After the panoramic camera is connected to the computer, it collects continuous frames of images of the 360° scene around the smart car and then inputs them into the computer. The panoramic vision and laser radar fusion SLAM system loaded in the computer will process these image sequences. First, a mask is used to avoid the blind area in the image and extract the feature points of the annular effective area; the laser radar scans the scene in front in real time, and at the same time transmits the point cloud information to the computer, and fuses the laser point cloud information with the image information. If the system has not been initialized, the initialization process will be performed: these feature points are matched between frames, and then the correctly matched feature point pairs are screened to estimate the initial pose of the current camera, and the initial map points in the environment are obtained based on the data fusion results with the laser point cloud. The initial pose of the camera and the initial map points are optimized using the local bundle adjustment (Bundle Adjustment, BA).
[0013] After completing the initialization of the panoramic vision and laser radar fusion SLAM system, the system uses a suitable tracking model to track existing map points based on the current frame image collected by the panoramic camera. Then the current camera motion pose is estimated, the local map is updated, and then the local BA is used to optimize the current pose and map points. Since the panoramic camera has a large field of view, when the smart car moves quickly and turns, the panoramic vision and laser radar fusion SLAM system can also effectively track map points and has high robustness.
[0014] When the panoramic vision and laser radar fusion SLAM system recognizes that the smart car returns to the location it has passed before, it will determine whether the closed-loop detection conditions are met based on the similarity between the current key frame and the key frame generated when it last passed through the place. Since the panoramic camera has a 360° field of view, even if it returns to the previously passed location from the opposite direction, it can ensure that there is sufficient similarity between the current key frame and the key frame generated when it last passed through the place, thereby completing the closed-loop detection and correcting the scale drift. Then global BA is performed to obtain more accurate motion poses and map points.
[0015] When the smart car is driving, the laser point cloud map is integrated with the image information collected by the panoramic camera to obtain more accurate spatial point coordinates, thereby improving the positioning accuracy of the smart car. At the same time, it is judged whether there is an obstacle in front based on the acquired point cloud information. If an obstacle is identified, the computer will send instructions to the motor drive module based on the distance from the obstacle to the smart car, and then control the movement of the wheels through the motor to achieve deceleration and turning of the smart car.
[0016] The smart car based on panoramic vision and laser radar fusion SLAM system provided by the present invention has the following advantages over the existing technology:
[0017] 1. Use panoramic cameras to collect 360° scene image information around the smart car, without the need for image stitching and other operations, which improves the efficiency of the algorithm and reduces the computing cost requirements for the equipment;
[0018] 2. The panoramic vision and lidar fusion SLAM system based on panoramic cameras can not only identify closed loops when moving in the same direction, but also when moving in the opposite direction. Even if the smart car passes the same place from the opposite direction, closed loop detection can be achieved to correct scale drift.
[0019] 3. Since the panoramic camera has a 360° field of view, when the panoramic vision and laser radar fusion SLAM smart car turns quickly, it can also be tracked stably to achieve the positioning of the smart car and the construction of a sparse environment map, which overcomes the shortcomings of the traditional visual SLAM system based on a camera with a limited field of view;
[0020] 4. The SLAM smart car based on panoramic vision and laser radar fusion can perform more accurate positioning and mapping by fusing the data of cameras and laser radars, while realizing functions such as ranging and obstacle avoidance, thereby ensuring the safety of the smart car during driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 The figure is a flowchart of the operation of the panoramic vision and lidar fusion SLAM system.
[0022] Figure 2 This is a front view of an intelligent car based on a panoramic vision and lidar fusion SLAM system.
[0023] Figure 3 This is the left view of an intelligent vehicle based on a panoramic vision and lidar fusion SLAM system.
[0024] Figure 4 This is a bird's-eye view of an intelligent vehicle based on a panoramic vision and lidar fusion SLAM system.
[0025] Figure 5 A schematic diagram of imaging of a panoramic camera. DETAILED DESCRIPTION
[0026] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.
[0027] Figure 1 It is an operation flow chart of a panoramic vision and laser radar fusion SLAM system. After the panoramic camera is connected to the computer, it can collect continuous frames of images of the 360° scene around the smart car and then input them into the computer. The panoramic vision and laser radar fusion SLAM system loaded in the computer will process these image sequences. First, a mask is used to avoid the blind area in the image and extract the feature points of the annular effective area. The laser radar will scan the scene in front in real time, then transmit the point cloud information to the computer, and fuse the laser point cloud information with the image information. If the system has not been initialized, the initialization process will be performed: these feature points will be matched between frames, and then the correctly matched feature point pairs will be selected to estimate the initial pose of the current camera, and the initial map points in the environment will be obtained based on the data fusion results with the laser point cloud. The local bundle adjustment (Bundle Adjustment, BA) is used to optimize the initial pose of the camera and the initial map points.
[0028] After completing the initialization of the panoramic vision and LiDAR fusion SLAM system, the system will use a suitable tracking model to track existing map points based on the current frame image captured by the panoramic camera. Then the current camera motion pose is estimated and the local map is updated, and then the current pose and map points are optimized using local BA. Since the panoramic camera has a large field of view, when the smart car moves quickly and turns, the panoramic vision and LiDAR fusion SLAM system can also effectively track map points and has high robustness.
[0029] When the panoramic vision and laser radar fusion SLAM system recognizes that the smart car returns to the location it has passed before, it will determine whether the closed-loop detection conditions are met based on the similarity between the current key frame and the key frame generated when it last passed through the place. Since the panoramic camera has a 360° field of view, even if it returns to the previously passed location from the opposite direction, it can ensure that there is sufficient similarity between the current key frame and the key frame generated when it last passed through the place, thereby completing the closed-loop detection and correcting the scale drift. Then global BA is performed to obtain more accurate motion poses and map points.
[0030] like Figure 2-4 The figure shows an intelligent car based on a panoramic vision and laser radar fusion SLAM system, including a camera hood 1, a panoramic camera 2, a camera bracket 3, a laser radar 4, an upper aluminum alloy plate 5, wheels 6, a storage box 7 for a battery and a motor drive module, a lower aluminum alloy plate 8, a motor 9, an aluminum profile 10 and a computer 11.
[0031] The upper aluminum alloy plate 5 and the lower aluminum alloy plate 8 are fixedly connected by a plurality of aluminum profiles 10, and the height of the aluminum profiles can be selected according to actual needs. A storage box 7 for the battery and the motor drive module is placed in the middle of the upper surface of the lower aluminum alloy plate 8, and the lithium battery and the motor drive module are fixed therein. The lithium battery is used to power the smart car to ensure the normal operation of each module. The motor drive module is equipped with a wireless network and a Bluetooth module, etc., which can be connected to a computer or a mobile phone APP, as well as a wireless remote control, etc., to realize the control of the four motors. Motors are installed under the four corners of the lower aluminum alloy plate, and the output shaft of the motor is connected to four wheels, which are controlled to rotate to realize the driving of the smart car in different directions. The wheels used in this embodiment are universal wheels, and other wheels can also be used to achieve similar functions, which are not limited here.
[0032] A camera bracket 3 is installed in the middle of the upper surface of the upper aluminum alloy plate 5. The camera hood 1 and the panoramic camera 2 are placed on the camera bracket 3 and fixed to prevent the shaking from causing the captured image to be blurred. The height of the camera bracket 3 can be adjusted according to the field of view of the panoramic camera 2 to avoid other devices on the smart car blocking its field of view. The camera hood 1 can reduce the interference of stray light and improve the imaging effect. The image sequence captured by the panoramic camera can be transmitted to a computer for processing. Behind the upper surface of the upper aluminum alloy plate 5 is a computer 11, and platforms such as Raspberry Pi can also be used, without limitation. A laser radar 4 is installed in front of the upper surface of the upper aluminum alloy plate 5, which can be connected to a computer to detect objects in a certain field of view in front of the smart car and generate a point cloud map. The panoramic vision and laser radar fusion SLAM system is used to fuse the collected image information with the point cloud generated by the laser radar, which can perform more accurate positioning and mapping, and realize functions such as ranging and obstacle avoidance.
[0033] Figure 5 It is a schematic diagram of the imaging of a panoramic camera. After the light is double-returned through the panoramic annular lens 12, it passes through the aperture 17 and the relay lens 13 and finally forms an image on the image plane 14. The image includes an annular effective area 15 and a blind area 16. When the SLAM system based on panoramic vision and laser radar fusion extracts feature points from the image, it is necessary to avoid the blind area 16 to reduce mismatching. The panoramic camera used in this embodiment is based on a panoramic annular lens, and other lenses can also be used to achieve similar functions, which is not limited here.
[0034] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions, when executed by the processor, may create a device for implementing the functions / operations described in these block diagrams and / or flow charts.
[0035] Therefore, the panoramic vision and laser radar fusion SLAM system can take the form of a computer program product on a computer-readable storage medium storing instructions, which can be used by an instruction execution system or in combination with an instruction execution system. In the context of the present invention, a computer-readable storage medium can be any medium that can contain, store, transmit, propagate or transmit instructions. For example, a computer-readable storage medium can include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device, device or propagation medium. Specific examples of computer-readable storage media include: magnetic storage devices, such as magnetic tapes or hard disks (HDD); optical storage devices, such as compact disks (CD-ROM); memory, such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.
Claims
1. A smart car based on panoramic vision and laser radar fusion SLAM system, It is characterized in that It includes an upper aluminum alloy plate and a lower aluminum alloy plate, a panoramic camera, a laser radar, a camera hood, a camera bracket, a computer, a lithium battery, a motor drive module, a motor and wheels; wherein: The upper aluminum alloy plate and the lower aluminum alloy plate are fixedly connected by a plurality of aluminum profiles, and the height of the aluminum profiles is selected according to actual needs; There are four motors, which are respectively installed under the four corners of the lower aluminum alloy plate. The output shafts of the motors are connected to the four wheels of the smart car. By controlling the rotation of the four wheels, the smart car can be driven in different directions. The lithium battery and the motor drive module are placed in a storage box, which is placed in the middle of the upper surface of the lower aluminum alloy plate; the lithium battery is used to power the smart car to ensure the normal operation of each module; the motor drive module is equipped with a wireless network and Bluetooth module, which can be connected to a computer or mobile phone APP, as well as a wireless remote control to control the four motors; The camera bracket is installed in the middle of the upper surface of the upper aluminum alloy plate, and the camera hood and the panoramic camera are placed above the camera bracket and fixed to prevent shaking and blurring of the captured image; the camera bracket can adjust its height according to the field of view of the panoramic camera to avoid other devices on the smart car blocking its field of view; The computer is arranged behind the upper surface of the upper aluminum alloy plate and is used to receive and process the image sequence taken by the panoramic camera; The laser radar is installed in front of the upper surface of the upper aluminum alloy plate and connected to the computer to detect objects in a certain field of view in front of the smart car and generate a point cloud map; The computer is equipped with a panoramic vision and laser radar fusion SLAM system. The panoramic image sequence collected by the panoramic camera and the point cloud map generated by the laser radar are fused in the SLAM system, and the vehicle's posture is estimated and the scene map is reconstructed; at the same time, the distance measurement and obstacle avoidance functions are realized; The operation process of the panoramic vision and lidar fusion SLAM system is as follows: After the panoramic camera is connected to the computer, it collects continuous frames of images of the 360° scene around the smart car and then inputs them into the computer; the SLAM system processes these image sequences; first, a mask is used to avoid the blind area in the image and extract the feature points of the annular effective area; the laser radar scans the scene in front in real time, transmits the point cloud information to the computer, and fuses the laser point cloud information with the image information; If the system has not been initialized, the initialization process is performed: these feature points are matched between frames, and then the correctly matched feature point pairs are selected to estimate the initial pose of the current camera, and the initial map points in the environment are obtained based on the data fusion results with the laser point cloud; the initial pose of the camera and the initial map points are optimized using the local bundle adjustment method; After completing the initialization of the panoramic vision and lidar fusion SLAM system, the system uses the tracking model to track the existing map points based on the current frame image collected by the panoramic camera; estimates the current camera motion pose and updates the local map; then uses local BA to optimize the current pose and map points; When the panoramic vision and laser radar fusion SLAM system recognizes that the smart car returns to the position it has passed before, it determines whether the closed-loop detection condition is met based on the similarity between the current key frame and the key frame generated when it last passed the place. Since the panoramic camera has a 360° field of view, even if it returns to the position it has passed before from the opposite direction, it can ensure that there is enough similarity between the current key frame and the key frame generated when it last passed the place, thereby completing the closed-loop detection and correcting the scale drift. Then, global BA is performed to obtain more accurate motion posture and map points. When the smart car is driving, the laser point cloud map is integrated with the image information collected by the panoramic camera to obtain more accurate spatial point coordinates, thereby improving the positioning accuracy of the smart car. At the same time, it is judged whether there are obstacles ahead based on the acquired point cloud information. If an obstacle is identified, the computer will send instructions to the motor drive module based on the distance from the obstacle to the smart car, and control the movement of the wheels through the motor to achieve deceleration and turning of the smart car.
