Synchronous positioning and mapping method based on panoramic stereoscopic vision camera

Through the feature point matching and triangulation technology of panoramic stereoscopic vision cameras, the problem of insufficient positioning accuracy in the existing technology is solved, large viewing angles and map scale estimation is realized, and positioning accuracy and system stability are improved.

CN118918253APending Publication Date: 2024-11-08SOUTHERN UNIVERSITY OF SCIENCE AND TECHNOLOGY
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410952274.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art cannot simultaneously realize the problem of large viewing angles and able to estimate the real scale of the map, resulting in insufficient positioning accuracy.

Method used

Using the synchronous positioning and mapping method based on panoramic stereoscopic cameras, a fisheye binocular camera with non-parallel optical axis is set up to obtain the initial camera picture for feature points matching and triangulation, and the feature points in the common viewing area are determined. , build the initial map and track and locate it.

Benefits of technology

It realizes the estimation of large visual angles and real map scales, improves positioning accuracy and system stability, and is suitable for application scenarios such as robots, unmanned driving and VR/AR.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918253A_ABST
    Figure CN118918253A_ABST
Patent Text Reader

Abstract

According to the synchronous positioning and mapping method based on the panoramic stereoscopic vision camera, the panoramic stereoscopic vision camera comprises a plurality of fisheye cameras which are sequentially arranged and connected end to end, and every two adjacent fisheye cameras form a fisheye binocular camera with a non-parallel optical axis; the method comprises the following steps: performing feature point matching by acquiring initial camera pictures shot by two adjacent fisheye cameras in a panoramic stereoscopic vision camera to obtain common-view-area feature points; triangularizing the common-view-area feature points to determine three-dimensional coordinate points of the common-view-area feature points in a world coordinate system; and constructing an initial map, and tracking and positioning the panoramic stereoscopic vision camera based on the initial map. According to the method, the three-dimensional coordinates of the feature points are recovered by using the three-dimensional information of the panoramic camera, and the pose of the panoramic stereoscopic vision camera is positioned according to the three-dimensional coordinates of the feature points, so that synchronous positioning and mapping are realized, a large visual angle is realized, the real scale of a map can be estimated, and the positioning precision is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of synchronous positioning and mapping, and in particular to a synchronous positioning and mapping method based on a panoramic stereo vision camera. Background Art

[0002] Simultaneous localization and mapping (SLAM) technology is widely used, providing important basic environmental perception capabilities for upper-level applications in the fields of VR, unmanned driving, and robots. According to the different sensors used, SLAM technology can be divided into multiple implementation methods such as monocular cameras, binocular stereo cameras, and lidar. Since each sensor has its inherent advantages and disadvantages, for example, monocular and binocular stereo cameras are low-cost, but are easily affected by changes in illumination, and monocular cameras cannot obtain the true scale; LiDAR sensors can directly obtain distances, but cannot obtain texture information. SLAM systems based on different sensors also have their inherent advantages and disadvantages. In order to overcome the limitations of a single sensor and improve the robustness of the SLAM system, the hybrid SLAM technology that uses multiple sensors and combines inertial measurement units (IMUs) and global satellite navigation systems has become a research hotspot, but the increase in sensors also increases costs.

[0003] The prior art generally uses monocular or binocular SLAM technology, that is, uses a monocular or binocular camera as a SLAM system sensor. In addition, some use panoramic SLAM technology, that is, uses a panoramic camera as a sensor of the SLAM system.

[0004] The monocular SLAM system uses a monocular camera with a small viewing angle. It is easy to get lost during the tracking process, especially when there are large obstructions blocking the camera. In addition, the monocular SLAM system cannot correctly estimate the true scale of the map. The binocular SLAM system can correctly estimate the scale of the map, but the viewing angle is still small and it is easy to be blocked. It is easy to lose tracking in areas with less texture. The panoramic SLAM system mainly uses the panoramic images of the panoramic camera as the input of the SLAM system. The viewing angle is large, but because the principle is similar to that of the monocular camera, it is necessary to rely on the camera images at different times to build the map, which makes it impossible to estimate the true scale of the map.

[0005] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a synchronous positioning and mapping method based on a panoramic stereo vision camera is provided, aiming to solve the problem in the prior art that a large viewing angle cannot be achieved and the true scale of the map cannot be estimated at the same time.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A synchronous positioning and mapping method based on a panoramic stereo vision camera, wherein the panoramic stereo vision camera comprises a plurality of fisheye cameras arranged in sequence and connected end to end, and each adjacent two fisheye cameras form a fisheye binocular camera with non-parallel optical axes; the method comprises:

[0009] Acquire an initial camera image captured by each fisheye camera in the panoramic stereo vision camera;

[0010] Perform feature point matching on the initial camera images taken by two adjacent fisheye cameras, and use the successfully matched feature points as the common view area feature points;

[0011] Triangulating the feature points in the common view area to determine three-dimensional coordinate points of the feature points in the common view area in a world coordinate system;

[0012] An initial map is constructed based on each of the three-dimensional coordinate points, and tracking and positioning of the panoramic stereo vision camera is performed based on the initial map.

[0013] In one implementation of the present application, before obtaining the initial camera images taken by each fisheye camera in the panoramic stereo vision camera, the method further includes:

[0014] Calibrate the internal parameters of each fisheye camera in the panoramic stereo vision camera separately to obtain the calibrated internal parameters of each fisheye camera;

[0015] Calibrate adjacent fisheye cameras in pairs for the external parameters of the fisheye cameras in the panoramic stereo vision camera to obtain the calibrated external parameters of each fisheye camera;

[0016] Coordinate transformation is performed based on the calibration internal parameters and the calibration external parameters to obtain a coordinate transformation matrix between the camera coordinate system of each fisheye camera and the panoramic spherical coordinate system of the panoramic stereo vision camera.

[0017] In one implementation of the present application, the matching of feature points of the initial camera images taken by two adjacent fisheye cameras and using the successfully matched feature points as the common view area feature points includes:

[0018] Extracting ORB features from the initial camera images taken by each of the fisheye cameras;

[0019] According to each of the ORB features, ORB feature matching is performed on the camera images taken by two adjacent fisheye cameras, and the successfully matched ORB features are used as feature points of the common view area.

[0020] In one implementation of the present application, before triangulating the common view area feature point to determine the three-dimensional coordinate point of the common view area feature point in the world coordinate system, the method further includes:

[0021] A reference fisheye camera for each feature point in the common viewing area is determined, and a camera coordinate system of the reference fisheye camera is used as a target camera coordinate system, wherein the reference fisheye camera is one of the fisheye binocular cameras.

[0022] In one implementation of the present application, triangulating the common view area feature points to determine the three-dimensional coordinate points of the common view area feature points in the world coordinate system includes:

[0023] Triangulating each of the common view area feature points to obtain the three-dimensional coordinates of each of the common view area feature points in the target camera coordinate system;

[0024] Finding the coordinate transformation matrix between the camera coordinate system of each fisheye camera and the panoramic spherical coordinate system of the panoramic stereo vision camera, and obtaining the target coordinate transformation matrix between the target camera coordinate system and the panoramic spherical coordinate system;

[0025] According to the target coordinate transformation matrix, the three-dimensional coordinates of the feature points in the common view area in the target camera coordinate system are converted into three-dimensional coordinates in the panoramic spherical coordinate system;

[0026] The initial position and posture of the panoramic stereo vision camera is obtained, and the three-dimensional coordinates of the feature points in the common view area in the panoramic spherical coordinate system are converted into three-dimensional coordinate points in the world coordinate system according to the initial position and posture of the panoramic stereo vision camera.

[0027] In one implementation of the present application, tracking and positioning of a panoramic stereo vision camera based on the initial map includes:

[0028] When the current camera images captured by each fisheye camera are determined to be key frames, feature points of the current camera images captured by two adjacent fisheye cameras are matched, and the successfully matched feature points are used as feature points of the current common view area;

[0029] Triangulating the feature points of the current common view area, and converting the three-dimensional coordinates in the camera coordinate system obtained after triangulation into three-dimensional coordinates in the panoramic spherical coordinate system;

[0030] Obtaining a panoramic stereoscopic camera pose result of the current frame, and converting the three-dimensional coordinates of the feature points of the current common view area in the panoramic spherical coordinate system into the current three-dimensional coordinate points in the world coordinate system according to the panoramic stereoscopic camera pose result of the current frame;

[0031] Inserting each of the current three-dimensional coordinate points into the current map to obtain an updated map, and performing local optimization and global optimization on the updated map;

[0032] Tracking and positioning of panoramic stereo vision cameras based on the updated map;

[0033] The current map is obtained by iteratively updating the initial map based on the acquired key frames.

[0034] In one implementation of the present application, the synchronous positioning and mapping method based on a panoramic stereo vision camera further includes:

[0035] Each frame of the camera image taken by the panoramic stereo vision camera is obtained in real time through the network, and the panoramic stereo vision camera is tracked and positioned in real time based on each frame of the camera image.

[0036] The embodiment of the present application also provides a synchronous positioning and mapping device based on a panoramic stereo vision camera, wherein the panoramic stereo vision camera includes a plurality of fisheye cameras arranged in sequence and connected end to end, and each two adjacent fisheye cameras form a fisheye binocular camera with non-parallel optical axes; the device includes:

[0037] An acquisition module, used to acquire an initial camera image captured by each fisheye camera in the panoramic stereo vision camera;

[0038] A matching module is used to match feature points of initial camera images taken by two adjacent fisheye cameras, and use the successfully matched feature points as common view area feature points;

[0039] A determination module, used for triangulating the feature points of the common view area to determine the three-dimensional coordinate points of the feature points of the common view area in a world coordinate system;

[0040] A positioning module is used to construct an initial map based on each of the three-dimensional coordinate points, and to track and position the panoramic stereo vision camera based on the initial map.

[0041] An embodiment of the present application also provides a terminal, which includes: a memory, a processor, and a synchronous positioning and mapping program based on a panoramic stereo vision camera stored in the memory and executable on the processor, wherein the synchronous positioning and mapping program based on the panoramic stereo vision camera implements the steps of the synchronous positioning and mapping method based on the panoramic stereo vision camera as described above when executed by the processor.

[0042] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the synchronous positioning and mapping method based on the panoramic stereo vision camera as described above.

[0043] The present invention provides a synchronous positioning and mapping method based on a panoramic stereo vision camera, wherein the panoramic stereo vision camera includes a plurality of fisheye cameras arranged in sequence and connected end to end, and each two adjacent fisheye cameras form a fisheye binocular camera with non-parallel optical axes; the method includes: obtaining the initial camera images taken by each fisheye camera in the panoramic stereo vision camera; matching the feature points of the initial camera images taken by the two adjacent fisheye cameras, and using the successfully matched feature points as the common view area feature points; triangulating the common view area feature points to determine the three-dimensional coordinate points of the common view area feature points in the world coordinate system; constructing an initial map based on each of the three-dimensional coordinate points, and tracking and positioning the panoramic stereo vision camera based on the initial map. The present invention restores the three-dimensional coordinates of the feature points by using the stereo information of the panoramic camera, and locates the posture of the panoramic stereo vision camera according to the three-dimensional coordinates of the feature points, thereby realizing synchronous positioning and mapping, which not only realizes a large viewing angle but also can estimate the real scale of the map, thereby improving the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a panoramic stereo vision camera in the present invention;

[0045] Figure 2 It is a flow chart of a preferred embodiment of a synchronous positioning and mapping method based on a panoramic stereo vision camera of the present invention;

[0046] Figure 3 It is the overall process logic diagram of panoramic stereo SLAM in the present invention;

[0047] Figure 4 It is a functional principle block diagram of a preferred embodiment of a synchronous positioning and mapping device based on a panoramic stereo vision camera in the present invention;

[0048] Figure 5 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] Traditional visual SLAM technology uses a monocular camera, a binocular camera or a panoramic camera as input. The embodiment of the present application uses a unique panoramic camera as input to provide omnidirectional stereoscopic visual information for the SLAM system.

[0051] In order to obtain omnidirectional stereoscopic visual information, traditional panoramic cameras do not meet the requirements, or the common viewing area is insufficient, or the common viewing area is sufficient but the stereoscopic visual information cannot be obtained in real time. The embodiment of the present application designs a special panoramic stereoscopic vision camera. The panoramic stereoscopic vision camera includes a plurality of fisheye cameras arranged in sequence and connected end to end, and each adjacent two fisheye cameras form a fisheye binocular camera with non-parallel optical axes.

[0052] In one embodiment, the panoramic stereo vision camera is composed of 6 fisheye cameras, and the field of view of the fisheye camera is 120 degrees. In the panoramic stereo vision camera, in addition to providing a 360-degree panoramic video, each two adjacent fisheye cameras can form a fisheye binocular camera with non-parallel optical axes. The panoramic stereo vision camera is set to multiple, preferably 6, because the viewing angle of the fisheye camera used is 120 degrees, and the combination of 6 just makes the entire panoramic picture a common viewing area, while not generating too much common viewing area. Too many lenses can produce a larger common viewing area, and too few will reduce the common viewing area.

[0053] See also Figure 1 , the visible area of ​​a single camera is divided into three parts. Taking camera 0 as an example, the three visible areas are: the right common viewing area of ​​camera 0 and camera 1, the left common viewing area of ​​camera 0 and camera 5, and the non-common viewing area in the middle where only camera 0 is visible. The formation of the non-common viewing area is mainly due to the camera installation gap when the horizontal viewing angle of the fisheye camera is less than or equal to 120 degrees. Therefore, the non-common viewing area is very small. Alternatively, when the horizontal viewing angle of the fisheye camera is greater than 120 degrees, the non-common viewing area will decrease or even disappear. Therefore, most of the area in the panoramic picture is the common viewing area, which solves the problem of small common viewing area of ​​ordinary panoramic cameras.

[0054] In addition, the camera's stereoscopic vision information can be encoded in real time by an embedded edge computing device and acquired in real time through the network, solving the problem of not being able to acquire stereoscopic vision information in real time.

[0055] See also Figure 2 , Figure 2 Flowchart of the synchronous positioning and mapping method based on panoramic stereo vision camera in the present invention. Figure 2 As shown, the synchronous positioning and mapping method based on a panoramic stereo vision camera according to an embodiment of the present invention includes:

[0056] Step S100: obtaining an initial camera image captured by each fisheye camera in the panoramic stereo vision camera.

[0057] Specifically, when the system is initialized, the first frame of 6 camera images of the panoramic stereo camera are obtained, and the first frame of 6 camera images are used as the initial camera images.

[0058] In one embodiment of the present application, before step S100, the following steps are further included:

[0059] Step S10, calibrating the internal parameters of each fisheye camera in the panoramic stereo vision camera separately to obtain the calibrated internal parameters of each fisheye camera;

[0060] Step S20, calibrating the external parameters of the fisheye cameras in the panoramic stereo vision camera by calibrating adjacent fisheye cameras in pairs to obtain the calibrated external parameters of each fisheye camera;

[0061] Step S30: performing coordinate transformation based on the calibration internal parameters and the calibration external parameters to obtain a coordinate transformation matrix between the camera coordinate system of each fisheye camera and the panoramic spherical coordinate system of the panoramic stereo vision camera.

[0062] Specifically, when performing synchronous tracking and mapping, it is necessary to know the coordinate transformation matrix between each fisheye camera and the panoramic stereo camera coordinates. Since the camera uses a fisheye camera, the ordinary camera model cannot be well represented, so the simplified KannalaBrandt8 fisheye camera model is used. The intrinsic parameters of each camera are calibrated separately using kalibr, and the external parameters of the camera are also calibrated in pairs using the same software. Finally, through coordinate transformation, it is converted into the coordinate transformation matrix of each camera relative to the panoramic stereo camera coordinates.

[0063] The embodiment of the present application obtains the mathematical transformation between a series of three-dimensional points and corresponding two-dimensional image points through a camera calibration process. Through this process, the intrinsic and extrinsic parameters of the camera can be calculated, thereby establishing a connection between the image coordinate system and the world coordinate system. This connection can ensure the accuracy and precision of image stitching and improve the quality of panoramic images.

[0064] like Figure 2 As shown, the synchronous positioning and mapping method based on the panoramic stereo vision camera described in this embodiment also includes:

[0065] Step S200: performing feature point matching on initial camera images taken by two adjacent fisheye cameras, and using successfully matched feature points as common view area feature points.

[0066] In the embodiment of the present application, the step S200 specifically includes:

[0067] Step S210, extracting ORB features from the initial camera images taken by each of the fisheye cameras;

[0068] Step S220: performing ORB feature matching on camera images captured by two adjacent fisheye cameras according to each of the ORB features, and using successfully matched ORB features as feature points of the common view area.

[0069] Specifically, ORB features are extracted from the six camera images of the first frame, and two adjacent images in the six images are combined into six binocular stereo vision systems. ORB features are matched for each group of common view areas. ORB feature matching is based on the ORB (Oriented FAST and Rotated BRIEF) algorithm, which combines the FAST corner detection algorithm and the BRIEF descriptor algorithm to achieve efficient feature extraction and matching speed.

[0070] like Figure 1 As shown, the synchronous positioning and mapping method based on the panoramic stereo vision camera described in this embodiment also includes:

[0071] Step S300: triangulate the feature points in the common view area to determine the three-dimensional coordinate points of the feature points in the common view area in a world coordinate system.

[0072] The three-dimensional coordinate point is a 3D point.

[0073] In an embodiment of the present application, before step S300, the method further includes: determining a reference fisheye camera for each feature point in the common view area, and using the camera coordinate system of the reference fisheye camera as the target camera coordinate system, wherein the reference fisheye camera is one of the fisheye binocular cameras.

[0074] Specifically, the camera on the left side of the fisheye binocular camera may be used as a reference fisheye camera, or the camera on the right side of the fisheye binocular camera may be used as a reference fisheye camera, so as to determine the three-dimensional coordinates of the feature points.

[0075] In one embodiment of the present application, step S300 specifically includes:

[0076] Step S310, triangulating each of the common view area feature points to obtain the three-dimensional coordinates of each of the common view area feature points in the target camera coordinate system;

[0077] Step S320, searching for a coordinate transformation matrix between the camera coordinate system of each fisheye camera and the panoramic spherical coordinate system of the panoramic stereo vision camera, and obtaining a target coordinate transformation matrix between the target camera coordinate system and the panoramic spherical coordinate system;

[0078] Step S330: converting the three-dimensional coordinates of the feature points in the common view area in the target camera coordinate system into three-dimensional coordinates in the panoramic spherical coordinate system according to the target coordinate transformation matrix;

[0079] Step S340: Acquire the initial position and posture of the panoramic stereo vision camera, and convert the three-dimensional coordinates of the feature points in the common view area in the panoramic spherical coordinate system into three-dimensional coordinate points in the world coordinate system according to the initial position and posture of the panoramic stereo vision camera.

[0080] Specifically, the successfully matched feature points are triangulated, that is, their three-dimensional coordinates in the binocular left camera or the binocular right camera are calculated. The three-dimensional coordinate representation of the feature point in the panoramic stereo camera is obtained through the coordinate transformation matrix of the binocular left camera relative to the panoramic stereo camera or the coordinate transformation matrix of the binocular right camera relative to the panoramic stereo camera. Since the first frame is set to the world coordinate system of the map by default, the three-dimensional coordinates of the feature point in the panoramic stereo camera are also the coordinates in the world coordinate system. In one embodiment, all six groups of feature points are triangulated, that is, the panoramic feature points are triangulated.

[0081] Triangulation is based on the principles of parallax and triangulation. The three-dimensional coordinates of the object under test can be derived through the triangles formed between the image planes of the two cameras and the object under test. This enables the binocular camera to not only capture two-dimensional images, but also obtain depth information, achieving non-contact three-dimensional information acquisition. The triangulation method has the characteristics of high precision. By using the known baseline distance, the focal length of the camera, and the pixel coordinates, the precise three-dimensional coordinates of the object under test can be calculated.

[0082] The embodiment of the present application triangulates the feature points of each common view area so that the panoramic stereo vision camera can obtain the three-dimensional coordinates of the feature points in the world coordinate system through several camera images at a certain moment, and then construct a map, while achieving a large viewing angle and being able to estimate the true scale of the map.

[0083] like Figure 2 As shown, the synchronous positioning and mapping method based on the panoramic stereo vision camera described in this embodiment also includes:

[0084] Step S400: constructing an initial map based on each of the three-dimensional coordinate points, and tracking and positioning the panoramic stereo vision camera based on the initial map.

[0085] Specifically, all six groups of feature points are triangulated, that is, the panoramic feature points are triangulated. At this time, the map is initialized successfully and the initial map is obtained. The SLAM system enters the tracking step. In one embodiment, the second frame of the panoramic stereo camera is obtained, and the ORB features are extracted and matched with the existing feature points in the initial map. The current panoramic stereo camera posture is optimized using the bundle adjustment algorithm based on the successfully matched feature points. During optimization, the 3D feature points in the map are transformed to the camera coordinate system through the initial values ​​of the panoramic spherical coordinates and the calibrated camera coordinates and panoramic spherical coordinate conversion matrix. The image coordinates are obtained by projecting the KannalaBrandt8 fisheye camera model, and the coordinate measurement values ​​(u, v) of the image feature points that match the 3D feature points in the image are used to calculate the error. The reprojection error function is as follows:

[0086]

[0087] Where e is the reprojection error, n is the number of all 3D feature points, and X w is the three-dimensional coordinate of the 3D feature point in the world coordinate system, T pw is the panoramic spherical camera pose, is the coordinate transformation matrix from the panoramic spherical coordinates to the camera m, K m (·) is the projection function of the m-th camera.

[0088] The bundle adjustment method can simultaneously refine motion and structural parameters by minimizing the projection error between observed and predicted image points. For each successfully matched feature point, the bundle adjustment method adjusts the camera's pose and the 3D point coordinates in the scene so that the sum of the deviations between the projected point coordinates of all 3D points in each view and their corresponding observed values ​​is minimized. This optimization process significantly improves the accuracy of 3D reconstruction and camera pose estimation.

[0089] The embodiment of the present application uses the bundle adjustment method to optimize the current panoramic stereo camera posture, which can significantly improve the accuracy, robustness and ability to handle complex scenes of three-dimensional reconstruction and camera posture estimation.

[0090] In one embodiment of the present application, after step S400, the following steps are further included:

[0091] When the current camera images captured by each fisheye camera are determined to be key frames, feature points of the current camera images captured by two adjacent fisheye cameras are matched, and the successfully matched feature points are used as feature points of the current common view area;

[0092] Triangulating the feature points of the current common view area, and converting the three-dimensional coordinates in the camera coordinate system obtained after triangulation into three-dimensional coordinates in the panoramic spherical coordinate system;

[0093] Obtaining a panoramic stereoscopic camera pose result of the current frame, and converting the three-dimensional coordinates of the feature points of the current common view area in the panoramic spherical coordinate system into the current three-dimensional coordinate points in the world coordinate system according to the panoramic stereoscopic camera pose result of the current frame;

[0094] Inserting each of the current three-dimensional coordinate points into the current map to obtain an updated map, and performing local optimization and global optimization on the updated map;

[0095] Tracking and positioning of panoramic stereo vision cameras based on the updated map;

[0096] The current map is obtained by iteratively updating the initial map based on the acquired key frames.

[0097] Specifically, the above three-dimensional coordinates are all 3D points. After obtaining the panoramic spherical camera posture, it is determined whether the frame is inserted into the map as a key frame. The determination of whether to insert the map as a key frame can be a combination of a series of conditions, such as whether the local optimization thread is idle, whether the number of feature points successfully tracked in the current frame is less than a threshold, etc.

[0098] The local optimization module uses binocular triangulation to restore the 3D coordinates of the feature points of the inserted keyframes, and adds the newly created map points to the map according to the pose of the keyframes. Then the local optimization module and the global optimization module perform local and global optimization on the pose of the keyframes and feature points. The overall process is as follows: Figure 3 As shown in the figure. The initialization phase is executed only once when the system starts to initialize the map. In the tracking phase, keyframe-based matching or fast matching based on motion model can be used to solve the pose.

[0099] The newly created map point refers to the feature point after binocular triangulation, that is, the 3D point. In addition, since the position of the panoramic sphere is known at this time, the position of the six sub-cameras relative to the panoramic sphere is also known, so the transformation matrix between the camera i in any key frame n and the camera j in the key frame m can be obtained by transformation. Therefore, the sub-cameras i and j between two different frames can also form binocular triangulation feature points, and the three-dimensional coordinates of the feature points, that is, 3D points, can be obtained.

[0100] The embodiment of the present application can expand the initial map by continuing to triangulate the common view area feature points of the key frames, thereby improving the accuracy of map construction and preventing tracking failures.

[0101] In an embodiment of the present application, the synchronous positioning and mapping method based on the panoramic stereo vision camera also includes: acquiring each frame of camera image taken by the panoramic stereo vision camera in real time through the network, and performing real-time tracking and positioning of the panoramic stereo vision camera based on each frame of camera image.

[0102] That is to say, the camera stereoscopic vision information obtained by the panoramic stereoscopic vision camera in the embodiment of the present application is encoded in real time by the embedded edge computing device and can be obtained in real time through the network, which solves the problem that ordinary panoramic cameras cannot obtain stereoscopic vision information in real time.

[0103] In this way, the embodiment of the present application is based on the SLAM method of a panoramic stereo vision camera. Compared with monocular SLAM, panoramic stereo SLAM solves the problem that a monocular camera cannot estimate the true scale of the map, while greatly increasing the viewing angle and improving the stability of the SLAM system; compared with binocular SLAM, it increases the viewing angle and improves the system stability; compared with panoramic SLAM, it solves the problem of being unable to estimate the true scale of the map; since 3D coordinate recovery uses binocular triangulation recovery, it is more accurate than monocular panoramic recovery, thereby increasing the positioning accuracy.

[0104] Therefore, the embodiments of the present application can not only accurately estimate the true scale of the map, but also due to the advantage of the 360-degree field of view, it has a stronger ability to resist low texture, light changes, and occlusion, making the system more robust and can be used in application scenarios such as robots, unmanned driving, and VR / AR. The embodiments of the present application can better perceive the surrounding environment while ensuring low cost, provide higher-precision positioning and more accurate map construction capabilities.

[0105] In one embodiment, if Figure 4 As shown, based on the above-mentioned synchronous positioning and mapping method based on a panoramic stereo vision camera, the present invention also provides a synchronous positioning and mapping device based on a panoramic stereo vision camera, wherein the panoramic stereo vision camera comprises a plurality of fisheye cameras arranged in sequence and connected end to end, and each adjacent two fisheye cameras form a fisheye binocular camera with non-parallel optical axes; the device comprises:

[0106] An acquisition module 100 is used to acquire an initial camera image captured by each fisheye camera in the panoramic stereo vision camera;

[0107] A matching module 200 is used to match feature points of initial camera images taken by two adjacent fisheye cameras, and use the successfully matched feature points as common view area feature points;

[0108] A determination module 300 is used to triangulate the feature points in the common view area to determine the three-dimensional coordinate points of the feature points in the common view area in a world coordinate system;

[0109] The positioning module 400 is used to construct an initial map based on each of the three-dimensional coordinate points, and to track and position the panoramic stereo vision camera based on the initial map.

[0110] Figure 5 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0111] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0112] When the processor 502 executes the program, the synchronous positioning and mapping method based on the panoramic stereo vision camera provided in the above embodiment is implemented.

[0113] Furthermore, the terminal further includes:

[0114] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0115] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0116] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0117] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0118] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0119] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0120] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned synchronous positioning and mapping method based on a panoramic stereo vision camera is implemented.

[0121] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0123] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0125] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0126] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0128] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0129] In summary, the present invention discloses a synchronous positioning and mapping method based on a panoramic stereo vision camera, wherein the panoramic stereo vision camera includes a plurality of fisheye cameras arranged in sequence and connected end to end, and each two adjacent fisheye cameras form a fisheye binocular camera with non-parallel optical axes; the method includes: obtaining the initial camera images taken by each fisheye camera in the panoramic stereo vision camera; matching the feature points of the initial camera images taken by two adjacent fisheye cameras, and using the successfully matched feature points as the common view area feature points; triangulating the common view area feature points to determine the three-dimensional coordinate points of the common view area feature points in the world coordinate system; constructing an initial map based on each of the three-dimensional coordinate points, and tracking and positioning the panoramic stereo vision camera based on the initial map. The present invention restores the three-dimensional coordinates of the feature points by using the stereo information of the panoramic camera, and locates the posture of the panoramic stereo vision camera according to the three-dimensional coordinates of the feature points, thereby realizing synchronous positioning and mapping, which not only realizes a large viewing angle but also can estimate the real scale of the map, thereby improving the positioning accuracy.

[0130] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A synchronous positioning and mapping method based on panoramic stereo vision camera, It is characterized in that The panoramic stereo vision camera comprises a plurality of fisheye cameras arranged in sequence and connected end to end, and each two adjacent fisheye cameras form a fisheye binocular camera with non-parallel optical axes; the method comprises: Acquire an initial camera image captured by each fisheye camera in the panoramic stereo vision camera; Perform feature point matching on the initial camera images taken by two adjacent fisheye cameras, and use the successfully matched feature points as the common view area feature points; Triangulating the feature points in the common view area to determine three-dimensional coordinate points of the feature points in the common view area in a world coordinate system; An initial map is constructed based on each of the three-dimensional coordinate points, and tracking and positioning of the panoramic stereo vision camera is performed based on the initial map.

2. The method for synchronous positioning and mapping based on a panoramic stereo vision camera according to claim 1, It is characterized in that Before obtaining the initial camera images taken by each fisheye camera in the panoramic stereo vision camera, the method further includes: Calibrate the internal parameters of each fisheye camera in the panoramic stereo vision camera separately to obtain the calibrated internal parameters of each fisheye camera; Calibrate adjacent fisheye cameras in pairs for the external parameters of the fisheye cameras in the panoramic stereo vision camera to obtain the calibrated external parameters of each fisheye camera; Coordinate transformation is performed based on the calibration internal parameters and the calibration external parameters to obtain a coordinate transformation matrix between the camera coordinate system of each fisheye camera and the panoramic spherical coordinate system of the panoramic stereo vision camera.

3. The method for synchronous positioning and mapping based on a panoramic stereo vision camera according to claim 1, It is characterized in that The matching of feature points of the initial camera images taken by two adjacent fisheye cameras and taking the successfully matched feature points as the common view area feature points includes: Extracting ORB features from the initial camera images taken by each of the fisheye cameras; According to each of the ORB features, ORB feature matching is performed on the camera images taken by two adjacent fisheye cameras, and the successfully matched ORB features are used as feature points of the common view area.

4. The method for synchronous positioning and mapping based on a panoramic stereo vision camera according to claim 2, It is characterized in that The method further includes triangulating the feature points of the common view area to determine the three-dimensional coordinate points of the feature points of the common view area in the world coordinate system: A reference fisheye camera for each feature point in the common viewing area is determined, and a camera coordinate system of the reference fisheye camera is used as a target camera coordinate system, wherein the reference fisheye camera is one of the fisheye binocular cameras.

5. The method for synchronous positioning and mapping based on a panoramic stereo vision camera according to claim 4, It is characterized in that Triangulating the common view area feature points to determine the three-dimensional coordinate points of the common view area feature points in the world coordinate system, including: Triangulating each of the common view area feature points to obtain the three-dimensional coordinates of each of the common view area feature points in the target camera coordinate system; Finding the coordinate transformation matrix between the camera coordinate system of each fisheye camera and the panoramic spherical coordinate system of the panoramic stereo vision camera, and obtaining the target coordinate transformation matrix between the target camera coordinate system and the panoramic spherical coordinate system; According to the target coordinate transformation matrix, the three-dimensional coordinates of the feature points in the common view area in the target camera coordinate system are converted into three-dimensional coordinates in the panoramic spherical coordinate system; The initial position and posture of the panoramic stereo vision camera is obtained, and the three-dimensional coordinates of the feature points in the common view area in the panoramic spherical coordinate system are converted into three-dimensional coordinate points in the world coordinate system according to the initial position and posture of the panoramic stereo vision camera.

6. The method for synchronous positioning and mapping based on a panoramic stereo vision camera according to claim 5, It is characterized in that Tracking and positioning the panoramic stereo vision camera based on the initial map also includes: When the current camera images captured by each fisheye camera are determined to be key frames, feature points of the current camera images captured by two adjacent fisheye cameras are matched, and the successfully matched feature points are used as feature points of the current common view area; Triangulating the feature points of the current common view area, and converting the three-dimensional coordinates in the camera coordinate system obtained after triangulation into three-dimensional coordinates in the panoramic spherical coordinate system; Obtaining a panoramic stereoscopic camera pose result of the current frame, and converting the three-dimensional coordinates of the feature points of the current common view area in the panoramic spherical coordinate system into the current three-dimensional coordinate points in the world coordinate system according to the panoramic stereoscopic camera pose result of the current frame; Inserting each of the current three-dimensional coordinate points into the current map to obtain an updated map, and performing local optimization and global optimization on the updated map; Tracking and positioning of panoramic stereo vision cameras based on the updated map; The current map is obtained by iteratively updating the initial map based on the acquired key frames.

7. The method for synchronous positioning and mapping based on a panoramic stereo vision camera according to claim 1, It is characterized in that The synchronous positioning and mapping method based on the panoramic stereo vision camera also includes: Each frame of the camera image taken by the panoramic stereo vision camera is obtained in real time through the network, and the panoramic stereo vision camera is tracked and positioned in real time based on each frame of the camera image.

8. A synchronous positioning and mapping device based on a panoramic stereo vision camera, It is characterized in that The panoramic stereo vision camera comprises a plurality of fisheye cameras arranged in sequence and connected end to end, and each adjacent two fisheye cameras form a fisheye binocular camera with non-parallel optical axes; the device comprises: An acquisition module, used to acquire an initial camera image captured by each fisheye camera in the panoramic stereo vision camera; A matching module is used to match feature points of initial camera images taken by two adjacent fisheye cameras, and use the successfully matched feature points as common view area feature points; A determination module, used for triangulating the feature points of the common view area to determine the three-dimensional coordinate points of the feature points of the common view area in a world coordinate system; A positioning module is used to construct an initial map based on each of the three-dimensional coordinate points, and to track and position the panoramic stereo vision camera based on the initial map.

9. A terminal, It is characterized in that include: A memory, a processor, and a synchronous positioning and mapping program based on a panoramic stereo vision camera stored in the memory and executable on the processor, wherein the synchronous positioning and mapping program based on a panoramic stereo vision camera, when executed by the processor, implements the steps of the synchronous positioning and mapping method based on a panoramic stereo vision camera as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the simultaneous positioning and mapping method based on a panoramic stereo vision camera as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Omnidirectional vision positioning system and method and unmanned aerial vehicle

    CN121089720A