Positioning method and device, computer device and computer-readable storage medium

Abstract

This disclosure relates to a positioning method and apparatus, a computer device, and a computer-readable storage medium. The positioning method includes: pre-processing radar and visual mapping using radar and visual sensors; combining radar positioning with visual positioning, and using the visual positioning results for navigation on a grid map generated by radar positioning. This disclosure integrates laser SLAM and visual SLAM, leveraging their complementary advantages to address inherent operational challenges of both, providing a low-cost and stable positioning solution for navigational objects.

Description

Technical Field

[0001] The present disclosure relates to the field of positioning, and in particular to a positioning method and device, a computer device, and a computer-readable storage medium. Background Art

[0002] LiDAR-based indoor positioning technology. LiDAR is widely used for indoor mobile robot positioning due to its accurate ranging information. Currently, the mainstream positioning method uses laser data and grid map matching to perform positioning. This method creates a search window around the predicted current pose, creates multiple candidate poses within the search window, and determines the most suitable positioning pose based on the matching scores.

[0003] Vision-based indoor positioning technology. Vision-based positioning is also commonly known as visual SLAM (Simultaneous Localization and Mapping) technology. Visual SLAM utilizes multi-view geometry theory to locate the camera based on the image information captured by the camera and simultaneously construct a map of the surrounding environment. Visual SLAM primarily includes visual odometry, back-end optimization, loop closure detection, and mapping. Visual odometry studies the transformation relationship between image frames to achieve real-time pose tracking. It processes the input image, calculates pose changes, and obtains the motion relationship between cameras. However, over time, errors accumulate due to only estimating the motion between two images. The back-end primarily uses optimization methods to reduce the error of the entire framework (including camera pose and spatial map points). Loop closure detection, also known as loop closure detection, primarily uses the similarity between images to determine whether a previous position has been reached, thereby eliminating accumulated errors and obtaining a globally consistent trajectory and map. Mapping involves creating a map corresponding to the task requirements based on the estimated trajectory. Summary of the Invention

[0004] Related technologies: Problems with radar-based indoor positioning technology: some low-cost radars have limited ranging range and cannot obtain effective ranging information in large-scale scenarios; laser SLAM will produce motion degradation problems when facing long corridor environments; due to the small amount of radar information, laser SLAM is generally less likely to produce loops than visual SLAM.

[0005] Problems with indoor positioning technology based on visual sensors: The positioning accuracy of visual SLAM will decrease when facing weak texture environments such as white walls; visual sensors are generally sensitive to light, which causes visual SLAM to work in environments with large lighting changes, and the positioning stability will deteriorate; the constructed map cannot be directly used for navigation objects.

[0006] In view of at least one of the above technical problems, the present disclosure provides a positioning method and device, a computer device and a computer-readable storage medium, which integrate laser SLAM and visual SLAM to complement each other's advantages and solve the problems encountered by laser SLAM and visual SLAM themselves during their work.

[0007] According to one aspect of the present disclosure, a positioning method is provided, including:

[0008] Performing radar mapping and visual mapping using radar and visual sensors, wherein the visual mapping includes keyframe poses;

[0009] The radar positioning and the visual positioning are combined based on the key frame pose, and the visual positioning result is used for navigation on the map obtained by the radar positioning.

[0010] In some embodiments of the present disclosure, the pre-use of radar and visual sensors to perform radar mapping and visual mapping includes:

[0011] Radar and visual sensors are used simultaneously for mapping. Radar mapping produces a grid map for positioning and navigation, while visual mapping produces a visual map.

[0012] Bind the keyframe pose provided by visual mapping to the radar pose provided by radar mapping.

[0013] In some embodiments of the present disclosure, combining radar positioning with visual positioning based on the keyframe pose and using the visual positioning result for navigation on a map obtained by radar positioning includes:

[0014] Determine the candidate keyframe poses and current frame pose under the visual trajectory;

[0015] Convert the candidate key frame poses and current frame poses under the visual track to the candidate key frame poses and current frame poses under the radar track;

[0016] According to the candidate key frame pose under the radar track and the current frame pose, determine the pose transformation matrix from the candidate key frame under the radar track to the current frame;

[0017] The preliminary pose of the navigation object under the radar track is determined according to the pose transformation matrix and the radar pose bound to the key frame pose.

[0018] In some embodiments of the present disclosure, combining radar positioning with visual positioning based on the key frame pose, and using the visual positioning result for navigation on a map obtained by radar positioning further includes:

[0019] The preliminary pose of the navigation object with 6 degrees of freedom is projected onto 3 degrees of freedom to determine the pose of the navigation object in the grid map coordinate system.

[0020] In some embodiments of the present disclosure, determining the current frame pose under the visual trajectory includes:

[0021] Load the visual map;

[0022] Extract feature points from the current frame image of the visual map and retrieve candidate key frames in the mapping database based on the current frame image descriptor;

[0023] Visual relocalization is performed based on the feature point information of the current frame and the candidate key frames to obtain the current frame pose under the visual trajectory.

[0024] In some embodiments of the present disclosure, determining candidate key frame poses under the visual trajectory includes:

[0025] According to the candidate key frame rotation matrix and the global position of the candidate key frame under the visual track, the candidate key frame pose under the visual track is determined.

[0026] In some embodiments of the present disclosure, converting the candidate key frame poses under the visual track into the candidate key frame poses under the radar track includes:

[0027] Determine the candidate key frame rotation matrix under the radar trajectory based on the candidate key frame rotation matrix under the visual trajectory and the extrinsic rotation matrix between the visual sensor and the radar;

[0028] Calculate the rotation matrix between the visual track and the radar track;

[0029] Determine the global position of the candidate key frame under the radar track according to the global position of the candidate key frame under the visual track and the rotation matrix between the visual track and the radar track;

[0030] The candidate key frame pose under the radar track is determined according to the candidate key frame global position under the radar track and the candidate key frame rotation matrix under the radar track.

[0031] In some embodiments of the present disclosure, determining the candidate key frame rotation matrix under the radar trajectory based on the candidate key frame rotation matrix under the visual trajectory and the extrinsic rotation matrix between the visual sensor and the radar includes:

[0032] Determine the pose of the candidate key frame under the visual track according to the candidate key frame rotation matrix and the global position of the candidate key frame under the visual track;

[0033] The candidate key frame rotation matrix under the radar trajectory is determined according to the candidate key frame rotation matrix under the visual trajectory and the extrinsic rotation matrix between the visual sensor and the radar.

[0034] In some embodiments of the present disclosure, converting the current frame pose under the visual track into the current frame pose under the radar track includes:

[0035] Determine the current frame rotation matrix under the radar track based on the current frame rotation matrix under the visual track and the extrinsic rotation matrix between the visual sensor and the radar;

[0036] Calculate the rotation matrix between the visual track and the radar track;

[0037] Determine the current frame global position under the radar track based on the current frame global position under the visual track and the rotation matrix between the visual track and the radar track;

[0038] The current frame pose under the radar track is determined according to the current frame global position under the radar track and the current frame rotation matrix under the radar track.

[0039] According to another aspect of the present disclosure, there is provided a positioning device, comprising:

[0040] a fusion mapping module configured to perform radar mapping and visual mapping using radar and visual sensors, wherein the visual mapping includes keyframe poses;

[0041] The fusion positioning module is configured to combine the radar positioning and the visual positioning based on the key frame posture, and use the visual positioning result for navigation on the map obtained by the radar positioning.

[0042] In some embodiments of the present disclosure, a fusion mapping module is configured to simultaneously use radar and visual sensors for mapping, wherein radar mapping obtains a map for positioning and navigation, and visual mapping obtains a visual map; and the key frame pose provided by the visual mapping is bound to the radar pose provided by the radar mapping.

[0043] In some embodiments of the present disclosure, the fusion positioning module is configured to determine the candidate key frame poses and the current frame poses under the visual track; convert the candidate key frame poses and the current frame poses under the visual track into the candidate key frame poses and the current frame poses under the radar track; determine the pose transformation matrix from the candidate key frame under the radar track to the current frame based on the candidate key frame poses and the current frame poses under the radar track; determine the preliminary pose of the navigation object under the radar track based on the pose transformation matrix and the radar pose bound to the key frame pose.

[0044] In some embodiments of the present disclosure, the fusion positioning module is further configured to project the preliminary pose of the navigation object with 6 degrees of freedom onto 3 degrees of freedom to determine the pose of the navigation object in the grid map coordinate system.

[0045] In some embodiments of the present disclosure, the positioning device is used to perform operations to implement the positioning method described in any of the above embodiments.

[0046] According to another aspect of the present disclosure, there is provided a computer device, comprising:

[0047] a memory configured to store instructions;

[0048] The processor is configured to execute the instructions so that the computer device performs operations to implement the positioning method as described in any of the above embodiments.

[0049] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the positioning method as described in any of the above embodiments is implemented.

[0050] The present invention integrates laser SLAM and visual SLAM to complement each other's advantages, solves the problems encountered by laser SLAM and visual SLAM themselves during their work, and provides a low-cost and stable positioning solution for navigation objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 Illustration of the trajectory and grid map visualization using both radar and vision sensors on the same navigation object.

[0053] Figure 2 Schematic diagram of some embodiments of the positioning method disclosed herein.

[0054] Figure 3 Schematic diagram of some embodiments of the laser vision fusion mapping method disclosed in the present invention.

[0055] Figure 4 Schematic diagram of some embodiments of the laser vision fusion positioning method disclosed in the present invention.

[0056] Figure 5 Schematic diagrams of other embodiments of the laser vision fusion positioning method disclosed in the present invention.

[0057] Figure 6 This is an effect diagram of the trajectory after fusion positioning in some embodiments of the present disclosure.

[0058] Figure 7 Schematic diagrams of some embodiments of the positioning device disclosed herein.

[0059] Figure 8 This is a schematic structural diagram of a computer device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0061] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0062] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0063] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0064] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0065] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0066] Figure 1 The figure shows the trajectory and grid map visualized by using radar mapping and visual sensor mapping on the same navigation object. The navigation object can be a visual sensor, the radar can be a lidar, and the visual sensor can be a camera. Figure 1As shown, track 1 is the track left by laser SLAM; the light-colored portion 4 is the occupancy grid map created by laser SLAM, which is used to navigate the navigation object; track 2 is the track left by visual SLAM. Although both laser SLAM and visual SLAM describe the movement of the navigation object, due to the different installation positions and angles of the radar and vision sensors and the different scales of environmental description, the positioning tracks of the two do not overlap in the world coordinate system and are generally rotated and scaled. This means that if the radar is not working properly, using the visual sensor directly for positioning will not provide the navigation object with the position and pose in the navigation map coordinate system.

[0067] The present disclosure aims to provide a laser and vision fusion positioning solution. When a problem cannot be solved by using only laser or vision for positioning, it can smoothly switch to another sensor for positioning.

[0068] The mainstream approach to indoor object navigation involves path planning based on an occupancy grid map to control robot motion. This technology, based on LiDAR-based positioning and navigation solutions, is typically divided into two parts: mapping and positioning and navigation. Mapping uses LiDAR to create a two-dimensional occupancy grid map of the environment. Positioning matches LiDAR data with the occupancy grid map to determine the current position of the navigation object in the coordinate system of the occupancy grid map. Navigation involves path planning on the occupancy grid map from the current position obtained by positioning to the target point, controlling the robot's movement to the designated destination.

[0069] Figure 2 Schematic diagrams of some embodiments of the positioning method disclosed herein. Preferably, this embodiment can be performed by the positioning device disclosed herein or the computer device disclosed herein. The method may include steps 1 and 2, wherein:

[0070] Step 1: Perform laser vision fusion mapping.

[0071] In some embodiments of the present disclosure, step 1 may include: performing radar mapping and visual mapping using a radar and a visual sensor, wherein the radar may be a lidar and the visual sensor may be a camera.

[0072] In some embodiments of the present disclosure, step 1 may include: using radar and visual sensors to build maps simultaneously, wherein radar mapping obtains a grid map for positioning and navigation, and visual mapping obtains a visual map; and binding the key frame pose provided by the visual mapping to the radar pose provided by the radar mapping.

[0073] Figure 3 Schematic diagram of some embodiments of the laser vision fusion mapping method disclosed in the present invention. Figure 3 As shown, the laser vision fusion mapping method disclosed in the present invention (for example Figure 2Step 1) of the embodiment may include steps 11 to 14, wherein:

[0074] Step 11: During mapping, radar and visual sensors are used simultaneously for mapping, wherein radar mapping provides radar poses and visual mapping provides keyframe poses, wherein the radar poses may be lidar poses.

[0075] Step 12: Search for the nearest radar pose by timestamp near each visual keyframe for pose binding.

[0076] Step 13: When saving the visual map, the radar pose corresponding to the visual keyframe is also saved.

[0077] Step 14: radar mapping saves the occupancy grid map for radar positioning and navigation object navigation.

[0078] Step 2: Perform laser-vision fusion positioning. Combine radar positioning with vision positioning, and use the vision positioning results for navigation on the grid map obtained by radar positioning.

[0079] Figure 4 Schematic diagram of some embodiments of the laser vision fusion positioning method disclosed in the present invention. Figure 4 As shown, the laser vision fusion mapping method disclosed in the present invention (for example Figure 2 Step 2) of the embodiment may include steps 21 to 25, wherein:

[0080] Step 21: Determine the candidate key frame poses and the current frame pose under the visual trajectory.

[0081] Step 22: Convert the candidate key frame poses and the current frame pose under the visual track to the candidate key frame poses and the current frame pose under the radar track.

[0082] Step 23: Determine the pose transformation matrix from the candidate key frame to the current frame based on the candidate key frame pose and the current frame pose under the radar track.

[0083] Step 24: Determine a preliminary pose of the navigation object under the radar track according to the pose transformation matrix and the radar pose bound to the key frame pose, wherein the navigation object may be a visual sensor.

[0084] Step 25 : Project the preliminary pose of the navigation object with 6 degrees of freedom onto 3 degrees of freedom to determine the pose of the navigation object in the navigation grid map coordinate system.

[0085] Figure 5 Schematic diagram of other embodiments of the laser vision fusion positioning method disclosed in the present invention. Figure 5 As shown, the laser vision fusion mapping method disclosed in the present invention (for example Figure 2Step 2) of the embodiment may include steps 51 to 58, wherein:

[0086] Step 51, before visual positioning, first load the visual map, wherein the visual map includes 3D (three-dimensional) map point information of the mapping key frame, image 2D (two-dimensional) point information and descriptor information corresponding to the 2D point.

[0087] Step 52: extract feature points from the current frame image of the visual map, and use the global descriptor of the current frame image to retrieve candidate key frames in the mapping database.

[0088] Step 53: Perform visual repositioning based on the current frame information and the candidate keyframes to obtain the global pose of the current frame under the visual trajectory.

[0089] Step 54, calculate the rotation matrix between the visual track and the radar track

[0090] In some embodiments of the present disclosure, Figure 1 As shown, there is rotation and scaling between the visual sensor track 1 and the radar track 2. The rotation is caused by the different initialization of the visual sensor and the radar, which leads to different starting directions of the track in the world coordinate system. The scaling is because the visual SLAM works on the navigation object, whether it is monocular, binocular or visual IMU (Inertial measurement unit) fusion, it is difficult to ensure that the scale is absolutely consistent with the real scale. Since the navigation object only moves on the plane, the two tracks only have a rotation angle in the yaw angle (rotation around the direction of gravity as the axis), and this rotation angle is basically fixed. The present disclosure uses the visual key frame position vector and the laser position vector saved during mapping to calculate the angle between the two tracks, which is expressed as a rotation matrix:

[0091] In some embodiments of the present disclosure, the rotation matrix between the visual track and the radar track is is the extrinsic rotation matrix between the visual sensor and the radar.

[0092] Step 55 : Convert the candidate key frame poses and the current frame pose under the visual track to the candidate key frame poses and the current frame pose under the radar track.

[0093] In some embodiments of the present disclosure, the step of converting the candidate keyframe poses under the visual track to the candidate keyframe poses under the radar track in step 55 may include steps 551 to 554, wherein:

[0094] Step 551: Determine the position of the candidate key frame under the visual track according to the candidate key frame rotation matrix and the candidate key frame global position under the visual track.

[0095] In some embodiments of the present disclosure, step 551 may include: determining the candidate key frame pose under the visual trajectory according to formula (1) as

[0096]

[0097] In formula (1), is the candidate keyframe rotation matrix under the visual trajectory, is the global position of the candidate keyframe under the visual trajectory.

[0098] Step 552: Rotate the matrix of the candidate keyframe according to the visual trajectory Extrinsic rotation matrix between vision sensor and radar Determine the candidate keyframe rotation matrix under the radar track

[0099] In some embodiments of the present disclosure, step 552 may include: according to formula (2) to rotate the external parameter matrix between the visual sensor and the radar Switch to radar track.

[0100]

[0101] In formula (2), is the extrinsic rotation matrix between the visual sensor and the radar.

[0102] Step 553: Determine the global position of the candidate key frame under the radar track based on the global position of the candidate key frame under the visual track and the rotation matrix between the visual track and the radar track.

[0103] In some embodiments of the present disclosure, step 553 may include: using the rotation matrix between the two trajectories according to formula (3) Switch to radar track:

[0104]

[0105] In formula (3), is the global position of the candidate keyframe under the radar trajectory.

[0106] Step 554: Determine the position of the candidate key frame under the radar track according to the candidate key frame global position under the radar track and the candidate key frame rotation matrix under the radar track.

[0107] In some embodiments of the present disclosure, step 554 may include: determining the candidate key frame poses under the radar track according to formula (4):

[0108]

[0109] In some embodiments of the present disclosure, the current frame pose under the visual trajectory Convert to the current frame pose under the radar track The method is similar to the appeal method.

[0110] In some embodiments of the present disclosure, in step 55, the step of converting the candidate keyframe poses under the visual track to the candidate keyframe poses under the radar track may include steps 55a to 55c, wherein:

[0111] Step 55a: Determine the current frame rotation matrix under the radar track based on the current frame rotation matrix under the visual track and the extrinsic rotation matrix between the visual sensor and the radar.

[0112] Step 55b: Determine the current frame global position under the radar track based on the current frame global position under the visual track and the rotation matrix between the visual track and the radar track.

[0113] Step 55c: Determine the current frame pose under the radar track based on the current frame global position under the radar track and the current frame rotation matrix under the radar track.

[0114] Step 56: Determine the pose transformation matrix from the candidate key frame to the current frame based on the candidate key frame pose and the current frame pose under the radar track.

[0115] In some embodiments of the present disclosure, step 56 may include: according to formula (5), the candidate key frame pose under the radar track is obtained. and the current frame pose under the radar trajectory Solve the pose transformation matrix from the candidate key frame to the current frame under the radar trajectory

[0116] Step 57 : Determine the preliminary pose of the navigation object under the radar track according to the pose transformation matrix and the radar pose bound to the key frame pose.

[0117] In some embodiments of the present disclosure, step 57 may include: according to formula (6), the radar pose bound to the key frame pose is obtained by And the pose transformation from the candidate key frame to the current frame under the radar trajectory The position and posture of the navigation object under the radar track can be preliminarily solved

[0118]

[0119] Step 58 : Project the preliminary position and posture of the navigation object with 6 degrees of freedom onto 3 degrees of freedom to determine the position and posture of the navigation object in the navigation grid map coordinate system.

[0120] Since the indoor navigation object only moves in a plane, the single-line radar can only provide 3DOF pose. In the process of fusion of visual 6DOF pose and radar 3DOF, other 3DOF errors will be introduced. Projected to 3DOF, the robot pose in the navigation grid map coordinate system is obtained

[0121] Figure 6 The trajectory 3 is the trajectory obtained by positioning with the visual sensor, and Figure 1 Compared to trajectory 2 obtained by radar positioning, the two are basically consistent in rotation and scale, and their positions on the navigation grid map are also the same. The pose obtained by visual sensor positioning can be directly used for navigation object navigation.

[0122] The positioning method provided in the above embodiments of the present disclosure is an indoor positioning method that integrates laser and vision. By integrating laser SLAM and visual SLAM, their advantages are complemented, and the problems encountered by the two in the working process are solved, a low-cost and stable positioning solution is provided for navigation objects such as mobile robots.

[0123] The above-mentioned embodiments of the present disclosure reduce the error caused by the fusion of different degrees of freedom postures during the fusion process of laser SLAM and visual SLAM.

[0124] When visual SLAM is applied to navigation objects, the scale is often inconsistent with the actual scale due to problems such as motion degradation or complex scenes. Through the fusion solution of the above embodiments of the present disclosure, the scale of visual SLAM and the scale of laser SLAM can be kept consistent.

[0125] The visual positioning results of the above embodiments of the present disclosure can be directly used for navigation of the navigation object on the grid map obtained by laser SLAM.

[0126] The input of the laser vision fusion positioning method of the above embodiment of the present disclosure is an image, and the output is a posture in the navigation grid map coordinate system.

[0127] Figure 7 Schematic diagram of some embodiments of the positioning device disclosed herein. Figure 7As shown, the positioning device of the present disclosure may include a fusion mapping module 71 and a fusion positioning module 72, wherein:

[0128] The fusion mapping module 71 is configured to use radar and visual sensors to perform radar mapping and visual mapping, wherein the visual mapping includes key frame poses.

[0129] In some embodiments of the present disclosure, the fusion mapping module 71 can be configured to use radar and visual sensors for mapping at the same time, wherein radar mapping obtains a grid map for positioning and navigation, and visual mapping obtains a visual map; and the key frame pose provided by the visual mapping is bound to the radar pose provided by the radar mapping.

[0130] The fusion positioning module 72 is configured to combine the radar positioning and the visual positioning based on the key frame pose, and use the visual positioning result for navigation on the map obtained by radar positioning.

[0131] In some embodiments of the present disclosure, the fusion positioning module 72 can be configured to determine the candidate key frame poses and the current frame poses under the visual track; convert the candidate key frame poses and the current frame poses under the visual track into the candidate key frame poses and the current frame poses under the radar track; determine the pose transformation matrix from the candidate key frame to the current frame under the radar track based on the candidate key frame poses and the current frame poses under the radar track; determine the preliminary pose of the navigation object under the radar track based on the pose transformation matrix and the radar pose bound to the key frame pose.

[0132] In some embodiments of the present disclosure, the fusion positioning module 72 may also be configured to project the preliminary pose of the navigation object with 6 degrees of freedom onto 3 degrees of freedom to determine the pose of the navigation object in the grid map coordinate system.

[0133] In some embodiments of the present disclosure, the fusion positioning module 72 can be configured to load a visual map when determining the current frame pose under the visual trajectory; extract feature points from the current frame image of the visual map, and retrieve candidate key frames in the mapping database based on the current frame image descriptor; perform visual repositioning based on the current frame feature point information and the candidate key frames to obtain the current frame pose under the visual trajectory.

[0134] In some embodiments of the present disclosure, when determining the candidate key frame poses under the visual trajectory, the fusion positioning module 72 can be configured to determine the candidate key frame poses under the visual trajectory based on the candidate key frame rotation matrix under the visual trajectory and the candidate key frame global position under the visual trajectory.

[0135] In some embodiments of the present disclosure, when converting the candidate key frame poses under the visual track into the candidate key frame poses under the radar track, the fusion positioning module 72 can be configured to determine the candidate key frame rotation matrix under the radar track based on the candidate key frame rotation matrix under the visual track and the extrinsic rotation matrix between the visual sensor and the radar; calculate the rotation matrix between the visual track and the radar track; determine the candidate key frame global position under the radar track based on the candidate key frame global position under the visual track and the rotation matrix between the visual track and the radar track; determine the candidate key frame poses under the radar track based on the candidate key frame global position under the radar track and the candidate key frame rotation matrix under the radar track.

[0136] In some embodiments of the present disclosure, the fusion positioning module 72 determines the candidate key frame rotation matrix under the radar track according to the candidate key frame rotation matrix under the visual track, the external parameter rotation matrix between the visual sensor and the radar, and can be configured to determine the candidate key frame pose under the visual track according to the candidate key frame rotation matrix under the visual track and the candidate key frame global position under the visual track. According to the candidate key frame rotation matrix under the visual trajectory Extrinsic rotation matrix between vision sensor and radar Determine the candidate keyframe rotation matrix under the radar track

[0137] In some embodiments of the present disclosure, when converting the current frame pose under the visual track to the current frame pose under the radar track, the fusion positioning module 72 can be configured to determine the current frame rotation matrix under the radar track based on the current frame rotation matrix under the visual track and the extrinsic rotation matrix between the visual sensor and the radar; calculate the rotation matrix between the visual track and the radar track; determine the current frame global position under the radar track based on the current frame global position under the visual track and the rotation matrix between the visual track and the radar track; determine the current frame pose under the radar track based on the current frame global position under the radar track and the current frame rotation matrix under the radar track.

[0138] In some embodiments of the present disclosure, the positioning device is configured to implement any of the above embodiments (e.g. Figure 2-Figure 5 The operation of the positioning method described in any embodiment).

[0139] The positioning device provided based on the above-mentioned embodiment of the present disclosure is an indoor positioning device that integrates laser and vision. By integrating laser SLAM and visual SLAM, the advantages of both are complemented, the problems encountered by the two in the working process are solved, and a low-cost and stable positioning solution is provided for the navigation object.

[0140] The above-mentioned embodiments of the present disclosure reduce the error caused by the fusion of different degrees of freedom postures during the fusion process of laser SLAM and visual SLAM.

[0141] When visual SLAM is applied to navigation objects, the scale is often inconsistent with the actual scale due to problems such as motion degradation or complex scenes. Through the fusion solution of the above embodiments of the present disclosure, the scale of visual SLAM and the scale of laser SLAM can be kept consistent.

[0142] The visual positioning results of the above embodiments of the present disclosure can be directly used for navigation of the navigation object on the grid map obtained by laser SLAM.

[0143] Figure 8 FIG. 1 is a schematic diagram of the structure of a computer device according to another embodiment of the present disclosure. Figure 8 As shown, the computer device includes a memory 81 and a processor 82 .

[0144] The memory 81 is used to store instructions. The processor 82 is coupled to the memory 81. The processor 82 is configured to execute the method involved in the above embodiment (for example, Figure 2-Figure 5 The positioning method of any embodiment).

[0145] like Figure 8 As shown, the computer device also includes a communication interface 83 for exchanging information with other devices. At the same time, the computer device also includes a bus 84, through which the processor 82, the communication interface 83, and the memory 81 communicate with each other.

[0146] Memory 81 may include high-speed RAM memory or non-volatile memory, such as at least one disk storage device. Memory 81 may also be a memory array. Memory 81 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0147] Furthermore, the processor 82 may be a central processing unit (CPU), or may be an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0148] Based on the computer device provided by the above embodiments of the present disclosure, by integrating laser SLAM and visual SLAM, their advantages are complemented, the problems encountered by the two in the working process are solved, and a low-cost and stable positioning solution is provided for the navigation object.

[0149] The above-mentioned embodiments of the present disclosure reduce the error caused by the fusion of different degrees of freedom postures during the fusion process of laser SLAM and visual SLAM.

[0150] When visual SLAM is applied to navigation objects, the scale is often inconsistent with the actual scale due to problems such as motion degradation or complex scenes. Through the fusion solution of the above embodiments of the present disclosure, the scale of visual SLAM and the scale of laser SLAM can be kept consistent.

[0151] The visual positioning results of the above embodiments of the present disclosure can be directly used for navigation of the navigation object on the grid map obtained by laser SLAM.

[0152] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the computer-readable storage medium implements any of the above embodiments (e.g., Figure 2-Figure 5 The positioning method described in any embodiment).

[0153] The positioning device provided based on the above-mentioned embodiment of the present disclosure is an indoor positioning device that integrates laser and vision. By integrating laser SLAM and visual SLAM, the advantages of both are complemented, the problems encountered by the two in the working process are solved, and a low-cost and stable positioning solution is provided for the navigation object.

[0154] The above-mentioned embodiments of the present disclosure reduce the error caused by the fusion of different degrees of freedom postures during the fusion process of laser SLAM and visual SLAM.

[0155] When visual SLAM is applied to navigation objects, the scale is often inconsistent with the actual scale due to problems such as motion degradation or complex scenes. Through the fusion solution of the above embodiments of the present disclosure, the scale of visual SLAM and the scale of laser SLAM can be kept consistent.

[0156] The visual positioning results of the above embodiments of the present disclosure can be directly used for navigation of the navigation object on the grid map obtained by laser SLAM.

[0157] The positioning device and computer device described above can be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components or any appropriate combination thereof for performing the functions described in this application.

[0158] The present disclosure has been described in detail so far. To avoid obscuring the concept of the present disclosure, some details known in the art have not been described. Based on the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein.

[0159] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0160] The description of the present disclosure is provided for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the disclosed form. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure and design various embodiments with various modifications suitable for specific applications.

Claims

1. A positioning method, characterized in that: include: Performing radar mapping and visual mapping using radar and visual sensors, wherein the visual mapping includes keyframe poses; Combining radar positioning with visual positioning based on the key frame pose, and using the visual positioning result for navigation on the map obtained by radar positioning; The combining of radar positioning and visual positioning based on the key frame pose and using the visual positioning result for navigation on a map obtained by radar positioning includes: Determine the candidate keyframe poses and current frame pose under the visual trajectory; Convert the candidate key frame poses and current frame poses under the visual track to the candidate key frame poses and current frame poses under the radar track; According to the candidate key frame pose and current frame pose under the radar track The pose transformation matrix from the candidate key frame to the current frame under the radar trajectory; The preliminary pose of the navigation object under the radar track is determined according to the pose transformation matrix and the radar pose bound to the key frame pose.

2. The positioning method according to claim 1, wherein: The radar mapping and visual mapping using radar and visual sensors include: Radar and visual sensors are used simultaneously for mapping. Radar mapping produces a map for positioning and navigation, while visual mapping produces a visual map. Bind the keyframe pose provided by visual mapping to the radar pose provided by radar mapping.

3. The positioning method according to claim 1 or 2, characterized in that: The combining of radar positioning and visual positioning based on the key frame pose and using the visual positioning result for navigation on a map obtained by radar positioning further includes: The preliminary pose of the navigation object with 6 degrees of freedom is projected onto 3 degrees of freedom to determine the pose of the navigation object in the grid map coordinate system.

4. The positioning method according to claim 1 or 2, characterized in that: Determining the current frame pose under the visual trajectory includes: Load the visual map; Extract feature points from the current frame image of the visual map and retrieve candidate key frames in the mapping database based on the current frame image descriptor; Visual relocalization is performed based on the feature point information of the current frame and the candidate key frames to obtain the current frame pose under the visual trajectory.

5. The positioning method according to claim 1 or 2, characterized in that: Determining the candidate key frame poses under the visual trajectory includes: According to the candidate key frame rotation matrix and the global position of the candidate key frame under the visual track, the candidate key frame pose under the visual track is determined.

6. The positioning method according to claim 1 or 2, characterized in that: The converting of the candidate key frame poses under the visual track into the candidate key frame poses under the radar track comprises: Determine the candidate key frame rotation matrix under the radar trajectory based on the candidate key frame rotation matrix under the visual trajectory and the extrinsic rotation matrix between the visual sensor and the radar; Calculate the rotation matrix between the visual track and the radar track; Determine the global position of the candidate key frame under the radar track according to the global position of the candidate key frame under the visual track and the rotation matrix between the visual track and the radar track; The candidate key frame pose under the radar track is determined according to the candidate key frame global position under the radar track and the candidate key frame rotation matrix under the radar track.

7. The positioning method according to claim 6, characterized in that: Determining the candidate key frame rotation matrix under the radar trajectory according to the candidate key frame rotation matrix under the visual trajectory and the extrinsic rotation matrix between the visual sensor and the radar includes: Determine the pose of the candidate key frame under the visual track according to the candidate key frame rotation matrix and the global position of the candidate key frame under the visual track; The candidate key frame rotation matrix under the radar trajectory is determined according to the candidate key frame rotation matrix under the visual trajectory and the extrinsic rotation matrix between the visual sensor and the radar.

8. The positioning method according to claim 1 or 2, characterized in that: The converting the current frame pose under the visual track into the current frame pose under the radar track includes: Determine the current frame rotation matrix under the radar track based on the current frame rotation matrix under the visual track and the extrinsic rotation matrix between the visual sensor and the radar; Calculate the rotation matrix between the visual track and the radar track; Determine the current frame global position under the radar track based on the current frame global position under the visual track and the rotation matrix between the visual track and the radar track; The current frame pose under the radar track is determined according to the current frame global position under the radar track and the current frame rotation matrix under the radar track.

9. A positioning device, characterized in that: include: a fusion mapping module configured to perform radar mapping and visual mapping using radar and visual sensors, wherein the visual mapping includes keyframe poses; a fusion positioning module configured to combine radar positioning with visual positioning based on the key frame pose, and use the visual positioning result for navigation on a map obtained by radar positioning; The fusion positioning module is configured to determine the candidate key frame poses and the current frame poses under the visual track; convert the candidate key frame poses and the current frame poses under the visual track into the candidate key frame poses and the current frame poses under the radar track; and A pose transformation matrix from a candidate keyframe to a current frame under the radar track; and determining a preliminary pose of the navigation object under the radar track based on the pose transformation matrix and the radar pose bound to the keyframe pose.

10. The positioning device according to claim 9, characterized in that The fusion mapping module is configured to simultaneously use radar and visual sensors for mapping, where radar mapping generates a map for positioning and navigation, and visual mapping generates a visual map; the key frame poses provided by the visual mapping are bound to the radar poses provided by the radar mapping.

11. The positioning device according to claim 9 or 10, characterized in that: The fusion positioning module is further configured to project the preliminary pose of the navigation object with 6 degrees of freedom onto 3 degrees of freedom to determine the pose of the navigation object in the grid map coordinate system.

12. The positioning device according to claim 9 or 10, characterized in that: The positioning device is used to perform operations to implement the positioning method according to any one of claims 4 to 8.

13. A computer device, characterized in that: include: a memory configured to store instructions; The processor is configured to execute the instructions so that the computer device performs operations to implement the positioning method according to any one of claims 1 to 8.

14. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the positioning method according to any one of claims 1 to 8 is implemented.

Images