Location identification method and apparatus, computer device, and computer-readable storage medium

Abstract

A localization method and device, a computer device, and a computer-readable storage medium. The localization method includes the steps of: (1) performing radar mapping and visual mapping in advance by using a radar sensor and a vision sensor; and (2) combining the radar localization and the visual localization to provide a visual localization result of a navigation object for navigation on a grid map obtained by the radar localization. According to the localization method, laser SLAM and visual SLAM are fused and their advantages are complemented, the problems during the work process of both laser SLAM and visual SLAM are solved, and a set of low-cost and stable localization schemes is provided for the navigation object.

Description

Technical Field

[0001] Cross - reference to Related Applications This application is based on Chinese Patent Application No. 202110074516.0 filed on January 20, 2021, claims the priority thereof, and the disclosure thereof is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to the field of location determination, and in particular, to a location determination method and apparatus, a computer device, and a computer - readable storage medium.

Background Art

[0003] In the case of lidar - based indoor location determination technology, lidar is widely applied to indoor location determination of mobile robots due to its accurate ranging information. Location determination by matching laser data with a grid map is the current mainstream location determination method. That is, in order to determine the current optimal location determination pose according to the matching score, a search window is opened in the vicinity of the currently obtained pose by prediction, and several candidate poses are created within the search window.

[0004] In the case of vision-based indoor localization technology, vision-based localization is also known as visual SLAM (Simultaneous Localization and Mapping) technology. Visual SLAM applies the theory of multiple view geometries to determine the position of a camera and simultaneously build a map of the surrounding environment according to the image information captured by the camera. Visual SLAM technology mainly consists of a vision odometer, backend optimization, loop detection, and mapping. The vision odometer processes the input image, calculates pose changes, and obtains motion relationships between cameras by examining the transformation relationships between image frames to complete real-time pose tracking. However, errors accumulate over time, which is due to estimating motion only between two images. The backend mainly uses optimization methods to reduce errors across frames, with camera poses and spatial map points. Loop detection, also called closed-loop detection, mainly uses similarity between images to determine whether the previous position has been reached, eliminate accumulated errors, and obtain a globally consistent trajectory and map. In mapping, a map corresponding to the task requirements is created according to the estimated trajectory. [Overview of the project] [Means for solving the problem]

[0005] According to one aspect of the present disclosure, a localization method is provided. The method comprises the steps of performing radar mapping and vision mapping using radar and vision sensors, respectively, wherein the vision mapping step comprises determining keyframe poses, and combining the radar localization with vision localization based on keyframe poses in order to use the vision localization results for navigation on a map acquired by radar mapping.

[0006] In some embodiments of the present disclosure, the step of performing radar mapping and vision mapping by using radar and vision sensors comprises the steps of performing mapping by using radar and vision sensors simultaneously, wherein a map for localization and navigation is obtained by radar mapping and a vision map is obtained by vision mapping, and the steps of combining keyframe poses provided by vision mapping with radar poses provided by radar mapping.

[0007] In some embodiments of the present disclosure, the steps of combining radar localization with vision localization based on keyframe poses in order to use vision localization results for navigation on a map acquired by radar mapping include: determining the pose of a candidate keyframe and the pose of the current frame under the vision trajectory; transforming the pose of the candidate keyframe and the pose of the current frame under the vision trajectory to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory; and determining a pose transformation matrix from the candidate keyframe to the current frame under the radar trajectory according to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory. , Series transformation matrix and linked to keyframe poses Radar pose and The method comprises the step of determining a provisional pose for a navigation object below the radar trajectory.

[0008] In some embodiments of the present disclosure, the step of combining radar localization with vision localization based on keyframe poses in order to use vision localization results for navigation on a map acquired by radar mapping further comprises the step of determining the pose of the navigation object in the coordinate system of a grid map by projecting the provisional pose of a 6-degree-of-freedom navigation object onto the provisional pose of a 3-degree-of-freedom navigation object.

[0009] In some embodiments of the present disclosure, the step of determining the pose of the current frame under a vision trajectory comprises the steps of loading a vision map, extracting feature points from the image of the current frame in the vision map, searching for candidate keyframes in a mapping database according to descriptors of the image of the current frame, and performing a vision repositioning according to the information of the candidate keyframes and the feature points of the current frame in order to obtain the pose of the current frame under a vision trajectory.

[0010] In some embodiments of the present disclosure, the step of determining the pose of a candidate keyframe under a vision trajectory comprises the step of determining the pose of a candidate keyframe under a vision trajectory according to the rotation matrix of the candidate keyframe under a vision trajectory and the global position of the candidate keyframe under a vision trajectory.

[0011] In some embodiments of the present disclosure, the step of converting the pose of candidate keyframes under a vision trajectory to the pose of candidate keyframes under a radar trajectory comprises: determining a rotation matrix of candidate keyframes under a radar trajectory according to a rotation matrix of candidate keyframes under a vision trajectory and an external parameter rotation matrix between a vision sensor and a radar; calculating a rotation matrix between a vision trajectory and a radar trajectory; determining the global position of candidate keyframes under a radar trajectory according to the global position of candidate keyframes under a vision trajectory and the rotation matrix between a vision trajectory and a radar trajectory; and determining the pose of candidate keyframes under a radar trajectory according to the global position of candidate keyframes under a radar trajectory and the rotation matrix of candidate keyframes under a radar trajectory.

[0012] In some embodiments of the present disclosure, the step of determining a rotation matrix of candidate keyframes under a radar trajectory according to a rotation matrix of candidate keyframes under a vision trajectory and an external parameter rotation matrix between a vision sensor and a radar comprises the steps of determining the pose of a candidate keyframe under a vision trajectory according to the rotation matrix of candidate keyframes under a vision trajectory and the global position of a candidate keyframe under a vision trajectory, and determining a rotation matrix of candidate keyframes under a radar trajectory according to a rotation matrix of candidate keyframes under a vision trajectory and an external parameter rotation matrix between a vision sensor and a radar.

[0013] In some embodiments of the present disclosure, the step of converting the pose of the current frame under a vision trajectory to the pose of the current frame under a radar trajectory comprises the steps of: determining the rotation matrix of the current frame under a radar trajectory according to the rotation matrix of the current frame under a vision trajectory and the external parameter rotation matrix between the vision sensor and the radar; calculating the rotation matrix between the vision trajectory and the radar trajectory; determining the global position of the current frame under a radar trajectory according to the global position of the current frame under a vision trajectory and the rotation matrix between the vision trajectory and the radar trajectory; and determining the pose of the current frame under a radar trajectory according to the global position of the current frame under a radar trajectory and the rotation matrix of the current frame under a radar trajectory.

[0014] According to another aspect of the present disclosure, a localization device is provided. The device comprises a fused mapping module configured to perform radar mapping and vision mapping by using a radar and a vision sensor, respectively, wherein a vision mapping step comprises determining a keyframe pose, and a fused localization module configured to combine radar localization with vision localization based on a keyframe pose for using vision localization results for navigation on a map acquired by radar mapping.

[0015] In some embodiments of the present disclosure, the fused mapping module is configured to perform mapping by simultaneously using radar and vision sensors, wherein radar mapping acquires a map for localization and navigation, vision mapping acquires a vision map, and keyframe poses provided by vision mapping are combined with radar poses provided by radar mapping.

[0016] In some embodiments of the present disclosure, the fused positioning module determines the pose of a candidate keyframe and the pose of the current frame under the vision trajectory; transforms the pose of the candidate keyframe and the pose of the current frame under the vision trajectory into the pose of the candidate keyframe and the pose of the current frame under the radar trajectory; and determines a pose transformation matrix from the candidate keyframe to the current frame under the radar trajectory according to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory. , Series transformation matrix and linked to keyframe poses Radar pose and It is configured to determine the provisional pose of the navigation object below the radar trajectory accordingly.

[0017] In some embodiments of the present disclosure, the fused positioning module is further configured to determine the pose of a navigation object in a grid map coordinate system by projecting the provisional pose of a 6-degree-of-freedom navigation object onto the provisional pose of a 3-degree-of-freedom navigation object.

[0018] In some embodiments of the present disclosure, the location device is configured to perform an operation that performs a location method according to any one of the embodiments described above.

[0019] According to another aspect of the present disclosure, a computer device is provided. The device includes a memory configured to store instructions, and a processor configured to execute the instructions such that the computer device performs an operation of executing a positioning method according to any one of the above-described embodiments.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, and the computer-readable storage medium stores computer instructions that, when executed by a processor, execute a positioning method according to any one of the above-described embodiments.

[0021] For the sake of more clearly explaining the technical solutions in the embodiments of the present disclosure or related technologies, the accompanying drawings that need to be used in the description of the embodiments or related technologies are briefly introduced below. It is obvious that the accompanying drawings described below are only a part of the embodiments of the present disclosure. Those skilled in the art can also obtain other accompanying drawings according to such accompanying drawings on the premise that no inventive effort is involved.

Brief Description of the Drawings

[0022] [Figure 1] It is a schematic diagram of a trajectory and a grid map visualized by simultaneously using radar mapping and vision mapping for the same navigation object. [Figure 2] It is a schematic diagram of some embodiments of the positioning method according to the present disclosure. [Figure 3] It is a schematic diagram of some embodiments of the laser-vision fusion mapping method according to the present disclosure. [Figure 4] It is a schematic diagram of some embodiments of the laser-vision fusion positioning method according to the present disclosure. [Figure 5] It is a schematic diagram of other embodiments of the laser-vision fusion positioning method according to the present disclosure. [Figure 6] It is a diagram showing the rendering of the trajectory after fusion positioning according to some embodiments of the present disclosure. [Figure 7] Schematic diagram of some embodiments of the positioning device of the present disclosure. [Figure 8] Schematic structural diagram of a computer device according to a further embodiment of the present disclosure.

Embodiments for Carrying Out the Invention

[0023] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part, not all, of the embodiments of the present disclosure. The following description of at least one exemplary embodiment is actually only an illustration and does not limit the present disclosure or its application or use in any way. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without inventive efforts fall within the protection scope of the present disclosure.

[0024] Unless otherwise specified, the relative arrangements, numerical expressions, and numerical values of the components and steps described in these examples do not limit the scope of the present disclosure.

[0025] At the same time, for the sake of simplicity of explanation, it should be understood that the dimensions of various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship.

[0026] Techniques, methods, and devices known to those skilled in the relevant art may not be described in detail. However, the techniques, methods, and devices shall be regarded as part of the given description when appropriate.

[0027] In all the examples shown and discussed in this specification, any specific value should be construed as illustrative rather than limiting. Therefore, other examples in the exemplary embodiments may have different values.

[0028] In the following attached drawings, similar reference symbols and letters indicate the same items; therefore, it should be noted that once an item is defined in one attached drawing, it is necessary to further describe the same item in subsequent attached drawings.

[0029] Radar-based indoor positioning technologies in related fields have the following problems: Some low-cost radars have limited ranging ranges, making it difficult to obtain effective ranging information in large-scale scenarios. Laser SLAM may have motion degradation issues when faced with long corridor environments. When using small amounts of radar information, laser SLAM is generally less likely to generate loopbacks compared to visual SLAM.

[0030] Vision sensor-based indoor positioning technologies have the following problems: Visual SLAM's positioning accuracy decreases when faced with environments with weak texture, such as white walls. Vision sensors are generally very sensitive to lighting, and as a result, the stability of positioning decreases when visual SLAM operates in environments with large fluctuations in lighting. The resulting maps cannot be directly used for navigating navigation objects.

[0031] Taking into consideration at least one of the technical problems described above, this disclosure provides a location method and apparatus, a computer apparatus, and a computer-readable storage medium. By integrating laser SLAM and visual SLAM, the advantages of laser SLAM and visual SLAM complement each other in order to solve problems that arise in the working processes of both laser SLAM and visual SLAM themselves.

[0032] Figure 1 is a schematic diagram of the trajectory and grid map visualized using radar mapping and vision mapping simultaneously for the same navigation object, where the navigation object could be a vision sensor, the radar could be a lidar, and the vision sensor could be a camera. As shown in Figure 1, trajectory 1 is the trajectory left by laser SLAM, the brightly colored area 4 is the occupied grid map constructed by laser SLAM for the navigation of the navigation object, and trajectory 2 is the trajectory left by visual SLAM. Although both laser SLAM and visual SLAM describe the movement of the navigation object, the trajectories located by both do not coincide in the world coordinate system because the installation position and angle of the radar and vision sensors, and the environmental description scale are different, and rotation and zoom generally occur. Thus, if the radar does not function properly, it is impossible to provide a pose for the navigation object under the navigation map coordinate system by directly using the positioning of the vision sensor.

[0033] The purpose of this disclosure is to provide a laser and vision localization fusion solution that allows for a smooth transition to another sensor for localization when problems arise that cannot be solved by laser or vision localization alone.

[0034] The dominant navigation method for indoor navigation objects in related technologies is to plan a path on an occupied grid map, thereby controlling the robot's movement. Lida-based localization and navigation solutions in related technologies are typically divided into two components: mapping and localization and navigation. The function of mapping is to create a two-dimensional occupied grid map of the environment using Lida. Localization is implemented by matching Lida data with the occupied grid map to obtain the navigation object's current pose in the coordinate system of the occupied grid map. Navigation is implemented by planning a path from the current pose obtained by localization to a target point on the occupied grid map and controlling the robot to move to the specified target point.

[0035] Figure 2 is a schematic diagram of several embodiments of the localization method according to the present disclosure. Preferably, these embodiments can be performed by a localization device or a computer device of the present disclosure. The method may comprise steps 1 and 2.

[0036] In Step 1, laser-vision fusion mapping is performed.

[0037] In some embodiments of the present disclosure, step 1 may comprise the step of performing radar mapping and vision mapping by using a radar and a vision sensor, respectively, wherein the radar may be a lidar and the vision sensor may be a camera.

[0038] In some embodiments of the present disclosure, step 1 may be a step of mapping by using radar and vision sensors simultaneously, wherein a grid map for localization and navigation is obtained by radar mapping and a vision map is obtained by vision mapping, and a step of combining keyframe poses provided by vision mapping with radar poses provided by radar mapping.

[0039] Figure 3 is a schematic diagram of several embodiments of the laser-vision fusion mapping method according to the present disclosure. As shown in Figure 3, the laser-vision fusion mapping method according to the present disclosure (for example, step 1 of the embodiment in Figure 2) may comprise steps 11 to 14.

[0040] In step 11, during mapping, the mapping is performed by using radar and vision sensors simultaneously, where radar mapping provides radar poses, vision mapping provides keyframe poses, radar poses can be Lidar poses, radar poses can be laser poses, radar mappings can be laser mappings, and radar mappings can be Lidar mappings.

[0041] In step 12, the nearest radar pose near each vision keyframe is searched according to the pose binding timestamp.

[0042] In step 13, when the vision map is saved, the radar pose corresponding to the vision keyframe is also saved at the same time.

[0043] In step 14, the radar mapping process saves the occupied grid map for radar localization and navigation of navigation objects.

[0044] In step 2, laser-vision fusion localization is performed. By combining radar localization with vision localization, the vision localization results may be used by navigation objects for navigation on a grid map acquired by radar mapping.

[0045] Figure 4 is a schematic diagram of several embodiments of the laser-vision fusion localization method according to the present disclosure. As shown in Figure 4, the laser-vision fusion mapping method according to the present disclosure (for example, step 2 of the embodiment in Figure 2) may comprise steps 21 to 25.

[0046] In step 21, the pose of the candidate keyframe and the pose of the current frame under the vision trajectory are determined.

[0047] In step 22, the current frame under the candidate keyframe's pose and vision trajectory is converted to the current frame under the candidate keyframe's pose and radar trajectory.

[0048] In step 23, the pose transformation matrix from the candidate keyframe to the current frame is determined according to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory.

[0049] In step 24, the provisional pose of the navigation object below the radar trajectory is determined according to the radar pose combined with the pose transformation matrix and keyframe pose, and the navigation object may be a vision sensor.

[0050] In step 25, the pose of the navigation object in the grid map's navigation coordinate system is determined by projecting the temporary pose of the 6-degree-of-freedom navigation object onto the temporary pose of the 3-degree-of-freedom navigation object.

[0051] Figure 5 is a schematic diagram of another embodiment of the laser-vision fusion localization method according to the present disclosure. As shown in Figure 5, the laser-vision fusion mapping method according to the present disclosure (for example, step 2 of the embodiment in Figure 2) may comprise steps 51 to 58.

[0052] In step 51, before vision localization, a vision map is first loaded, which includes 3D (three-dimensional) map point information of mapping keyframes, 2D (two-dimensional) point information of the image, and descriptor information corresponding to the 2D points.

[0053] In step 52, feature points are extracted from the image of the current frame of the vision map, and candidate keyframes are searched in the mapping database using the global descriptor of the image of the current frame.

[0054] In step 53, the global pose of the current frame under the vision trajectory.

[0055]

number

[0056] To obtain this, the vision is rearranged according to the current frame information and candidate keyframes.

[0057] In step 54, the rotation matrix between the vision orbit and the radar orbit.

[0058]

number

[0059] This is calculated.

[0060] In some embodiments of this disclosure, rotation and zoom exist between the vision sensor trajectory 1 and the radar trajectory 2, as shown in Figure 1. Rotation is due to various orientations of the trajectories starting in the world coordinate system, which is due to various initializations of the vision sensor and radar. Zoom arises from the fact that when visual SLAM operates on a navigation object, whether monocular, binocular, or vision IMU (inertial measurement unit) fusion, it is very difficult to ensure that the scale perfectly matches the actual scale. Since the navigation object only moves in a plane, there are two trajectories with rotation angles of only yaw angle (rotating around the direction of gravity), and the rotation angles are approximately constant. In this disclosure, the angle between the two trajectories is calculated by using the vision keyframe position vector and the laser position vector stored during mapping, and the rotation matrix

[0061]

number

[0062] It is expressed as follows.

[0063] In some embodiments of this disclosure, the rotation matrix between the vision trajectory and the radar trajectory is described.

[0064]

number

[0065] This is the external parameter rotation matrix between the vision sensor and the radar.

[0066] In step 55, the candidate keyframe pose and the current frame pose under the vision trajectory are converted to the candidate keyframe pose and the current frame pose under the radar trajectory.

[0067] In some embodiments of the present disclosure, the step of converting the pose of candidate keyframes under the vision trajectory to the pose of candidate keyframes under the radar trajectory in step 55 may comprise steps 551 to 554.

[0068] In step 551, the pose of the candidate keyframes under the vision trajectory

[0069]

number

[0070] This is determined according to the rotation matrix of the candidate keyframes under the vision trajectory and the global position of the candidate keyframes under the vision trajectory.

[0071] In some embodiments of the present disclosure, step 551 is to pause candidate keyframes under the vision trajectory according to formula (1).

[0072]

number

[0073] It may be possible to make a decision.

[0074]

number

[0075] In equation (1),

[0076]

number

[0077] This is the rotation matrix of candidate keyframes under the vision trajectory,

[0078]

number

[0079] This is the global position of the candidate keyframes under the vision trajectory.

[0080] In step 552, the rotation matrix of the candidate keyframes under the radar trajectory.

[0081]

number

[0082] This is the rotation matrix of candidate keyframes under the vision trajectory.

[0083]

number

[0084] And, the external parameter rotation matrix between the vision sensor and the radar.

[0085]

number

[0086] It is determined according to [the following].

[0087] In some embodiments of the present disclosure, step 552 is performed according to equation (2) by rotating an external parameter matrix between the vision sensor and the radar.

[0088]

number

[0089] This determines the rotation matrix of candidate keyframes under the vision trajectory.

[0090]

number

[0091] This is the rotation matrix of candidate keyframes under the radar trajectory.

[0092]

number

[0093] It may be capable of converting to [a certain format].

[0094]

number

[0095] In equation (2),

[0096]

number

[0097] This is the external parameter rotation matrix between the vision sensor and the radar.

[0098] In step 553, the global position of the candidate keyframes below the radar orbit.

[0099]

number

[0100] This is determined according to the global position of the candidate keyframes under the vision trajectory and the rotation matrix between the vision trajectory and the radar trajectory.

[0101] In some embodiments of the present disclosure, step 553 is the rotation matrix between the two orbits according to equation (3).

[0102]

number

[0103] The global position of candidate keyframes under the vision trajectory

[0104]

number

[0105] global position of candidate keyframes under the radar orbit

[0106]

number

[0107] It may be capable of converting to [a certain format].

[0108]

number

[0109] In equation (3),

[0110]

number

[0111] This is the global position of the candidate keyframes below the radar trajectory.

[0112] In step 554, the pose of the candidate keyframes below the radar trajectory.

[0113]

number

[0114] This is determined according to the global position of the candidate keyframes under the radar trajectory and the rotation matrix of the candidate keyframes under the radar trajectory.

[0115] In some embodiments of the present disclosure, step 554 is the pose of candidate keyframes under the radar trajectory according to equation (4).

[0116]

number

[0117] It may include a step to determine this.

[0118]

number

[0119] In some embodiments of this disclosure, the pose of the current frame under the vision trajectory

[0120]

number

[0121] The current frame's pose below the radar orbit

[0122]

number

[0123] The method for converting to is the same as the method described above.

[0124] In some embodiments of the present disclosure, the step in step 55 of converting the pose of the current frame under the vision trajectory to the pose of the current frame under the radar trajectory may comprise steps 55a to 55c.

[0125] In step 55a, the rotation matrix of the current frame under the radar trajectory is determined according to the rotation matrix of the current frame under the vision trajectory and the external parameter rotation matrix between the vision sensor and the radar.

[0126] In step 55b, the global position of the current frame under the radar orbit is determined according to the global position of the current frame under the vision orbit and the rotation matrix between the vision orbit and the radar orbit.

[0127] In step 55c, the position of the current frame under the radar orbit is determined according to the global position of the current frame under the radar orbit and the rotation matrix of the current frame under the radar orbit.

[0128] In step 56, the pose transformation matrix from the candidate keyframe to the current frame is determined according to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory.

[0129] In some embodiments of the present disclosure, step 56 is the pose of candidate keyframes under the radar trajectory according to equation (5).

[0130]

number

[0131] And the current keyframe pose below the radar orbit

[0132]

number

[0133] This creates a pose transformation matrix from the candidate keyframe to the current frame under the radar trajectory.

[0134]

number

[0135] It is possible to solve it.

[0136]

number

[0137] In step 57, the provisional pose of the navigation object below the radar trajectory is determined according to the pose transformation matrix and the radar pose combined with the keyframe pose.

[0138] In some embodiments of the present disclosure, step 57 poses the navigation object below the radar trajectory according to formula (6).

[0139]

number

[0140] This is the radar pose, which is linked to the keyframe pose.

[0141]

number

[0142] And, the pose transformation matrix from the candidate keyframe to the current frame under the radar trajectory.

[0143]

number

[0144] It is possible that this may resolve the issue.

[0145]

number

[0146] In step 58, the pose of the navigation object in the grid map's navigation coordinate system is determined by projecting the temporary pose of the 6-degree-of-freedom (6DOF) navigation object onto the temporary pose of the 3-degree-of-freedom (3DOF) navigation object.

[0147] Because indoor navigation objects move only in a plane, a single bundle radar can only provide 3DOF poses, and other 3DOF errors may be introduced during the fusion process of the vision 6DOF pose and the radar 3DOF pose. This disclosure provides 6DOF poses of navigation objects below the radar trajectory.

[0148]

number

[0149] The robot's pose is projected onto 3DOF, under the navigation coordinate system of the grid map.

[0150]

number

[0151] This is obtained.

[0152] Figure 6 shows renderings of fused orbits according to several embodiments of the present disclosure. Orbit 3 is the orbit obtained by vision sensor orbiting, and when compared to orbit 1 obtained by radar orbiting in Figure 1, both are nearly identical in rotation and scale, and their positions on the navigation grid map are the same. The pose obtained by vision sensor orbiting can be used directly for the navigation of the navigation object.

[0153] The localization method provided based on the above-described embodiments of this disclosure is a laser-vision fusion indoor localization method. By fusing laser SLAM and visual SLAM, the advantages of laser SLAM and visual SLAM complement each other to solve problems that arise in the working processes of both laser SLAM and visual SLAM themselves, providing a low-cost and stable localization solution for navigation objects such as mobile robots.

[0154] In the embodiments described above of this disclosure, errors caused by pose fusion of different degrees of freedom are reduced during the fusion process of laser SLAM and visual SLAM.

[0155] Applying visual SLAM to navigation objects presents technical challenges such as motion degradation and complex scenarios, often resulting in a mismatch between the scale and the actual scale. The fused solution in the embodiments described above makes it possible to match the scale of the visual SLAM with the scale of the laser SLAM.

[0156] The vision localization results of the above-described embodiments can be used directly by a navigation object for navigation on a grid map acquired by laser SLAM.

[0157] The input to the laser-vision fusion localization method in the above-described embodiment of the present disclosure is an image, and the output is a pose in a grid map navigation coordinate system.

[0158] Figure 7 is a schematic diagram of several embodiments of the localization device of the present disclosure. As shown in Figure 7, the localization device of the present disclosure may comprise a fused mapping module 71 and a fused localization module 72.

[0159] The fused mapping module 71 is configured to perform radar mapping and vision mapping by using radar and vision sensors, respectively, with the vision mapping step comprising determining keyframe poses.

[0160] In some embodiments of the present disclosure, the fused mapping module 71 may be configured to perform mapping by simultaneously using radar and vision sensors, wherein radar mapping acquires a map for localization and navigation, vision mapping acquires a vision map, and keyframe poses provided by vision mapping are combined with radar poses provided by radar mapping.

[0161] The fused positioning module 72 is configured to combine radar positioning with vision positioning based on keyframe poses in order to use vision positioning results for navigation on a map acquired by radar mapping.

[0162] In some embodiments of the present disclosure, the fused positioning module 72 determines the pose of a candidate keyframe and the pose of the current frame under the vision trajectory; transforms the pose of the candidate keyframe and the pose of the current frame under the vision trajectory into the pose of the candidate keyframe and the pose of the current frame under the radar trajectory; and determines a pose transformation matrix from the candidate keyframe to the current frame under the radar trajectory according to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory. , Series transformation matrix and linked to keyframe poses Radar pose and Accordingly, it may be configured to determine the provisional pose of a navigation object below the radar trajectory.

[0163] In some embodiments of the present disclosure, the fused positioning module 72 may be further configured to determine the pose of a navigation object in a grid map coordinate system by projecting the provisional pose of a 6-degree-of-freedom navigation object onto the provisional pose of a 3-degree-of-freedom navigation object.

[0164] In some embodiments of the present disclosure, the fused localization module 72 may be configured to load a vision map, extract feature points from the current frame image of the vision map, search for candidate keyframes in a mapping database according to descriptors of the current frame image, and perform a vision repositioning according to the information of the candidate keyframes and the feature points of the current frame in order to obtain the current frame pose under the vision trajectory.

[0165] In some embodiments of the present disclosure, the fused positioning module 72 may be configured such that, if the pose of a candidate keyframe is determined under a vision trajectory, the step of determining the pose of a candidate keyframe under a vision trajectory is to determine the pose of the candidate keyframe under a vision trajectory according to the rotation matrix of the candidate keyframe under a vision trajectory and the global position of the candidate keyframe under a vision trajectory.

[0166] In some embodiments of the present disclosure, when converting the pose of candidate keyframes under a vision trajectory to the pose of candidate keyframes under a radar trajectory, the fused positioning module 72 may be configured to: determine the rotation matrix of candidate keyframes under a radar trajectory according to the rotation matrix of candidate keyframes under a vision trajectory and an external parameter rotation matrix between the vision sensor and the radar; calculate the rotation matrix between the vision trajectory and the radar trajectory; determine the global position of candidate keyframes under a radar trajectory according to the global position of candidate keyframes under a vision trajectory and the rotation matrix between the vision trajectory and the radar trajectory; and determine the pose of candidate keyframes under a radar trajectory according to the global position of candidate keyframes under a radar trajectory and the rotation matrix of candidate keyframes under a radar trajectory.

[0167] In some embodiments of this disclosure, if the rotation matrix of candidate keyframes under the radar trajectory is determined according to the rotation matrix of candidate keyframes under the vision trajectory and the external parameter rotation matrix between the vision sensor and the radar, the fused localization module 72 poses the keyframes under the vision trajectory according to the rotation matrix of candidate keyframes under the vision trajectory and the global position of the candidate keyframes under the vision trajectory.

[0168]

number

[0169] This involves determining the rotation matrix of candidate keyframes under the vision trajectory.

[0170]

number

[0171] And, the external parameter rotation matrix between the vision sensor and the radar.

[0172]

number

[0173] Accordingly, the rotation matrix of candidate keyframes under the radar trajectory.

[0174]

number

[0175] It can be configured to make a decision and to perform the action.

[0176] In some embodiments of the present disclosure, when converting the pose of the current frame under a vision trajectory to the pose of the current frame under a radar trajectory, the fused positioning module 72 may be configured to: determine the rotation matrix of the current frame under a radar trajectory according to the rotation matrix of the current frame under a vision trajectory and the external parameter rotation matrix between the vision sensor and the radar; calculate the rotation matrix between the vision trajectory and the radar trajectory; determine the global position of the current frame under a radar trajectory according to the global position of the current frame under a vision trajectory and the rotation matrix between the vision trajectory and the radar trajectory; and determine the pose of the current frame under a radar trajectory according to the global position of the current frame under a radar trajectory and the rotation matrix of the current frame under a radar trajectory.

[0177] In some embodiments of this disclosure, the location device is configured to perform an operation that executes a location method according to any one of the embodiments described above (for example, any one of the embodiments in Figures 2 to 5).

[0178] The positioning device provided based on the above embodiments of this disclosure is a laser-vision fusion indoor positioning device. By fusing laser SLAM and visual SLAM, the advantages of laser SLAM and visual SLAM complement each other to solve problems that arise in the working processes of both laser SLAM and visual SLAM themselves, providing a low-cost and stable positioning solution for navigation objects such as mobile robots.

[0179] In the embodiments described above of this disclosure, errors caused by pose fusion of different degrees of freedom are reduced during the fusion process of laser SLAM and visual SLAM.

[0180] Applying visual SLAM to navigation objects presents technical challenges such as motion degradation and complex scenarios, often resulting in a mismatch between the scale and the actual scale. The fused solution in the embodiments described above makes it possible to match the scale of the visual SLAM with the scale of the laser SLAM.

[0181] The vision localization results of the above-described embodiments can be used directly by a navigation object for navigation on a grid map acquired by laser SLAM.

[0182] Figure 8 is a schematic diagram of the structure of a computer device according to a further embodiment of the present disclosure. As shown in Figure 8, the computer device comprises a memory 81 and a processor 82.

[0183] The memory 81 is configured to store instructions, and the processor 82 is coupled to the memory 81. The processor 82 is configured to perform methods related to the embodiments described above (for example, the location method according to any one of the embodiments in Figures 2 to 5) based on the instructions stored in the memory.

[0184] As shown in Figure 8, the computer device also includes a communication interface 83 for information interaction with other devices. At the same time, the computer device also includes a bus 84 through which the processor 82, the communication interface 83, and the memory 81 communicate with each other.

[0185] The memory 81 may include high-speed RAM memory or non-volatile memory, such as at least one disk memory. The memory 81 may also be a memory array. The memory 81 may be further divided into blocks that can be combined into virtual volumes according to certain rules.

[0186] Furthermore, the processor 82 may be a central processing unit CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0187] The computer device provided based on the embodiments of the present disclosure described above integrates laser SLAM and visual SLAM, with the advantages of both laser SLAM and visual SLAM complementing each other to solve problems that arise in the working processes of both laser SLAM and visual SLAM themselves, providing a low-cost and stable localization solution for navigation objects such as mobile robots.

[0188] In the embodiments described above of this disclosure, errors caused by pose fusion of different degrees of freedom are reduced during the fusion process of laser SLAM and visual SLAM.

[0189] Applying visual SLAM to navigation objects presents technical challenges such as motion degradation and complex scenarios, often resulting in a mismatch between the scale and the actual scale. The fused solution in the embodiments described above makes it possible to match the scale of the visual SLAM with the scale of the laser SLAM.

[0190] The vision localization results of the above-described embodiments can be used directly by a navigation object for navigation on a grid map acquired by laser SLAM.

[0191] According to another aspect of the present disclosure, a non-temporary computer-readable storage medium is provided which, when executed by a processor, stores computer instructions that perform a location method according to any one of the embodiments described above (for example, any one of the embodiments in Figures 2 to 5).

[0192] The positioning device provided based on the above embodiments of this disclosure is a laser-vision fusion indoor positioning device. By fusing laser SLAM and visual SLAM, the advantages of laser SLAM and visual SLAM complement each other to solve problems that arise in the working processes of both laser SLAM and visual SLAM themselves, providing a low-cost and stable positioning solution for navigation objects such as mobile robots.

[0193] In the embodiments described above, errors caused by pose fusion of different degrees of freedom are reduced during the fusion process of laser SLAM and visual SLAM.

[0194] Applying visual SLAM to navigation objects presents technical challenges such as motion degradation and complex scenarios, often resulting in a mismatch between the scale and the actual scale. The fused solution in the embodiments described above makes it possible to match the scale of the visual SLAM with the scale of the laser SLAM.

[0195] The vision localization results of the above-described embodiments can be used directly by a navigation object for navigation on a grid map acquired by laser SLAM.

[0196] The above-described location and computer devices may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware assemblies, or any suitable combination thereof for performing the functions described in this application.

[0197] The present disclosure has been described in detail above. Some details well known in the art have not been described in order to avoid obscuring the concepts of this disclosure. Those skilled in the art will fully understand, based on the above description, how to implement the technical solutions disclosed herein.

[0198] Those skilled in the art will understand that all or part of the steps in the embodiments described above can be achieved by hardware or by a program that instructs the relevant hardware. The program may be stored on a computer-readable storage medium. The storage medium may be read-only memory, a magnetic disk, or an optical disk, etc.

[0199] The descriptions in this disclosure, made for illustrative and explanatory purposes, are not omitted or limit the disclosure to the disclosed form. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described to better illustrate the principles and practical applications of this disclosure and to enable those skilled in the art to understand this disclosure and design various embodiments with various modifications that are suitable for specific purposes. [Explanation of Symbols]

[0200] 1 orbit 2 orbits 3 orbits 4. Light-colored parts 71. Integrated Mapping Modules 72. Integrated localization modules 81 memory 82 processors 83 Communication Interface 84 Bus

Claims

1. A location determination method performed by a computer device, A step of performing radar mapping and vision mapping using radar and vision sensors, respectively, wherein the vision mapping step comprises determining a keyframe pose, and the pose of the keyframe is the pose of the navigation object in the keyframe; The process includes the step of combining radar localization with vision localization based on the pose of the keyframe in order to use the vision localization results for navigation on the map acquired by the radar mapping, The step of combining radar localization with vision localization based on the pose of the keyframe in order to use the vision localization results for navigation on the map acquired by the radar mapping is: A step of determining the pose of a candidate keyframe and the pose of the current frame under the vision trajectory, wherein the candidate keyframe is retrieved in the keyframes of the vision map mapping database by using the global descriptor of the image of the current frame, and the pose of the current frame is the pose of the navigation object in the current frame. The steps include converting the pose of the current frame under the pose of the candidate keyframe and the vision trajectory to the pose of the current frame under the pose of the candidate keyframe and the radar trajectory, The steps of determining a pose transformation matrix from the candidate keyframe to the current frame under the radar trajectory, according to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory, A method for determining a position, comprising the steps of determining a provisional pose of a navigation object under the radar trajectory according to the pose transformation matrix and the radar pose coupled to the pose of the keyframe.

2. The step of performing radar mapping and vision mapping by using radar and vision sensors, respectively, A step of performing mapping by simultaneously using the radar and the vision sensor, wherein a map for location identification and navigation is obtained by the radar mapping, and a vision map is obtained by the vision mapping; The positioning method according to claim 1, further comprising the step of combining the pose of the keyframe provided by the vision mapping with the radar pose provided by the radar mapping.

3. The step of combining radar localization with vision localization based on the pose of the keyframe in order to use the vision localization results for navigation on the map acquired by the radar mapping is: The positioning method according to claim 1 or 2, further comprising the step of determining the pose of the navigation object in the coordinate system of a grid map by projecting the provisional pose of the 6-degree-of-freedom navigation object onto the provisional pose of the 3-degree-of-freedom navigation object.

4. The step of determining the pose of the current frame under the vision trajectory, The steps to load the vision map, The steps include extracting feature points from the image of the current frame of the vision map, The steps of searching the mapping database for the candidate keyframe according to the global descriptor of the image of the current frame, A positioning method according to claim 1 or 2, comprising the step of performing a vision repositioning according to the information of the candidate keyframes and the feature points of the current frame in order to obtain the pose of the current frame under the vision trajectory.

5. The step of determining the pose of the candidate keyframe under the vision trajectory, A positioning method according to claim 1 or 2, comprising the step of determining the pose of the candidate keyframe under the vision trajectory according to the rotation matrix of the candidate keyframe under the vision trajectory and the global position of the candidate keyframe under the vision trajectory.

6. The step of converting the pose of the candidate keyframe under the vision trajectory to the pose of the candidate keyframe under the radar trajectory, The steps include determining the rotation matrix of the candidate keyframes under the radar trajectory according to the rotation matrix of the candidate keyframes under the vision trajectory and the external parameter rotation matrix between the vision sensor and the radar, The steps include calculating a rotation matrix between the vision trajectory and the radar trajectory, The steps include determining the global position of the candidate keyframe under the radar trajectory according to the global position of the candidate keyframe under the vision trajectory and the rotation matrix between the vision trajectory and the radar trajectory, A positioning method according to claim 1 or 2, comprising the steps of determining the pose of the candidate keyframe under the radar trajectory according to the global position of the candidate keyframe under the radar trajectory and the rotation matrix of the candidate keyframe under the radar trajectory.

7. The step of determining the rotation matrix of the candidate keyframes under the radar trajectory according to the rotation matrix of the candidate keyframes under the vision trajectory and the external parameter rotation matrix between the vision sensor and the radar, A step of determining the pose of the candidate keyframe under the vision trajectory according to the rotation matrix of the candidate keyframe under the vision trajectory and the global position of the candidate keyframe under the vision trajectory, The positioning method according to claim 6, comprising the step of determining the rotation matrix of the candidate keyframes under the radar trajectory according to the rotation matrix of the candidate keyframes under the vision trajectory and the external parameter rotation matrix between the vision sensor and the radar.

8. The step of converting the pose of the current frame under the vision trajectory to the pose of the current frame under the radar trajectory, The steps include determining the rotation matrix of the current frame under the radar trajectory according to the rotation matrix of the current frame under the vision trajectory and the external parameter rotation matrix between the vision sensor and the radar, The steps include calculating a rotation matrix between the vision trajectory and the radar trajectory, The steps include determining the global position of the current frame under the radar trajectory according to the global position of the current frame under the vision trajectory and the rotation matrix between the vision trajectory and the radar trajectory, A positioning method according to claim 1 or 2, comprising the step of determining the pose of the current frame under the radar trajectory according to the global position of the current frame under the radar trajectory and the rotation matrix of the current frame under the radar trajectory.

9. A fused mapping module configured to perform radar mapping and vision mapping using radar and vision sensors, respectively, wherein the vision mapping step comprises determining a keyframe pose, and the pose of the keyframe is the pose of the navigation object in the keyframe; A fused localization module configured to combine radar localization with vision localization based on the pose of the keyframe in order to use vision localization results for navigation on a map acquired by the radar mapping, wherein the pose of a candidate keyframe and the pose of the current frame under the vision trajectory are determined, the candidate keyframe is retrieved in the keyframes of the mapping database of the vision map by using a global descriptor of the image of the current frame, and the pose of the current frame is the pose of the navigation object in the current frame, and the candidate A positioning device comprising a fused positioning module configured to convert the pose of the current frame under the pose of the auxiliary keyframe and the vision trajectory to the pose of the current frame under the pose of the candidate keyframe and the radar trajectory; determine a pose conversion matrix from the candidate keyframe to the current frame under the radar trajectory according to the pose of the candidate keyframe and the pose of the current frame under the radar trajectory; and determine a provisional pose of the navigation object under the radar trajectory according to the pose conversion matrix and the radar pose coupled to the pose of the keyframe.

10. The localization device according to claim 9, wherein the fused mapping module is configured to perform mapping by simultaneously using the radar and the vision sensor, wherein the radar mapping acquires a map for localization and navigation, and the vision mapping acquires a vision map, and combines the poses of the keyframes provided by the vision mapping with the radar poses provided by the radar mapping.

11. The positioning device according to claim 9 or 10, wherein the fused positioning module is further configured to determine the pose of the navigation object in the coordinate system of a grid map by projecting the provisional pose of the 6-degree-of-freedom navigation object onto the provisional pose of the 3-degree-of-freedom navigation object.

12. A location-finding device according to claim 9, configured to perform an operation to carry out the location-finding method described in claim 4.

13. A computer device, Memory configured to store instructions, A computer device comprising a processor configured to execute instructions to perform an operation to perform the location identification method described in any one of claims 1 to 8.

14. A non-temporary computer-readable storage medium that stores computer instructions, when executed by a processor, for performing the location method described in any one of claims 1 to 8.

Images