Visual slam-based path planning method and apparatus, electronic device, and storage medium

Abstract

A visual SLAM-based path planning method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring an offline map of a current scene, the offline map of the current scene being pre-constructed by means of an SLAM algorithm (S100); acquiring a current image frame by means of a camera, and performing ORB feature point extraction on the current image frame (S110); matching ORB feature points with the offline map of the current scene to obtain pose information of the camera (S120); performing target point detection on the current image frame by means of a deep learning model, and using pose transformation to calculate position coordinates of a target point (S130); taking the pose information of the camera as a path planning starting point, and taking the position coordinates of the target point as a path planning end point, using a path planning algorithm to perform path planning on the offline map (S140). According to the method, information of an offline map is fully used, feature points can be accurately matched with map points to obtain accurate pose information.

Description

Visual SLAM path planning method, device, electronic device and storage medium Technical Field

[0001] The present application belongs to the field of simultaneous positioning and mapping technology, and in particular relates to a visual SLAM path planning method, device, electronic device, and storage medium. Background Art

[0002] Currently, there are 43.3 million blind people worldwide, and this number is projected to rise to 61 million by 2050. China has the world's largest population of blind people, with approximately 17 million blind people in the country, according to the China Blind Association. For the visually impaired, leaving home can be even more challenging than physical limitations: erratic sidewalks, unresponsive traffic lights, the absence of guide dogs, or transportation and restaurants that refuse guide dogs. Consequently, up to 30% of the visually impaired choose not to leave their homes. Helping the blind develop awareness and guidance in their daily lives is a pressing issue.

[0003] In the existing field of blind guidance technology, visual SLAM (Simultaneous Localization and Mapping) technology is usually used to guide the movement of visually impaired people. Visual SLAM can help visually impaired people locate their own position and establish a sense of the surrounding environment. The visual SLAM method used determines the effectiveness of blind guidance. SLAM refers to the technology of using sensors mounted on a robot to perceive the surrounding environment without prior information, build an environmental model during movement, and estimate its own position at the same time. However, the visual SLAM technology in the existing technology does not utilize the offline map information in the scene, resulting in inaccurate matching of feature points and map points, and the obtained posture information is not accurate, which ultimately affects the effectiveness of blind guidance.

[0004] Summary of the Invention

[0005] The present application provides a visual SLAM path planning method, device, electronic device and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0006] In order to solve the above problems, this application provides the following technical solutions:

[0007] A visual SLAM path planning method, comprising:

[0008] Obtain an offline map of the current scene, where the offline map of the current scene is pre-built by a SLAM algorithm;

[0009] Acquire a current image frame through a camera, extract ORB feature points from the current image frame using an ORB algorithm, and match the ORB feature points with an offline map of the current scene using a relocalization algorithm to obtain the camera's pose information;

[0010] Detecting target points on the current image frame using a deep learning model, and calculating the position coordinates of the target points using pose transformation;

[0011] The camera's position information is used as the starting point of path planning, the position coordinates of the target point are used as the end point of path planning, and a path planning algorithm is used to perform path planning on the offline map.

[0012] The technical solution adopted in the embodiment of the present application also includes: the offline map includes a point cloud map and an octree map of the current scene, and the offline map of the current scene is obtained specifically as follows:

[0013] Use the SLAM algorithm to map the current scene and obtain a point cloud map;

[0014] Using a hierarchical clustering algorithm to form a K-ary tree format for the map point descriptors in the point cloud map, and converting the point cloud map into an octree map;

[0015] The point cloud map and the octree map are saved in a file format.

[0016] The technical solution adopted in the embodiment of the present application further includes: the ORB feature points include key points and descriptors; obtaining the current image frame through the camera and extracting the ORB feature points from the current image frame using the ORB algorithm include:

[0017] Using the FAST algorithm to find pixels with high intensity changes in the current image frame as key points;

[0018] Calculate the grayscale gradient direction around the key point, determine the direction of the key point according to the grayscale gradient direction, and use it as the direction of the ORB feature point;

[0019] For each keypoint, a fixed-length binary descriptor is generated using the BRIEF algorithm, which generates a binary code by comparing pairs of pixels at specific locations, which corresponds to the image area around the keypoint.

[0020] The technical solution adopted in the embodiment of the present application further includes: matching the ORB feature points with the offline map of the current scene through the relocalization algorithm to obtain the camera's pose information specifically as follows:

[0021] Matching the descriptors of the ORB feature points with the map point descriptors of the offline map to obtain a plurality of matching point pairs;

[0022] The pose of the matching points is solved using an N-point perspective pose solving algorithm.

[0023] The technical solution adopted in the embodiment of the present application further includes: after matching the ORB feature points with the offline map of the current scene using the relocalization algorithm to obtain the camera's pose information, it also includes:

[0024] The bundle adjustment method is used to optimize the pose information of the current camera and the IMU data to obtain optimized pose information.

[0025] The technical solution adopted in the embodiment of the present application also includes: detecting the target point in the current image frame by using a deep learning model, and calculating the position coordinates of the target point by using posture transformation, specifically:

[0026] Perform target point detection on the current image frame using a deep learning model to obtain an image detection frame of the target point and image coordinates of the image detection frame;

[0027] The world coordinates of the target point are obtained by calculating the pose transformation based on the image coordinates.

[0028] The technical solution adopted in the embodiment of the present application further includes: after performing path planning on the offline map using the path planning algorithm, the following steps are further included:

[0029] Based on the path planning results, the route information is converted into voice playback.

[0030] Another technical solution adopted in the embodiment of the present application is: a visual SLAM path planning device, comprising:

[0031] Offline construction module: used to obtain the offline map of the current scene, where the offline map of the current scene is pre-constructed by the SLAM algorithm;

[0032] Online matching module: used to obtain the current image frame through the camera, extract ORB feature points from the current image frame using the ORB algorithm, and match the ORB feature points with the offline map of the current scene through the relocalization algorithm to obtain the camera's pose information;

[0033] Target detection module: used to detect target points on the current image frame through a deep learning model, and calculate the position coordinates of the target points using posture transformation;

[0034] Path planning module: used to use the camera's position information as the starting point of path planning, the position coordinates of the target point as the end point of path planning, and use a path planning algorithm to perform path planning on the offline map.

[0035] Another technical solution adopted in an embodiment of the present application is: an electronic device, which includes a processor, a memory, a computer program stored in the memory and runnable on the processor, a communication interface, and an external device; when the processor executes the computer program, the above-mentioned visual SLAM path planning method is implemented.

[0036] Another technical solution adopted in the embodiment of the present application is: a storage medium, wherein the storage medium stores a computer program executable by a processor, and the computer program is used to execute the above-mentioned visual SLAM path planning method.

[0037] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: the visual SLAM path planning method, device, electronic device, and storage medium of the embodiments of the present application pre-construct an offline map of the current scene, calculate the current pose information by matching the offline map with the relocation algorithm, use the deep learning model to detect the target point in the current image frame, obtain the target point detection frame coordinates in the current image frame and transform them into world coordinates, and perform path planning based on the current pose information and the world coordinates of the target point. The embodiments of the present application make full use of the information of the offline map, can more accurately match feature points with map points, obtain more accurate pose information, have better adaptability to the relocation task in visual SLAM, and can better help visually impaired people establish perception of the outside world and guide the blind in daily life. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a flow chart of a visual SLAM path planning method according to a first embodiment of the present application;

[0039] FIG2 is a flow chart of a visual SLAM path planning method according to a second embodiment of the present application;

[0040] FIG3 is a schematic diagram of the structure of a visual SLAM path planning device according to an embodiment of the present application;

[0041] FIG4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products, or electronic devices.

[0044] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0045] Please refer to Figure 1, which is a flow chart of the visual SLAM path planning method of the first embodiment of the present application. The visual SLAM path planning method can be applied to electronic devices such as smartphones, augmented reality (AR) / virtual reality (VR) devices, and tablet computers. The present embodiment of the application does not impose any restrictions on the specific type of electronic device. Specifically, the visual SLAM path planning method of the first embodiment of the present application includes the following steps:

[0046] S100: Acquire an offline map of the current scene, where the offline map of the current scene is pre-built by a SLAM algorithm;

[0047] In this step, the offline map includes the point cloud map and octree map of the current scene. The offline map is constructed in the following ways:

[0048] S101: Use the SLAM algorithm to map the current scene and obtain a point cloud map;

[0049] Point cloud mapping is a 3D map representation method consisting of a large number of discrete points, each with spatial coordinates and other attributes such as color and normals. Point cloud mapping uses lidar or other sensors to capture point cloud data and describes the 3D structure and characteristics of the environment. Point cloud mapping can provide more accurate and detailed environmental models to support applications such as autonomous driving and robotic navigation.

[0050] S102: using a hierarchical clustering algorithm to form a K-ary tree format for the map point descriptors in the point cloud map, and converting the point cloud map into an octree map;

[0051] Among them, the octree map is a commonly used data structure for point cloud maps. By dividing the space into eight equally sized sub-regions and recursively dividing each sub-region, the point cloud data is organized into a tree structure. Each node represents a sub-region and contains the point cloud data or other attributes within that region. This hierarchical structure enables the octree map to efficiently store and access point cloud data, making it better suited for application scenarios such as dynamic environments, local map updates, and fast searches.

[0052] S103: Save the point cloud map and the octree map into a file format;

[0053] S110: Based on the offline map of the current scene, a camera is used to obtain a current image frame, and an ORB (Oriented FAST and Rotated BRIEF) algorithm is used to extract ORB feature points from the current image frame.

[0054] In this step, the cameras used include but are not limited to binocular cameras or RGB-D cameras. The ORB algorithm is a manually designed feature point descriptor algorithm in computer vision. It combines the advantages of the FAST (Features from Accelerated Segment Test) key point detection algorithm and the BRIEF (Binary Robust Independent Elementary Features) descriptor algorithm. It can robustly detect and describe image feature points under different scales, rotations, and lighting conditions.

[0055] In the embodiment of the present application, the ORB feature point includes two parts: a key point and a descriptor. The ORB feature point extraction process using the ORB algorithm is as follows: first, the FAST algorithm is used to find pixels with high intensity changes in the current image frame as key points; then, the grayscale gradient direction around the key point is calculated, and the grayscale gradient direction is used as the direction of the ORB feature point, so that the ORB feature point can be invariant at different rotation angles, thereby improving robustness and matching performance. For each key point, a fixed-length binary descriptor is generated using the BRIEF algorithm, and a binary code is generated by comparing pixel pairs at a specific position. The binary code corresponds to the image area around the key point. Due to the use of binary descriptors, ORB feature points have the characteristics of fast matching and efficient storage.

[0056] S120: Matching the ORB feature points with the offline map of the current scene through a relocalization algorithm to obtain the current camera's pose information, and using the current camera's pose information as the starting point for path planning;

[0057] In an embodiment of the present application, the matching process of the ORB feature point and the offline map is specifically as follows: matching the descriptor of the ORB feature point of the current image frame with the map point descriptor of the offline map to obtain several pairs of matching points, and using the PnP (Perspective-n-Point, N-point perspective pose solution) algorithm to solve the pose of the matching points. Among them, the PnP algorithm is an algorithm used in computer vision to solve the camera pose, which estimates the camera pose through known three-dimensional points and two-dimensional points in the corresponding image. The basic principle of the PnP algorithm is to use at least 3 non-collinear three-dimensional points and their corresponding two-dimensional points in the image to calculate the camera pose by solving the perspective transformation matrix. The input of the PnP algorithm includes the coordinates of the three-dimensional point and the coordinates of the two-dimensional point of the corresponding image, and the output is the rotation matrix and translation vector of the camera.

[0058] Furthermore, PnP algorithms include EPnP (Efficient PnP), UPnP (Uncalibrated PnP), and Direct Linear Transform (DLT) and other solving algorithms. Among them: the EPnP algorithm solves the camera pose by minimizing the reprojection error, adopts the linear least squares method, and uses SVD decomposition to calculate the camera pose, which has high computational efficiency and accuracy. The UPnP algorithm estimates the camera pose by minimizing the ray projection error when the camera is not internally calibrated. The UPnP algorithm requires known constraints such as the camera viewing angle, camera plane, and depth information of three-dimensional points. The DLT algorithm is a classic experimental method for solving the PnP problem, which estimates the camera pose by linearly solving the perspective transformation matrix. The computational accuracy of the DLT algorithm may be affected by noise and errors, so it is usually necessary to use a nonlinear optimization algorithm for further precise estimation.

[0059] S130: Based on the current image frame, perform target point detection on the current image frame using a deep learning model, use the target point detection result as the path planning end point, and calculate the position coordinates of the path planning end point using posture transformation;

[0060] In this step, the deep learning model used includes but is not limited to YOLO or SSD (Single Shot MultiBox Detector). In the embodiment of the present application, taking the YOLO5 model as an example to detect the target point of the current image frame, the specific detection process includes:

[0061] S131: Use the yolov5 model to detect the target point on the current image frame, and obtain the image detection frame of the target point and the image coordinates of the image detection frame;

[0062] S132: Obtain the world coordinates of the target point through pose transformation calculation based on the image coordinates;

[0063] The calculation process of the world coordinates specifically includes:

[0064] Step 1: Get the depth z of each image point i ;

[0065] Among them, the depth z i The acquisition method includes: using an RGB-D depth camera to obtain the depth z of each image point from the depth map i , or use a stereo camera to recover the depth z from the disparity map i : Given the focal length f and baseline b of the binocular camera, OpenCV can be used to obtain the disparity d of each image point, and further calculate the depth z of each image point i :

[0066] Step 2: Calculate the average depth of all image points Average the depth As the distance between the current camera and the target point:

[0067] Optionally, an external ultrasonic rangefinder can be used to obtain the distance between the current camera and the target point.

[0068] Step 3: Based on distance The calibrated camera intrinsic parameter matrix K and camera pose R cw With t cw Calculate the world coordinates P of the target point w for:

[0069] In formula (3), the subscript cw represents the transformation from the world coordinate system to the camera coordinate system, and the subscript w represents the coordinate in the world coordinate system.

[0070] S140: Using the current camera's pose information as the path planning starting point and the target point's position coordinates as the path planning end point, a path planning algorithm is used to perform path planning on the octree map.

[0071] In this step, the path planning algorithm can select the bidirectional rapidly expanding random tree (RRT-Connect) algorithm or the rapid exploration random tree (Rapid-exploring random tree, RRT) algorithm. This application does not limit the specific implementation method of path planning. Taking the bidirectional rapidly expanding random tree (RRT-Connect) algorithm as an example, the path planning process is specifically as follows: two expansion trees are grown from the starting point and the target point respectively. In each iteration process, one of the trees is expanded, and the nearest node of the other tree is tried to be connected to expand the new node.

[0072] S150: Based on the path planning result, convert the route information into voice playback;

[0073] In this embodiment of the present application, the TextToSpeech engine can be used for voice broadcasting. The voice broadcast content includes but is not limited to: distance information to the target point And the route information obtained based on path planning, etc.

[0074] Based on the above, the visual SLAM path planning method of the first embodiment of the present application pre-builds an offline map of the current scene, calculates the current pose information by matching the relocation algorithm with the offline map, uses a deep learning model to detect the target point in the current image frame, obtains the target point detection frame coordinates in the current image frame and transforms them into world coordinates, and performs path planning and voice broadcast based on the current pose information and the world coordinates of the target point. The embodiment of the present application makes full use of the information of the offline map, can more accurately match feature points with map points, obtain more accurate pose information, has better adaptability to the relocation task in visual SLAM, and can better help visually impaired people establish perception of the outside world and guide the blind in daily life.

[0075] Please refer to Figure 2, which is a flow chart of the visual SLAM path planning method of the second embodiment of the present application. The visual SLAM path planning method of the second embodiment of the present application includes the following steps:

[0076] S200: Acquire an offline map of the current scene, where the offline map of the current scene is pre-built by a SLAM algorithm;

[0077] This step is the same as S100 in the first embodiment. For details, please refer to the relevant description of S100 and will not be repeated here.

[0078] S210: Based on the offline map of the current scene, a camera is used to obtain a current image frame, and an ORB (Oriented FAST and Rotated BRIEF) algorithm is used to extract feature points from the current image frame.

[0079] This step is the same as S110 in the first embodiment. For details, please refer to the relevant description of S110, which will not be repeated here.

[0080] S220: Matching the ORB feature points with the offline map of the current scene through a relocalization algorithm to obtain the current camera's pose information, and using the current camera's pose information as the starting point for path planning;

[0081] This step is the same as S120 in the first embodiment. For details, please refer to the relevant description of S120, which will not be repeated here.

[0082] S230: Optimizing the current camera's pose information and IMU data using a bundle adjustment method to obtain optimized pose information;

[0083] In this step, an IMU (Inertial Measurement Unit) is a device that integrates multiple inertial sensors to measure the acceleration and angular velocity of an object. An IMU is usually composed of two main sensors: an accelerometer and a gyroscope. Accelerometer: An accelerometer is used to measure the acceleration of an object in three axes and can detect the linear motion and acceleration of gravity of an object. The output unit of the accelerometer is m / s 2 , the speed and position can be calculated by integrating the acceleration data. Gyroscope: The gyroscope is used to measure the angular velocity of an object around three axes, and can detect the rotation and angle change of the object. The output unit of the gyroscope is degree / second or radian / second, and the attitude and direction can be calculated by integrating the angular velocity data. IMU is generally used in inertial navigation, attitude estimation and motion tracking. By combining the data of the accelerometer and gyroscope, the attitude and motion state of the object can be estimated. In visual inertial SLAM, it is necessary to measure the pose T of the i-th frame. i and speed v i , and the bias of the gyroscope and accelerometer and Specifically: T i =[R i ,p i ]∈SE(3) (4)

[0084] It is necessary to obtain the IMU pre-integrated measurement value between the consecutive visual frames i and i+1, which is ΔR i,i+1 , Δv i,i+1 , Δp i,i+1 and the covariance matrix of the entire measurement vector and inertial residuals

[0085] where Log:SO(3)→R3 represents the mapping from Lie group to Lie algebra. Together with the inertial residual, using frame i and position x j The reprojection error r between the 3D points j at ij :

[0086] Where, ∏:R3→R n is the projection function corresponding to the camera model, u ij is the observation of point j in image frame i, with covariance matrix Σ ij , T CB ∈SE(3) represents the rigid transformation from the body IMU to the camera (left or right), which is known from the calibration, is the SE(3) group in Element-wise transformation operations.

[0087] In the embodiment of the present application, the camera can be approximated to the position of the IMU, T CB It can be approximated as the identity matrix I; define the state vector:

[0088] Given a set of k+1 keyframes and their states and a set of l three-dimensional points and their states The visual-inertial optimization problem can be formulated as follows:

[0089] where κ j is the set of key frames for observing 3D point j. For the reprojection error, the robust Huber kernel ρ is used Hub To reduce the impact of mismatching.

[0090] S240: Based on the current image frame, perform target point detection on the current image frame using a deep learning model, use the target point detection result as the path planning end point, and calculate the position coordinates of the path planning end point using posture transformation;

[0091] This step is the same as S130 in the first embodiment. For details, please refer to the relevant description of S130, which will not be repeated here.

[0092] S250: Using the current camera's pose information as the starting point of path planning and the target point's position coordinates as the end point of path planning, a path planning algorithm is used to perform path planning on the octree map.

[0093] This step is the same as S140 in the first embodiment. For details, please refer to the relevant description of S140, which will not be repeated here.

[0094] S260: Based on the path planning result, convert the route information into voice playback;

[0095] This step is the same as S150 in the first embodiment. For details, please refer to the relevant description of S150, which will not be repeated here.

[0096] Based on the above, the visual SLAM path planning method of the second embodiment of the present application is based on the first embodiment. By simultaneously optimizing the IMU data and the camera pose information through the bundle adjustment method, the accuracy and robustness can be further improved, thereby obtaining more accurate pose information.

[0097] Please refer to Figure 3, which is a schematic diagram of the structure of the visual SLAM path planning device according to an embodiment of the present application. The visual SLAM path planning device according to an embodiment of the present application includes:

[0098] Offline construction module 31: used to obtain an offline map of the current scene, where the offline map of the current scene is pre-constructed by a SLAM algorithm;

[0099] The above-mentioned offline construction module 31 includes:

[0100] The scene reconstruction unit is used to map the current scene using the SLAM algorithm to obtain a point cloud map;

[0101] A map conversion unit, used to convert a point cloud map into an octree map;

[0102] The map saving unit is used to use a hierarchical clustering algorithm to form a K-ary tree format of feature point descriptors corresponding to map points in the point cloud map, and save the point cloud map and the octree map into a file format.

[0103] The online matching module 32 is used to obtain the current image frame using the camera based on the offline map of the current scene, and extract feature points from the current image frame using the ORB algorithm. The ORB feature points are matched with the offline map of the current scene using the relocalization algorithm to obtain the current camera's pose information, and the pose information of the current camera is used as the starting point for path planning.

[0104] The online matching module 32 includes:

[0105] A feature point extraction unit, used to extract feature points of the current image frame;

[0106] Relocalization unit, used to match feature points with map points of the offline map and calculate the current camera pose information;

[0107] The posture optimization unit uses the bundle adjustment method to optimize the posture information and IMU data to obtain more accurate posture information.

[0108] The target detection module 33 is used to detect target points on the current image frame using a deep learning model, use the target point detection result as the path planning end point, and calculate the position coordinates of the path planning end point using posture transformation;

[0109] The target detection module 33 includes:

[0110] The target detection unit is used to detect the target point in the current image frame using a deep learning model to obtain the image detection frame of the target point and the image coordinates of the image detection frame;

[0111] Distance estimation unit, used to calculate the distance between the current camera and the target point;

[0112] The coordinate calculation unit is used to calculate the world coordinates of the target point based on the distance, the calibrated camera intrinsic parameter matrix and the camera pose.

[0113] Path planning module 34: used to use the current camera's position information as the starting point of path planning and the position coordinates of the target point as the end point of path planning, and use the path planning algorithm to perform path planning on the octree map;

[0114] Voice broadcast module 35: used to convert route information into voice broadcast based on the path planning results.

[0115] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0116] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0117] Please refer to Figure 4, which is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 4 includes: one or more processors 41 (only one is shown in the figure), a memory 42, a computer program 43 stored in the memory 42 and executable on the processor 41, a communication interface 44, and an external device 45. When the processor 41 executes the computer program 43, it implements the steps of the above-mentioned various visual SLAM path planning method embodiments.

[0118] Those skilled in the art will understand that Figure 4 is merely an example of the electronic device 4 and does not constitute a limitation on the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0119] The processor 41 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0120] The memory 42 may be an internal storage unit of the electronic device 4, such as a hard drive or memory of the electronic device 4. The memory 42 may also be an external storage device of the electronic device 4, such as a plug-in hard drive, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device 4. Furthermore, the memory 42 may include both an internal storage unit of the electronic device 4 and an external storage device. The memory 42 is used to store the computer program and other programs and data required by the electronic device. The memory 42 may also be used to temporarily store data that has been output or is about to be output.

[0121] The external device may be a binocular camera and an IMU.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0123] An embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0124] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0125] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0130] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A visual SLAM path planning method, characterized in that: include: Obtaining an offline map of the current scene, where the offline map of the current scene is pre-built by a SLAM algorithm; Acquire a current image frame through a camera, extract ORB feature points from the current image frame using an ORB algorithm, and match the ORB feature points with an offline map of the current scene through a relocalization algorithm to obtain the position information of the camera; Detecting target points on the current image frame using a deep learning model, and calculating the position coordinates of the target points using posture transformation; The position information of the camera is used as the starting point of the path planning, the position coordinates of the target point are used as the end point of the path planning, and the path planning algorithm is used to perform the path planning on the offline map.

2. The visual SLAM path planning method according to claim 1, characterized in that: The offline map includes a point cloud map and an octree map of the current scene. The offline map of the current scene is obtained as follows: Use the SLAM algorithm to map the current scene to obtain a point cloud map; Using a hierarchical clustering algorithm to form a K-tree format for map point descriptors in the point cloud map, and converting the point cloud map into an octree map; The point cloud map and the octree map are saved in a file format.

3. The visual SLAM path planning method according to claim 2, characterized in that The ORB feature points include key points and descriptors, and the process of acquiring the current image frame through a camera and extracting ORB feature points from the current image frame using an ORB algorithm includes: Using the FAST algorithm, searching for pixels with high intensity changes in the current image frame as key points; Calculate the grayscale gradient direction around the key point, and determine the key point according to the grayscale gradient direction. Direction, and use it as the direction of the ORB feature point; For each keypoint, a fixed-length binary descriptor is generated using the BRIEF algorithm, which generates a binary code by comparing pairs of pixels at specific locations, which corresponds to the image area around the keypoint.

4. The visual SLAM path planning method according to claim 3, wherein: The ORB feature points are matched with the offline map of the current scene by the relocalization algorithm to obtain the position information of the camera: Matching the descriptor of the ORB feature point with the map point descriptor of the offline map to obtain a plurality of pairs of matching points; The pose of the matching points is solved using an N-point perspective pose solving algorithm.

5. The visual SLAM path planning method according to claim 4, characterized in that: After matching the ORB feature points with the offline map of the current scene by the relocalization algorithm to obtain the position information of the camera, the method further includes: The bundle adjustment method is used to optimize the pose information of the current camera and the IMU data to obtain optimized pose information.

6. The visual SLAM path planning method according to claim 1, characterized in that: The target point detection is performed on the current image frame by using the deep learning model, and the position coordinates of the target point are calculated by using the posture transformation as follows: Use a deep learning model to perform target point detection on the current image frame to obtain an image detection frame of the target point and image coordinates of the image detection frame; The world coordinates of the target point are calculated through posture transformation based on the image coordinates.

7. The visual SLAM path planning method according to any one of claims 1 to 6, characterized in that: After performing path planning on the offline map using the path planning algorithm, the method further includes: Based on the path planning results, the route information is converted into voice playback.

8. A visual SLAM path planning device, characterized in that: include: Offline construction module: used to obtain an offline map of the current scene, where the offline map of the current scene is pre-constructed by a SLAM algorithm; Online matching module: used to obtain the current image frame through the camera, extract ORB feature points from the current image frame using the ORB algorithm, and match the ORB feature points with the offline map of the current scene through the relocation algorithm to obtain the position information of the camera; Target detection module: used to detect target points on the current image frame through a deep learning model, and obtain the position coordinates of the target points by using posture transformation calculation; Path planning module: used to use the camera's position information as the starting point of path planning, the position coordinates of the target point as the end point of path planning, and use a path planning algorithm to perform path planning on the offline map.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory, a computer program stored in the memory and executable on the processor, a communication interface, and an external device; when the processor executes the computer program, the visual SLAM path planning method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program executable by a processor, and the computer program is used to execute the visual SLAM path planning method according to any one of claims 1 to 7.

Images