Navigation method and device based on depth camera, unmanned vehicle and storage medium

Abstract

The application provides a navigation method and device based on a depth camera, an unmanned vehicle and a storage medium. The method comprises the following steps: obtaining a target position according to target order information and a preset priority rule; obtaining a current position of a target unmanned vehicle according to the target position, a preset prior map corresponding to a target park and a real-time map corresponding to the target park, wherein the preset prior map is obtained based on a color image and a depth image output by a depth camera, and the real-time map is obtained by processing the preset prior map; performing path planning according to the current position and the target position to obtain an optimal path; and moving the target unmanned vehicle to the target position according to the optimal path. The depth camera is used to obtain a depth image, and a computer directly obtains the depth image, so that additional calculation is not needed, thereby reducing the hardware requirement for the target unmanned vehicle and improving the real-time navigation.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a navigation method and device based on a depth camera, an unmanned vehicle, and a storage medium. Background Art

[0002] With the growth of industries like food delivery and online shopping, the number of packages delivered by the logistics and distribution industry has steadily increased. Furthermore, with rising labor costs, traditional delivery methods relying on couriers and delivery drivers are not only expensive but also inefficient. The most critical challenge in logistics and distribution is connecting senders and recipients, which consumes significant manpower and time. While establishing collection points can partially address this issue, it hasn't fundamentally solved the fundamental problems of heavy logistics pressure and difficulty for users to receive their goods. Consumers are eager for more convenient ways to send and receive goods.

[0003] Currently, using robots to deliver packages within industrial parks through autonomous driving technology is widely recognized as an effective solution to these problems, with SLAM technology being a key technology. While commonly used binocular camera-based point feature SLAM solutions are easy to initialize and can acquire dimensional information, unmanned navigation applications require the construction of depth filters to acquire depth maps for dense point cloud mapping. However, this involves point cloud reconstruction, consuming significant hardware resources and exhibiting limited real-time performance. Summary of the Invention

[0004] The present invention provides a depth camera-based navigation method, device, unmanned vehicle and storage medium, the main purpose of which is to reduce the hardware requirements for the target unmanned vehicle and effectively improve the real-time performance of the target unmanned vehicle.

[0005] In a first aspect, an embodiment of the present invention provides a navigation method based on a depth camera, comprising:

[0006] Obtain the target location based on the target order information and preset priority rules;

[0007] Obtaining the current position of the target unmanned vehicle based on the target position, a preset a priori map corresponding to the target park, and a real-time map corresponding to the target park, wherein the preset a priori map is obtained based on a color image and a depth map output by a depth camera, and the real-time map is obtained by processing the preset a priori map;

[0008] Performing path planning based on the current position and the target position to obtain an optimal path;

[0009] According to the optimal path, the target unmanned vehicle is moved to the target position.

[0010] Preferably, the preset prior map is obtained based on the color map and depth map output by the depth camera, and is obtained by the following steps:

[0011] Acquire the color image and the depth image according to the depth camera;

[0012] Perform feature matching based on the color image to obtain feature point information;

[0013] According to the feature point information and the depth map, pose calculation is performed using a PnP algorithm, and the preset prior map is further obtained.

[0014] Preferably, the real-time map is obtained by processing the preset prior map through the following steps:

[0015] According to the key frames stored in the memory of the target unmanned vehicle, if a loop is detected, obtaining the loop key frames;

[0016] According to the preset time period corresponding to the loop key frame, obtaining the key frame in the hard disk of the target unmanned vehicle corresponding to the preset time period;

[0017] Obtaining an optimal loop based on the loop key frame and the key frame corresponding to the preset time period;

[0018] The poses and map points of the key frames in the optimal loop are optimized to obtain the real-time map.

[0019] Preferably, the key frames stored in the memory of the target unmanned vehicle and the key frames in the hard disk of the target unmanned vehicle are obtained by the following steps:

[0020] Selecting a plurality of key frames from the color image based on feature point information, wherein the feature points are obtained based on the color image and the depth image of the depth camera;

[0021] According to the overlap rate of feature points between two adjacent key frames, the loop possibility of each key frame is obtained;

[0022] The key frames whose loop possibility is not greater than the preset threshold are stored in the hard disk of the target unmanned vehicle, and the key frames whose loop possibility is greater than the preset threshold are stored in the memory of the target unmanned vehicle.

[0023] Preferably, the step of optimizing the poses and map points of the key frames in the optimal loop to obtain the real-time map comprises the following steps:

[0024] Processing the poses of the key frames in the optimal loop, the color images of the map points, and the depth images of the map points to generate a color point cloud image;

[0025] Segmenting the color point cloud image through a ground filter to obtain an octree map;

[0026] Projecting the octree map along the Z axis to obtain an occupancy grid map;

[0027] The octree map or the occupancy grid map is used as the real-time map.

[0028] Preferably, performing path planning based on the current position and the target position to obtain the optimal path includes the following steps:

[0029] Performing global path planning based on the current position and the target position to obtain a global path;

[0030] Moving the target unmanned vehicle according to the global path to obtain the actual position reached by the target unmanned vehicle;

[0031] If the actual position is not the target position, performing local path planning based on the actual position and the target position to obtain a local path;

[0032] According to the actual position, the target position and the local path, the target unmanned vehicle is moved, and the above steps are repeated until the actual position of the target unmanned vehicle is the target position.

[0033] Preferably, the navigation method further includes:

[0034] According to the color map, the YOLO v5 algorithm is used to detect the target and obtain roadblock information;

[0035] Determining the size and position of the roadblock information according to the roadblock information and the depth map;

[0036] The size and location of the roadblock information are added to the preset prior map.

[0037] In a second aspect, an embodiment of the present invention provides a navigation device based on a depth camera, comprising:

[0038] The location module is used to obtain the target location based on the target order information and preset priority rules;

[0039] a positioning module, configured to obtain the current position of the target unmanned vehicle based on the target position, a preset a priori map corresponding to the target park, and a real-time map corresponding to the target park, wherein the preset a priori map is obtained based on the color image and depth map output by the depth camera, and the real-time map is obtained by processing the preset a priori map;

[0040] A path module is used to plan a path based on the current location and the target location to obtain an optimal path;

[0041] A moving module is used to move the target unmanned vehicle to the target position according to the optimal path.

[0042] In a third aspect, an embodiment of the present invention provides an unmanned vehicle, comprising a navigation device based on a depth camera provided in the second aspect.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned navigation method based on a depth camera are implemented.

[0044] The present invention proposes a depth camera-based navigation method, device, unmanned vehicle and storage medium, which use a depth camera to obtain a depth map. A computer can directly obtain the depth map without the need for additional calculations, thereby reducing the hardware requirements for the target unmanned vehicle and improving the real-time performance of navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a depth camera-based navigation method provided by an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of obtaining a preset prior map in an embodiment of the present invention;

[0047] Figure 3 This is a flow chart of obtaining a preset prior map by a target unmanned vehicle in one embodiment of the present invention;

[0048] Figure 4 This is a flowchart of obtaining a real-time map in an embodiment of the present invention;

[0049] Figure 5 This is a flow chart of generating a real-time map by a target unmanned vehicle in one embodiment of the present invention;

[0050] Figure 6 This is a flowchart of obtaining key frames stored in the memory and hard disk of a target unmanned vehicle in an embodiment of the present invention;

[0051] Figure 7 This is a flowchart of optimizing the poses and map points of key frames in the optimal loop to obtain a real-time map in an embodiment of the present invention;

[0052] Figure 8 This is a flow chart of obtaining the best path in an embodiment of the present invention;

[0053] Figure 9A schematic diagram of a module of a depth camera-based navigation device provided by an embodiment of the present invention.

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0056] The embodiment of the present invention provides a visual SLAM solution based on a depth camera, which completes the positioning of the drone and mapping of the surrounding environment, constructs an octree map or an occupancy grid map, and realizes map construction; and designs a large-scale map management method to complete large-scale map construction and loop detection; designs a vision-based navigation method, which uses a preset prior map and target detection technology, combined with global path planning and local path planning, to realize the navigation of the unmanned vehicle in the target park or indoors.

[0057] Figure 1 A flowchart of a navigation method based on a depth camera provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0058] S110, obtaining a target location according to target order information and preset priority rules;

[0059] First, the target order information is input to the target unmanned vehicle. The target order information can be one or more. The specific input method can be inputting the target order information to the target unmanned vehicle through a keyboard or other means, or the host computer can send the target order information to the target unmanned vehicle through a local area network. The specific method can be determined according to the actual situation and is not specifically limited in the embodiment of the present invention. In addition, the target information includes the delivery method or pickup method, as well as the target location information. The specific method can be determined according to the actual situation and is not specifically limited in the embodiment of the present invention. The preset priority rule in the embodiment of the present invention means that when there are multiple orders in the target order information, the appropriate pickup order can be set according to the target location in each order. The order information can be divided into different categories according to the direction of the target address. Orders in the same direction can be processed together, thereby improving the delivery and pickup efficiency of the target unmanned vehicle. The target location in the embodiment of the present invention refers to the location information recorded in the target order information. When navigating the target unmanned vehicle to the target location, it first navigates to the target location in the target order information processed first, and then processes each order in turn according to the same method.

[0060] S120, obtaining a current position of the target unmanned vehicle based on the target position, a preset a priori map corresponding to the target park, and a real-time map corresponding to the target park, wherein the preset a priori map is obtained based on a color image and a depth map output by a depth camera, and the real-time map is obtained by processing the preset a priori map;

[0061] Then, the current position of the target unmanned vehicle is obtained based on the target location, the preset a priori map corresponding to the target park, and the real-time map corresponding to the target park. In the embodiment of the present invention, the target park refers to the express delivery and receiving area that the target unmanned vehicle is responsible for. It can be inside a certain park, indoors, or in a specific area. The preset a priori map corresponding to the target park refers to a pre-scanned map of the entire target park, or it can be designed in advance based on the engineering map of the target park. It can be determined according to actual conditions, and the embodiment of the present invention does not make specific restrictions on this. The real-time map corresponding to the target park refers to the actual map determined by the target unmanned vehicle based on the actual road conditions during the actual delivery or pickup process. For example, some obstacles are temporarily placed on the road surface, and the preset a priori map does not have the location information of the obstacle. Therefore, sometimes it is impossible to navigate to the target location based on the preset a priori map alone. In this case, it is necessary to redraw the actual map, record the location of the obstacles on the road, and perform local path planning based on the actual map.

[0062] It should be noted that the preset prior map in the embodiment of the present invention is obtained based on the color map and depth map output by the depth camera. In the prior art method of obtaining a depth map based on a binocular camera, an ordinary camera cannot obtain the depth information of the pixel points. Therefore, the computer needs to obtain the depth map through a large number of calculations based on the color map taken by the ordinary binocular camera. Since each pixel point needs to calculate the depth information, and the calculation process of the depth information is very complicated, the point feature visual SLAM solution based on the binocular camera has high requirements for hardware and low real-time performance. In the embodiment of the present invention, a depth camera is used to obtain a depth map, and the computer can directly obtain the depth map without the need for additional calculations, thereby reducing the hardware requirements for the target unmanned vehicle and improving the real-time performance of navigation.

[0063] S130, performing path planning based on the current position and the target position to obtain an optimal path;

[0064] Then, based on the current position and target position, path planning is performed. Path planning is divided into global path planning based on complete prior information and local path planning based on sensor information, depending on the degree of understanding of environmental information, that is, the preset prior map. Among them, from the perspective of whether the obstacle information is static or dynamic, global path planning belongs to static planning (also known as offline planning), and local path planning belongs to dynamic planning (also known as online planning). Global path planning requires the mastery of all environmental information and the planning of paths based on all information in the environmental map; local path planning only requires sensors to collect environmental information in real time, understand the environmental map information, and then determine the location of the map and its local obstacle distribution, so that the optimal path from the current node to a sub-target node can be selected.

[0065] S140: Move the target unmanned vehicle to the target location according to the optimal path.

[0066] Based on the optimal path, the target unmanned vehicle's wheel rotation angle and speed are controlled to move it to the target location. Typically, an unmanned vehicle consists of a chassis, body, industrial computer, driver board, display, antenna, depth camera, and ultrasonic sensor. The chassis includes four steering wheels controlled by motors.

[0067] An embodiment of the present invention provides a navigation method based on a depth camera, which uses a depth camera to obtain a depth map. A computer can directly obtain the depth map without the need for additional calculations, thereby reducing the hardware requirements for the target unmanned vehicle and improving the real-time performance of navigation.

[0068] Based on the above embodiment, preferably, Figure 2 This is a flow chart for obtaining a preset priori map in an embodiment of the present invention. Figure 2 As shown, the preset prior map is obtained based on the color image and depth image output by the depth camera. The specific steps are as follows:

[0069] S210, acquiring the color image and the depth image according to the depth camera;

[0070] S220, performing feature matching based on the color image to obtain feature point information;

[0071] S230 , performing pose calculation using a PnP algorithm based on the feature point information and the depth map, and further acquiring the preset prior map.

[0072] In the embodiment of the present invention, the color image and depth image captured by the depth camera are first obtained. The color image is a image in which pixels represent colors. In addition to the ordinary color image, the depth camera can also capture a depth image. The depth image is a image in which pixels represent object distances. Feature matching is performed on the color image and the depth image. The specific feature matching method is to first perform feature extraction and feature description to find feature points with significant features, and then describe two significant feature points separately. Based on the similarity of the description, it is determined whether the two significant feature points are the same feature point, thereby obtaining feature point information. Finally, based on the feature point information and depth map found, the pose calculation is performed using the PnP (Perspective-n-Point) algorithm. The preset prior map is calculated using the known three-dimensional spatial poses of the n points.

[0073] Figure 3 FIG. 1 is a flow chart of obtaining a preset prior map by a target unmanned vehicle in one embodiment of the present invention. Figure 3 As shown in the figure, we first obtain the model information of the target unmanned vehicle. The target unmanned vehicle has two depth cameras: a front depth camera and a rear depth camera. Next, we obtain information from the front depth camera, the rear depth camera, the binocular camera, and the ultrasonic sensor. We then construct the relationship between the depth camera, ultrasonic sensor, and chassis position changes, and build a TF tree to unify the coordinate systems.

[0074] Feature points are extracted from the first two color frames of the depth camera and matched. The depth information of the feature points is converted from the depth map into 3D points in the camera coordinate system. The RANSC algorithm is used to eliminate mismatched feature point pairs, and the PnP algorithm is used to calculate the pose of the current frame to obtain a preset prior map. The feature points extracted in this embodiment of the present invention are ORB point feature points. ORB stands for Oriented FAST and Rotated BRIEF, a fast feature point extraction and description algorithm. It is an improvement on the FAST (Features from Accelerated Segment Test) algorithm and uses an image pyramid to ensure the multi-scale invariance of feature points. Based on FAST, ORB calculates the grayscale centroid and adds orientation information to the feature points, achieving rotation invariance. A rotation factor is added to the BRIEF feature description, and rBRIEF (Rotation-Aware Brief) is used to describe ORB point features. The BRIEF feature descriptor selects 256 pairs of pixels pi and qi (i = 1, 2, ..., 256) within the neighborhood of a feature point. Then compare the grayscale values ​​of each point pair. If I(pi)>I(qi), generate a 1 in the binary string, otherwise it is 0. All point pairs are compared to generate a binary string of length 256.

[0075] It should be noted that, in the embodiment of the present invention, a feature point map is further provided to complete the storage of map points. Map points are three-dimensional points, which come from three-dimensional objects in the real world and have unique IDs, which come from extracted ORB feature points.

[0076] It should also be noted that in this embodiment of the present invention, when initialization begins, the first and second color image frames are set as keyframes, and the first frame's location is set as the starting point of the unmanned logistics vehicle, coordinates (0, 0, 0). Keyframes are an important concept in visual odometry. In addition to the two keyframes selected during initialization, the remaining keyframes are selected based on the following criteria: frames with clear images, sufficient feature points, and evenly distributed feature points are selected as keyframes. The relationship between keyframes and other keyframes must have a certain degree of common view but not too high a degree of duplication, so as to achieve the effect of both constraints and minimal information redundancy. Redundant keyframes will be deleted.

[0077] Based on the above embodiment, preferably, Figure 4 This is a flowchart of obtaining a real-time map in an embodiment of the present invention. Figure 4 As shown, the real-time map is obtained by processing the preset prior map, and is obtained through the following steps:

[0078] S410, based on the key frames stored in the memory of the target unmanned vehicle, if a loop is detected, obtaining the loop key frames;

[0079] S420, according to the preset time period corresponding to the loop key frame, obtaining the key frame in the hard disk of the target unmanned vehicle corresponding to the preset time period;

[0080] S430, obtaining an optimal loop based on the loop key frame and the key frame corresponding to the preset time period;

[0081] S440: Optimize the poses and map points of the key frames in the optimal loop to obtain the real-time map.

[0082] First, loop closure detection is performed on the keyframes stored in the target unmanned vehicle's memory. The reason for loop closure detection is as follows: the initial values ​​of the feature points and map are provided in the front, and the back is responsible for optimizing all of this data. However, if only keyframes at adjacent times are considered, the errors generated previously will inevitably accumulate to the next moment, causing cumulative errors in the real-time map and unreliable long-term estimation results. However, since the camera passes through the same place and collects similar data, the key to loop closure detection is how to effectively detect that the camera passes through the same place. If this can be successfully detected, it can provide more valid data for the back-end pose graph, enabling better estimation, especially a globally consistent estimate. Therefore, the loop closure module can provide constraints that are further in the future than just some moments in adjacent frames. Therefore, loop closure detection is equivalent to adding an extra spring to the image, improving system stability.

[0083] Through loop closure detection, loop keyframes are obtained. Then, based on the time corresponding to the loop keyframes, keyframes stored on the target unmanned vehicle's hard drive are found. Combining these two, the optimal loop is obtained. The pose and map points of the keyframes in the optimal loop are optimized to obtain a real-time map. The real-time map refers to the current real-time map of the target unmanned vehicle when it is operating, and any temporary obstacles on the target unmanned vehicle's operating path can be seen.

[0084] It should be noted that in addition to the map points generated by the initial two keyframes, ORB points connected by three keyframes will also generate map points. Map points that do not meet the observation conditions of three consecutive keyframes will be eliminated. The generated map points can participate in pose estimation, so that pose estimation is not limited to the reference keyframe and the previous frame. Pose estimation can be performed based on the local map, and can also participate in local pose optimization and loop pose optimization. The above-mentioned local pose optimization optimizes the keyframe just generated and its keyframes with a common view relationship, as well as all map points observed by these keyframes. Keeping the position of the map point unchanged, optimize the pose between keyframes with a common view relationship, use Gauss-Newton iteration to minimize the reprojection error, and obtain the optimal pose relationship.

[0085] Figure 5 This is a flow chart of generating a real-time map by a target unmanned vehicle in one embodiment of the present invention. Figure 5 As shown in the figure, if a loop is detected in the keyframes stored in memory, the pose and map points of the keyframes in the optimal loop can be optimized to eliminate accumulated errors. Traditional monocular camera-based map construction methods, due to the inability to directly obtain scale information, suffer from serious scale drift problems during large-scale simultaneous positioning and mapping due to the continuous accumulation of errors. This method has a loop detection function that can optimize and reduce accumulated errors, thereby improving the accuracy of map construction and further enhancing navigation accuracy.

[0086] Based on the above embodiment, preferably, Figure 6 This is a flowchart of obtaining key frames stored in the memory and hard disk of the target unmanned vehicle in an embodiment of the present invention, as shown in FIG. Figure 6 As shown, the key frames stored in the memory of the target unmanned vehicle and the key frames in the hard disk of the target unmanned vehicle are obtained by the following steps:

[0087] S610, selecting a plurality of key frames from the color image according to the feature point information;

[0088] S620, obtaining the loop possibility of each key frame based on the feature point overlap rate between two adjacent key frames;

[0089] S630, storing key frames whose loop possibility is not greater than the preset threshold in the hard disk of the target unmanned vehicle, and storing key frames whose loop possibility is greater than the preset threshold in the memory of the target unmanned vehicle.

[0090] According to the key frames obtained in the above steps, key frame information is obtained, and the loop possibility is judged according to the key frame information. The frames with low loop possibility are saved in the long-term memory module, that is, the hard disk; the frames with higher loopability are saved in the short-term memory module, that is, the memory. Here, the level of loopability is determined by comparing the loop possibility with the size of the preset threshold. When the loop possibility is greater than the preset threshold, it is considered that the loop possibility of the key frame is high, otherwise, it is considered that the loop possibility of the key frame is low. The size of the preset threshold can be determined according to the actual situation, and the embodiment of the present invention does not make specific restrictions on this. When a loop is detected in the short-term memory module, the key frames in the long-term memory module before and after the loop key frame in the short-term memory module are extracted, the optimal loop is selected, and loop optimization is performed. The key frame poses and map points in all loops are optimized as variables to eliminate cumulative errors.

[0091] In the embodiment of the present invention, a long-short storage strategy is used for map management. The loop probability is determined based on the similarity of feature points between adjacent keyframes. The storage location of keyframes is determined based on the loop probability. Keyframes with high loop probability are stored in memory, while keyframes with low loop probability are stored on the hard disk. This prevents the entire map from being stored in memory to cope with large-scale scenarios. It also provides navigation map generation and management, enabling map reuse. Furthermore, in a campus environment without a priori maps, it effectively completes large-scale depth camera-based positioning and mapping work, constructs a visual odometry, generates a color point cloud map, and uses a map management system for storage.

[0092] Based on the above embodiment, preferably, Figure 7 This is a flowchart of optimizing the poses and map points of key frames in the optimal loop to obtain a real-time map in an embodiment of the present invention. Figure 7 As shown, the step of optimizing the poses and map points of the key frames in the optimal loop to obtain the real-time map includes the following steps:

[0093] S710, processing the poses of the key frames in the optimal loop, the color images of the map points, and the depth images of the map points to generate a color point cloud image;

[0094] S720, segmenting the color point cloud image using a ground filter to obtain an octree map;

[0095] S730, projecting the octree map along the Z axis to obtain an occupancy grid map;

[0096] S740: Use the octree map or the occupancy grid map as the real-time map.

[0097] The embodiment of the present invention also processes the obtained key frame pose, color map, and depth map, splices them to generate a color point cloud map, performs voxel filtering to reduce redundant point clouds, and segments the ground point cloud through a ground filter to generate an octree map for 3D map navigation; the octree map is projected along the Z-axis to generate an occupancy grid map for 2D map navigation.

[0098] The embodiment of the present invention performs path planning based on an octree or occupancy grid map, includes optional global path planning and local path planning, can cruise along a fixed route, and uses a combination of real-time maps and prior maps to complete obstacle avoidance navigation of the unmanned logistics vehicle.

[0099] Based on the above embodiment, preferably, Figure 8 FIG. 1 is a flow chart for obtaining the best path in an embodiment of the present invention. Figure 8As shown, the path planning is performed based on the current position and the target position to obtain the best path, including the following steps:

[0100] S810, performing global path planning based on the current position and the target position to obtain a global path;

[0101] S820, moving the target unmanned vehicle according to the global path to obtain the actual position that the target unmanned vehicle finally reaches;

[0102] S830: If the actual position is not the target position, perform local path planning based on the actual position and the target position to obtain a local path;

[0103] S840, moving the target unmanned vehicle according to the actual position, the target position and the local path, and repeating the above steps until the actual position of the target unmanned vehicle is the target position.

[0104] Specifically, when planning the path of the target unmanned vehicle, global path planning is performed first to obtain the global path, and then the target unmanned vehicle is moved according to the global path. If the actual position of the target unmanned vehicle is not the target position, it means that there is an obstacle on the global path, and the preset prior map cannot know the specific information of the obstacle. In this case, it is necessary to combine the real-time map for local path planning, and continuously adjust the actual position that the target unmanned vehicle finally reaches after path planning until it finally reaches the target position.

[0105] Based on the above embodiment, preferably, the navigation method further includes:

[0106] According to the color map, the YOLO v5 algorithm is used to detect the target and obtain roadblock information;

[0107] Determining the size and position of the roadblock information according to the roadblock information and the depth map;

[0108] The size and location of the roadblock information are added to the preset prior map.

[0109] In the embodiment of the present invention, the color image captured by the depth camera is used to detect targets using the YOLO v5 algorithm to obtain roadblock information. In the embodiment of the present invention, roadblocks include vehicles, pedestrians, road signs, traffic lights, lane markings, traffic lights, etc., and the size and position of the roadblocks in the preset prior map are determined in combination with the depth image of the depth camera. The corresponding point cloud information is segmented, the semantic information is marked, and it is added to the real-time map for local path planning. If there is no predetermined global path information, the unmanned logistics vehicle can be repositioned in the global map based on the current depth sensor information, so that the real-time map can be updated to the global map, and the new map is used for global path planning.

[0110] Based on the above-mentioned global path planning and local path planning, the unmanned vehicle can achieve autonomous obstacle avoidance and navigation to the target point. The semantic information such as vehicles, pedestrians, road signs, traffic lights, lane lines, and signal lights in the local map can be used for speed control and motion decision-making of the unmanned logistics vehicle during its movement to the target point, realizing operations such as lane line keeping, maintaining speed limits, and waiting for traffic lights.

[0111] Traditional laser SLAM solutions for simultaneous mapping and navigation are difficult to detect in 3D because laser point clouds lack real-world texture information. This makes it difficult to quickly and effectively identify navigation information such as vehicles, pedestrians, lane markings, road signs, and traffic lights. In this embodiment, a depth camera's color image is used to identify lane markings, road signs, vehicles, and pedestrians, acquiring their position and traffic information in real time for obstacle avoidance and navigation.

[0112] Figure 9 A module diagram of a navigation device based on a depth camera provided by an embodiment of the present invention is shown as follows: Figure 9 As shown, the apparatus includes a position module 910, a positioning module 920, a path module 930, and a movement module 940, wherein:

[0113] The location module 910 is used to obtain the target location according to the target order information and the preset priority rules;

[0114] The positioning module 920 is used to obtain the current position of the target unmanned vehicle based on the target position, a preset priori map corresponding to the target park, and a real-time map corresponding to the target park, wherein the preset priori map is obtained based on the color image and depth map output by the depth camera, and the real-time map is obtained by processing the preset priori map;

[0115] The path module 930 is used to perform path planning based on the current location and the target location to obtain the optimal path;

[0116] The moving module 940 is used to move the target unmanned vehicle to the target location according to the optimal path.

[0117] This embodiment is a device embodiment corresponding to the above method, and its specific implementation process is the same as that of the above method embodiment. For details, please refer to the above method embodiment, and this device embodiment will not repeat it again.

[0118] Based on the above embodiment, preferably, the positioning module includes a camera unit, a feature unit and a priori map unit, wherein:

[0119] The camera unit is configured to obtain the color image and the depth image according to the depth camera;

[0120] The feature unit is used to perform feature matching based on the color image to obtain feature point information;

[0121] The prior map unit is used to perform pose calculation through a PnP algorithm based on the feature point information and the depth map, and further obtain the preset prior map.

[0122] Based on the above embodiment, preferably, the positioning module further includes a key frame unit, a loop unit, a memory unit and a hard disk unit, wherein:

[0123] The key frame unit is used to select a plurality of key frames from the color image according to the feature point information;

[0124] The loop closure unit is used to obtain the loop closure possibility of each key frame according to the overlap rate of feature points between two adjacent key frames;

[0125] The memory unit is used to store key frames with a loop probability greater than a preset threshold in the memory of the target unmanned vehicle;

[0126] The hard disk unit is used to store key frames whose loop probability is not greater than the preset threshold in the hard disk of the target unmanned vehicle.

[0127] Based on the above embodiment, preferably, the positioning module further includes a detection unit, a selection unit, an extraction unit and a real-time map unit, wherein:

[0128] The detection unit is configured to obtain a loop key frame if a loop is detected based on the key frame stored in the memory;

[0129] The selection unit is configured to obtain, according to the preset time period corresponding to the loop key frame, the key frame corresponding to the preset time period in the hard disk;

[0130] The extraction unit is used to obtain the optimal loop based on the loop key frame and the key frame corresponding to the preset time period;

[0131] The real-time map unit is used to optimize the poses and map points of the key frames in the optimal loop to obtain the real-time map.

[0132] Based on the above embodiment, preferably, the real-time map unit includes a color subunit, an octree subunit, a grid subunit and an output unit, wherein:

[0133] The color subunit is used to process the pose of the key frame in the optimal loop, the color map of the map point and the depth map of the map point, and splice them to generate a color point cloud map;

[0134] The octree subunit is used to segment the color point cloud image through a ground filter to obtain an octree map;

[0135] The grid subunit is used to project the octree map along the Z axis to obtain an occupied grid map;

[0136] The output unit is configured to use the octree map or the occupancy grid map as the real-time map.

[0137] Based on the above embodiment, preferably, the path module includes a global unit, an adjustment unit, a movement unit and a circulation unit, wherein:

[0138] The global unit is used to perform global path planning based on the current position and the target position to obtain a global path;

[0139] The adjustment unit is used to move the target unmanned vehicle according to the global path to obtain the actual position finally reached by the target unmanned vehicle;

[0140] The mobile unit is configured to perform local path planning based on the actual position and the target position to obtain a local path if the actual position is not the target position;

[0141] The circulation unit is used to move the target unmanned vehicle according to the actual position, the target position and the local path, and repeat the above steps until the actual position of the target unmanned vehicle is the target position.

[0142] Based on the above embodiment, preferably, it further includes a roadblock unit, an information unit and an additional unit, wherein:

[0143] The roadblock unit is used to detect the target according to the color image using the YOLO v5 algorithm to obtain roadblock information;

[0144] The information unit is used to determine the size and position of the roadblock information based on the roadblock information and the depth map;

[0145] The adding unit is used to add the size and position of the roadblock information to the preset prior map.

[0146] Each module in the above-mentioned depth camera-based navigation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0147] An embodiment of the present invention provides an unmanned vehicle, which includes a navigation device based on a depth camera provided in the above embodiment. The unmanned vehicle is easy to initialize, can directly obtain color images and depth maps, can construct dense point cloud maps, can realize accurate positioning and mapping of unmanned logistics vehicles in unknown environments, and can realize autonomous navigation; based on long-short storage strategies, it realizes map preservation, loop detection and optimization at a large scale, and provides a point cloud map processing method to facilitate real-time mapping and map reuse; combined with target detection technology, it can identify navigation information such as vehicles, pedestrians, lane lines, road signs, traffic lights, etc., and use a path planning scheme with an optional preset global path, combined with local path optimization, to realize speed control, obstacle avoidance, navigation and other functions of the unmanned logistics vehicle.

[0148] In one embodiment, a computer storage medium is provided, which stores a computer program. When executed by a processor, the computer program implements the steps of the depth camera-based navigation method described in the above embodiment. Alternatively, when executed by a processor, the computer program implements the functions of the modules / units of the depth camera-based navigation device described in the above embodiment.

[0149] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0150] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0151] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A navigation method based on a depth camera, characterized in that: include: Obtain the target location based on the target order information and preset priority rules; According to the preset a priori map corresponding to the target position and target park and the real-time map corresponding to the target park, the current position of the target unmanned vehicle is obtained, wherein the preset a priori map is obtained based on the color map and depth map output by the depth camera, and is obtained by the following steps: according to the depth camera, a color map and a depth map are obtained, and according to the color map, feature matching is performed to obtain feature point information; according to the feature point information and the depth map, a pose is calculated by the PnP algorithm, and the preset a priori map is further obtained; the real-time map is obtained by processing the preset a priori map, and is obtained by the following steps: according to the key frames stored in the memory of the target unmanned vehicle, if a loop is detected, a loop key frame is obtained, according to the preset time period corresponding to the loop key frame, the key frame in the hard disk of the target unmanned vehicle corresponding to the preset time period is obtained, according to the loop key frame and the key frame corresponding to the preset time period, the optimal loop is obtained, the pose and map points of the key frame in the optimal loop are optimized to obtain the real-time map; Performing path planning based on the current position and the target position to obtain an optimal path; According to the optimal path, the target unmanned vehicle is moved to the target position.

2. The depth camera-based navigation method according to claim 1, characterized in that: The key frames stored in the memory of the target unmanned vehicle and the key frames in the hard disk of the target unmanned vehicle are obtained by the following steps: Selecting a plurality of key frames from the color image based on feature point information, wherein the feature points are obtained based on the color image and the depth image of the depth camera; According to the overlap rate of feature points between two adjacent key frames, the loop possibility of each key frame is obtained; The key frames whose loop possibility is not greater than the preset threshold are stored in the hard disk of the target unmanned vehicle, and the key frames whose loop possibility is greater than the preset threshold are stored in the memory of the target unmanned vehicle.

3. The depth camera-based navigation method according to claim 1, wherein: Optimizing the poses and map points of the key frames in the optimal loop to obtain the real-time map includes the following steps: Processing the pose of the key frame in the optimal loop, the color map of the map point, and the depth map of the map point to generate a color point cloud image; Segmenting the color point cloud image through a ground filter to obtain an octree map; Projecting the octree map along the Z axis to obtain an occupancy grid map; The octree map or the occupancy grid map is used as the real-time map.

4. The depth camera-based navigation method according to claim 1, wherein: The step of performing path planning based on the current position and the target position to obtain the optimal path includes the following steps: Performing global path planning based on the current position and the target position to obtain a global path; Moving the target unmanned vehicle according to the global path to obtain the actual position reached by the target unmanned vehicle; If the actual position is not the target position, performing local path planning based on the actual position and the target position to obtain a local path; According to the actual position, the target position and the local path, the target unmanned vehicle is moved, and the above steps are repeated until the actual position of the target unmanned vehicle is the target position.

5. The depth camera-based navigation method according to any one of claims 1 to 4, characterized in that: The navigation method further includes: According to the color map, the YOLO v5 algorithm is used to detect the target and obtain roadblock information; Determining the size and position of the roadblock information according to the roadblock information and the depth map; The size and location of the roadblock information are added to the preset prior map.

6. A navigation device based on a depth camera, characterized in that: include: The location module is used to obtain the target location based on the target order information and preset priority rules; a positioning module, configured to obtain the current position of the target unmanned vehicle based on the target position, a preset a priori map corresponding to the target park, and a real-time map corresponding to the target park, wherein the preset a priori map is obtained based on the color image and depth map output by the depth camera, and the real-time map is obtained by processing the preset a priori map; The positioning module includes a camera unit, a feature unit and a priori map unit, wherein: The camera unit is configured to obtain the color image and the depth image according to the depth camera; The feature unit is used to perform feature matching based on the color image to obtain feature point information; The prior map unit is used to perform pose calculation using a PnP algorithm based on the feature point information and the depth map, and further obtain the preset prior map; The positioning module also includes a detection unit, a selection unit, an extraction unit and a real-time map unit, wherein: The detection unit is used to obtain a loop key frame if a loop is detected based on the key frame stored in the memory; The selection unit is configured to obtain a key frame corresponding to a preset time period in the hard disk according to the preset time period corresponding to the loop key frame; The extraction unit is used to obtain the optimal loop based on the loop key frame and the key frame corresponding to the preset time period; The real-time map unit is used to optimize the pose and map points of the key frames in the optimal loop to obtain the real-time map; A path module is used to plan a path based on the current location and the target location to obtain an optimal path; A moving module is used to move the target unmanned vehicle to the target position according to the optimal path.

7. An unmanned vehicle, characterized in that: Includes the depth camera-based navigation device as described in claim 6.

8. A computer storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the depth camera-based navigation method according to any one of claims 1 to 5 are implemented.

Images