Navigation method, device, medium, program product and system for mobile equipment

Abstract

The invention provides a navigation method and device of mobile equipment, a medium, a program product and a system. According to the method, the current environment three-dimensional information, the current environment image and the current global positioning data provide different types of information, more stable pose estimation and map construction can be realized by combining the current environment three-dimensional information, the current environment image and the current global positioning data, rich environment characteristics are provided by using an enhanced SLAM technology, and the laser radar provides accurate geometric information, so that the accuracy of the map construction is improved. The map constructed after fusion has richness of visual features and accuracy of geometric structures, so that compared with an existing scheme, the navigation error is reduced, the navigation safety is improved, and the problem that mobile equipment in the existing scheme excessively depends on a single navigation technology, and the navigation efficiency is improved is solved. The problems that the navigation error of the mobile equipment is large and the navigation safety of the mobile equipment is low due to the fact that a large error exists between a single navigation technology and an actual working condition in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of navigation technology, and in particular, to a navigation method, device, medium, program product and system for a mobile device. Background Art

[0002] Automatic guided vehicles (AGVs) play an increasingly important role in modern industrial and logistics applications. The navigation and positioning technology of AGVs is key to ensuring their efficient and safe operation in complex environments. Many current AGV systems rely on a single navigation technology, such as LiDAR or visual recognition, which may lead to positioning errors or navigation failures in dynamic environments.

[0003] The movable device of the existing solution is too dependent on a single navigation technology. Since there is a large error between the single navigation technology and the actual working conditions, the navigation error of the movable device is large, which in turn leads to low navigation safety of the movable device. Summary of the invention

[0004] The main purpose of the present application is to provide a navigation method, device, medium, program product and system for a movable device, so as to at least solve the problem that the movable device of the existing solution is too dependent on a single navigation technology. Since there is a large error between the single navigation technology and the actual working conditions, the navigation error of the movable device is large, which in turn leads to low navigation safety of the movable device.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a navigation method for a movable device is provided, the method comprising: obtaining at least current environment three-dimensional information, current environment image and current global positioning data, the current environment three-dimensional information being the three-dimensional information of the current environment where the movable device is located, the current environment image being the image of the current environment where the movable device is located, and the current global positioning data being the global positioning data of the current environment where the movable device is located; at least fusing the current environment three-dimensional information and the current environment image to obtain environment fusion data, and determining that the current environment data includes the environment fusion data and the current global positioning data; using enhanced SLAM technology and nonlinear least squares optimization algorithm to construct and optimize a map based on the current environment data to obtain current map information so as to minimize the errors of all edges on the map; and navigating the movable device based on the current map information.

[0006] Optionally, at least the current environment three-dimensional information and the current environment image are data-fused to obtain environment fusion data, and it is determined that the current environment data includes the environment fusion data and the current global positioning data, including: when the movable device is in an outdoor environment, the current environment three-dimensional information and the current environment image are data-fused to obtain the environment fusion data, and it is determined that the current environment data includes the environment fusion data and the current global positioning data; when the movable device is in an indoor environment, the current environment three-dimensional information and the current environment image are data-fused to obtain the environment fusion data; the current global positioning data and the current IMU data are data-fused to obtain global fusion data; it is determined that the current environment data includes the environment fusion data and the global fusion data; or the current environment three-dimensional information, the current environment image and the current IMU data are data-fused to obtain the environment fusion data; the current global positioning data and the current IMU data are data-fused to obtain global fusion data; it is determined that the current environment data includes the environment fusion data and the global fusion data, and the current IMU data is the current acceleration and angular velocity of the movable device collected by the IMU.

[0007] Optionally, the three-dimensional information of the current environment and the current environment image are data-fused to obtain the environment fusion data, including: using a Kalman filter algorithm or a particle algorithm to data-fuse the three-dimensional information of the current environment and the current environment image to obtain the environment fusion data; the current global positioning data and the current IMU data are data-fused to obtain global fusion data, including: using a Kalman filter algorithm to data-fuse the current global positioning data and the current IMU data to obtain the global fusion data.

[0008] Optionally, the three-dimensional information of the current environment, the current environment image and the current IMU data are data-fused to obtain the environment fusion data, including: using an extended Kalman filter to data-fuse the three-dimensional information of the current environment, the current environment image and the current IMU data to obtain the environment fusion data; the current global positioning data and the current IMU data are data-fused to obtain global fusion data, including: using a Kalman filter algorithm to data-fuse the current global positioning data and the current IMU data to obtain the global fusion data.

[0009] Optionally, in the process of navigating the movable device based on the current map information, the method also includes: searching for the target node based on f(n)=g(n)+h(n), wherein g(n) is the actual cost from the starting node to node n, h(n) is the estimated cost from node n to the target node, and f(n) is the total cost of node n; with the goal of minimizing the total cost of node n, searching for the target node until the target node is determined or all reachable nodes are searched.

[0010] Optionally, in the process of navigating the movable device based on the current map information, the method further includes: using a BWA algorithm to perform obstacle avoidance control on the movable device.

[0011] According to another aspect of the present application, a navigation device for a mobile device is provided, the device comprising:

[0012] an acquisition unit, configured to acquire at least current environment three-dimensional information, a current environment image, and current global positioning data, wherein the current environment three-dimensional information is three-dimensional information of a current environment where the movable device is located, the current environment image is an image of the current environment where the movable device is located, and the current global positioning data is global positioning data of the current environment where the movable device is located;

[0013] A first processing unit is used to perform data fusion on at least the current environment three-dimensional information and the current environment image to obtain environment fusion data, and determine that the current environment data includes the environment fusion data and the current global positioning data;

[0014] A second processing unit is used to construct and optimize a map based on the current environment data using enhanced SLAM technology and a nonlinear least squares optimization algorithm to obtain current map information so as to minimize errors of all edges on the map;

[0015] The third processing unit is used to navigate the movable device based on the current map information.

[0016] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0017] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the methods are implemented.

[0018] According to another aspect of the present application, a navigation system for a mobile device is provided, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the methods described.

[0019] By applying the technical solution of the present application, the current environmental three-dimensional information, the current environmental image and the current global positioning data provide different types of information. Combining the three can achieve more stable pose estimation and map construction. By using enhanced SLAM technology, rich environmental features are provided. The lidar provides accurate geometric information. The map constructed after fusion has both the richness of visual features and the accuracy of geometric structure. Compared with the existing solutions, the navigation error of the present application is reduced and the navigation safety is improved, thereby solving the problem that the movable device of the existing solution is too dependent on a single navigation technology. Since there is a large error between the single navigation technology and the actual working conditions, the navigation error of the movable device is large, which leads to low navigation safety of the movable device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 A schematic diagram of a flow chart of a navigation method for a mobile device provided according to an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram showing a flow chart of another navigation method for a mobile device provided according to an embodiment of the present application;

[0023] Figure 3 A structural block diagram of a navigation device for a mobile device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0028] The Gauss-Newton method is an iterative algorithm for nonlinear least squares problems that combines the advantages of the Gauss method and the Newton method. The algorithm works by converting the nonlinear least squares problem into a linear least squares problem and solving the optimal parameters by iteratively updating the parameters to minimize the residual sum of squares.

[0029] The Levenberg-Marquardt algorithm is an iterative algorithm for solving nonlinear least squares problems. The algorithm combines the advantages of the Gauss-Newton method and the gradient descent method, and ensures the stability and convergence of the algorithm by adding a regularization term to the Newton method. The Levenberg-Marquardt algorithm dynamically adjusts parameters in each iteration to accelerate the convergence process while ensuring convergence.

[0030] Extended Kalman Filter (EKF): EKF can fuse IMU, LiDAR and vision data in time series. IMU provides high-frequency pose prediction, and the observation data provided by vision and LiDAR is used to correct the prediction error. EKF gradually fuses multi-sensor data and generates more accurate pose estimates through a prediction-update cycle.

[0031] AGV is the abbreviation of Automated Guided Vehicle, also known as automatic transport vehicle, which is an unmanned vehicle that can operate and transport goods autonomously. AGV is usually used in places such as warehouses, factories and logistics centers. It can automatically drive according to preset routes and tasks, improve logistics transportation efficiency and reduce labor costs. AGV can be customized according to different needs and environments, and there are various types and specifications of AGV to choose from.

[0032] The A* algorithm is a heuristic search algorithm widely used in path planning and graph search. It uses a heuristic function to evaluate the priority of each node during the search process to find the best path as quickly as possible. The A* algorithm combines the advantages of the Dijkstra algorithm and the greedy best-first search algorithm, and has high search efficiency and good path planning capabilities.

[0033] The BWA algorithm is an algorithm used for DNA sequencing data alignment, and its full name is Burrows-Wheeler Aligner. The BWA algorithm is mainly used to align sequencing data with the reference genome to find the location of reads in the sequencing data in the genome. The BWA algorithm uses data structures such as Burrows-Wheeler transformation and suffix arrays to achieve an efficient alignment process, and is an algorithm widely used in genome alignment.

[0034] The Kalman filter algorithm is an algorithm used to estimate the state of a dynamic system. By taking a weighted average of the system's measured values ​​and model prediction values, the accuracy and stability of state estimation can be improved.

[0035] The particle filter algorithm is a filtering algorithm based on the Monte Carlo method. It uses a set of random particles to represent the state space of the system and approximates the real state distribution of the system by continuously updating and resampling these particles. The particle filter algorithm is suitable for nonlinear and non-Gaussian systems, and can handle non-Gaussian noise of nonlinear systems.

[0036] The Extended Kalman Filter (EKF) is an extended version of the Kalman filter for nonlinear systems. The traditional Kalman filter is suitable for linear systems, but for nonlinear systems, its model cannot accurately describe the dynamic characteristics of the system. The Extended Kalman Filter linearizes the system so that nonlinear systems can also use the Kalman filter for state estimation.

[0037] As introduced in the background technology, automated guided vehicles (AGVs) play an increasingly important role in modern industrial and logistics applications. The navigation and positioning technology of AGVs is the key to ensuring their efficient and safe operation in complex environments. Currently, many AGV systems rely on a single navigation technology, such as lidar or visual recognition, which may lead to positioning errors or navigation failures in dynamic environments. In order to solve the problem that the movable devices of existing solutions are too dependent on a single navigation technology, and there is a large error between the single navigation technology and the actual working conditions, which makes the navigation error of the movable device large, and thus leads to the problem of low navigation safety of the movable device, the embodiments of the present application provide a navigation method, device, medium, program product and system for a movable device.

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] In this embodiment, a navigation method for a mobile device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 1 FIG. 1 is a flow chart of a navigation method for a mobile device provided according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0041] Step S101, at least obtaining current environment three-dimensional information, current environment image and current global positioning data, wherein the current environment three-dimensional information is the three-dimensional information of the current environment where the movable device is located, the current environment image is the image of the current environment where the movable device is located, and the current global positioning data is the global positioning data of the current environment where the movable device is located;

[0042] The movable device may be an AGV, ie an automatic guided vehicle.

[0043] Specifically, LiDAR: used for high-precision distance measurement, providing three-dimensional information of the environment, and collecting the above-mentioned three-dimensional information of the current environment. Vision system: includes a forward-facing camera and a panoramic camera, used to capture image information of the environment, support feature recognition and vision-assisted positioning, and collect current environment images. Inertial Measurement Unit (IMU): used to detect the acceleration and angular velocity of the AGV, assist in dynamic positioning and posture adjustment, and collect current IMU data. Global Positioning System (GPS): used for global positioning in outdoor environments, supplement data from other sensors, and collect current global positioning data.

[0044] Step S102, at least fusing the above-mentioned current environment three-dimensional information and the above-mentioned current environment image to obtain environment fusion data, and determining that the current environment data includes the above-mentioned environment fusion data and the above-mentioned current global positioning data;

[0045] Specifically, when the movable device is in an outdoor environment, the current environment three-dimensional information and the current environment image are data-fused to obtain the environment fusion data, and the current environment data is determined to include the environment fusion data and the current global positioning data;

[0046] When the above-mentioned mobile device is in an indoor environment, there are two implementation methods:

[0047] The first implementation method:

[0048] The above-mentioned current environment three-dimensional information and the above-mentioned current environment image are subjected to data fusion to obtain the above-mentioned environment fusion data; the above-mentioned current global positioning data and the current IMU data are subjected to data fusion to obtain global fusion data; and it is determined that the above-mentioned current environment data includes the above-mentioned environment fusion data and the above-mentioned global fusion data;

[0049] Specifically, AGVs carry out logistics transportation in outdoor open areas. GPS provides global positioning information, and laser radar and vision systems provide local environmental information. The data fusion algorithm combines the data from GPS, laser radar and vision systems to provide accurate location information. Adaptive positioning correction technology is used to correct positioning errors, and positioning accuracy is improved by matching environmental features.

[0050] Adaptive positioning correction technology corrects positioning errors:

[0051] Optimization algorithm: Use a nonlinear least squares optimization algorithm (such as Gauss-Newton or Levenberg-Marquardt) to optimize the graph to minimize the error of all edges and obtain more accurate node positions. Error minimization: The goal is to minimize the error of all edges (constraints) by adjusting node positions to generate a globally consistent map.

[0052] In one embodiment of the present application,

[0053] The three-dimensional information of the current environment and the image of the current environment are subjected to data fusion to obtain the fusion data of the environment, including: using a Kalman filter algorithm or a particle algorithm to fuse the three-dimensional information of the current environment and the image of the current environment to obtain the fusion data of the environment;

[0054] The current global positioning data and the current IMU data are subjected to data fusion to obtain global fusion data, including: the current global positioning data and the current IMU data are subjected to data fusion using a Kalman filter algorithm to obtain the global fusion data.

[0055] The two fused data facilitate subsequent map generation and precise positioning.

[0056] The second implementation method:

[0057] The above-mentioned current environment three-dimensional information, the above-mentioned current environment image and the above-mentioned current IMU data are data-fused to obtain the above-mentioned environment fusion data; the above-mentioned current global positioning data and the above-mentioned current IMU data are data-fused to obtain global fusion data; it is determined that the above-mentioned current environment data includes the above-mentioned environment fusion data and the above-mentioned global fusion data, and the above-mentioned current IMU data is the current acceleration and angular velocity of the above-mentioned movable device collected by the IMU.

[0058] Specifically, in the preprocessing stage, time synchronization may also be involved to ensure that the data from different sensors are aligned at the same time. For example, interpolation methods are used to effectively fuse the data from lidar, visual cameras, and IMU at the same timestamp.

[0059] The data update frequency of IMU is usually much higher than that of vision and lidar data, and its motion information can provide relatively accurate attitude estimation in a short time. Therefore, IMU is often used to supplement the shortcomings of vision and lidar in attitude estimation. There are two main ways to fuse IMU data:

[0060] Tightly coupled fusion: Tightly coupled fusion optimizes the IMU motion state directly with the visual / laser pose estimation. This method can use the IMU motion equation as a constraint in a nonlinear optimization framework and jointly optimize it with the visual or laser observations.

[0061] For example, in visual SLAM, the IMU's attitude information can be combined with the displacement of feature points in the image, and the data of the two can be fused through a state estimator (such as an extended Kalman filter or a nonlinear optimizer). The IMU data provides short-term predictions of the pose, and the visual information provides long-term global corrections by matching feature points.

[0062] Loosely coupled fusion: Loosely coupled fusion methods process IMU data and vision / lidar data separately and then fuse the data at a higher level. For example, IMU can be used to estimate incremental changes in pose, while vision or lidar is used to globally optimize and correct pose. This method is relatively simple and suitable for systems with limited computing resources.

[0063] Fusion of visual data and lidar data:

[0064] Visual data and LiDAR data provide different types of information, one is the image features of the environment, and the other is geometric distance information. Combining the two can achieve more stable pose estimation and map construction. Feature fusion: Key points (such as ORB, SIFT, etc.) can be extracted from the visual image, and then these visual feature points can be matched with the point cloud features (such as corners, planes, etc.) generated by the LiDAR. By aligning these features, the accuracy of pose estimation can be improved. For example, visual feature points can be used to make up for the perception blind spots of LiDAR in some scenarios (such as open areas without obvious obstacles or mirror-reflecting objects). Conversely, the geometric information of LiDAR can help correct inaccurate pose estimates caused by problems such as illumination changes and sparse textures in the visual system. Fusion of depth information: If an RGB-D camera or a binocular camera is used, the image information can directly provide depth data. At this time, the depth data can be aligned with the point cloud of the LiDAR, and the coordinate systems of the two data can be unified through ICP (iterative closest point) or other matching algorithms. Fusion of motion estimation: Both lidar and vision sensors can provide motion estimation of the robot in the environment. Lidar estimates the pose change by aligning point clouds, and vision estimates the pose by matching feature points. Combining these two pose estimation methods together can improve the accuracy of pose estimation through filters (such as Kalman filters and particle filters) and reduce the error caused by a single sensor.

[0065] In one embodiment of the present application,

[0066] The three-dimensional information of the current environment, the current environment image and the current IMU data are subjected to data fusion to obtain the fused environment data, including: using an extended Kalman filter to fuse the three-dimensional information of the current environment, the current environment image and the current IMU data to obtain the fused environment data;

[0067] Specifically, IMU compensates for the shortcomings of vision and lidar in fast-moving or featureless scenes, while vision and lidar correct the long-term accumulated drift of IMU through feature matching. The complementarity of multiple sensors enables the system to maintain good performance in the event of sensor failure or data loss. For example, when the lighting changes drastically, lidar can provide stable distance information; when the lidar scan is insufficient, vision can supplement the environmental information.

[0068] The current global positioning data and the current IMU data are subjected to data fusion to obtain global fusion data, including: the current global positioning data and the current IMU data are subjected to data fusion using a Kalman filter algorithm to obtain the global fusion data.

[0069] Specifically, the fusion of GPS data and IMU data can complement each other and improve positioning accuracy. IMU data can make up for the signal blocking problem of GPS in environments such as high-rise buildings in cities, thereby improving positioning accuracy. The fusion of GPS and IMU data can improve the stability of positioning, because IMU data can provide real-time acceleration and angular velocity information, thereby reducing the error and drift of GPS data. By fusing GPS and IMU data, more accurate navigation and positioning functions can be achieved, thereby improving the user's navigation experience.

[0070] Step S103, using enhanced SLAM technology and nonlinear least squares optimization algorithm, constructing and optimizing a map based on the above current environment data to obtain current map information to minimize the errors of all edges on the map;

[0071] Specifically, by using enhanced SLAM technology, rich environmental features are provided, and lidar provides precise geometric information. The map constructed after fusion has both the richness of visual features and the accuracy of geometric structure.

[0072] Enhanced SLAM technology is a technology that combines traditional SLAM technology with other sensors and algorithms to improve the positioning and mapping accuracy, robustness and real-time performance of the SLAM system. Enhanced SLAM technology usually includes the following improvements:

[0073] Multi-sensor fusion: Improve the positioning and mapping accuracy of the SLAM system by fusing multiple sensor data (such as inertial measurement unit, camera, lidar, etc.).

[0074] Deep learning: Use deep learning technology to optimize the SLAM system and improve the accuracy and robustness of positioning and mapping.

[0075] Real-time optimization: Improve the real-time performance of the SLAM system and reduce latency by optimizing algorithms and hardware.

[0076] Improved robustness: By improving algorithms and sensor fusion strategies, the SLAM system's ability to adapt to various environments and scenarios can be improved.

[0077] In the process of using enhanced SLAM technology and nonlinear least squares optimization algorithm to construct and optimize the map based on the above current environmental data to obtain the current map information, enhanced SLAM and graph optimization technology can also be considered: combining graph optimization SLAM and multi-threaded processing technology to improve the positioning accuracy and real-time performance of AGV.

[0078] Graph optimization is an important step to ensure that the system can provide accurate positioning and high-quality maps during long-term operation. The goal of this process is to minimize errors by optimizing the nodes (representing the robot's pose or observed landmarks) and edges (representing relative position information or constraints) in the graph. Errors mainly come from sensor noise, inaccurate motion models, and accumulated drift during long-term operation. Through graph optimization, the system's estimated pose can be adjusted to make the robot's trajectory and observations more consistent with the real world.

[0079] The specific steps of optimization are:

[0080] Initialize the pose estimation of the node; calculate the errors on all edges; use gradient descent or other nonlinear least squares algorithms to gradually adjust the pose of the node to reduce the total error; iterate and update until the error converges to the minimum threshold.

[0081] Step S104: Navigate the movable device based on the current map information.

[0082] In the above steps, the current environment three-dimensional information, the current environment image and the current global positioning data provide different types of information. Combining the three can achieve more stable pose estimation and map construction. By using enhanced SLAM technology, rich environmental features are provided. The lidar provides accurate geometric information. The map constructed after fusion has both the richness of visual features and the accuracy of geometric structure. Compared with the existing solutions, the navigation error of this application is reduced and the navigation safety is improved, thereby solving the problem that the movable device of the existing solution is too dependent on a single navigation technology. Since there is a large error between the single navigation technology and the actual working conditions, the navigation error of the movable device is large, which leads to the problem of low navigation safety of the movable device.

[0083] In one embodiment of the present application, in the process of navigating the above-mentioned movable device based on the above-mentioned current map information, the above-mentioned method further includes:

[0084] Search for the target node based on f(n)=g(n)+h(n), where g(n) is the actual cost from the start node to node n, h(n) is the estimated cost from node n to the target node, and f(n) is the total cost of node n;

[0085] The goal is to minimize the total cost of node n, and the target node is searched until the target node is determined or all reachable nodes are searched.

[0086] The advantages of the A* algorithm are that it can find the shortest path and can efficiently search for paths when the heuristic evaluation function is reasonable. However, the performance of the A* algorithm is affected by the heuristic evaluation function, and different heuristic evaluation functions may lead to different search results. In addition, the A* algorithm needs to store and update a large amount of node information, which requires high memory and computing resources.

[0087] In one embodiment of the present application, in the process of navigating the above-mentioned movable device based on the above-mentioned current map information, the above-mentioned method further includes:

[0088] The BWA algorithm is used to perform obstacle avoidance control on the above movable device. The robot's path planning is adjusted in real time according to the dynamic changes of obstacles in the environment to avoid collision with obstacles. The BWA algorithm is usually used in fields such as autonomous driving and robot navigation, and can effectively improve the safety and stability of the system.

[0089] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the navigation method of the mobile device of the present application will be described in detail below in combination with specific embodiments.

[0090] This embodiment relates to a specific navigation method for a mobile device, such as Figure 2 As shown, including:

[0091] Obtain the current environment three-dimensional information, current environment image, current global positioning data and current IMU data. The current environment three-dimensional information is the three-dimensional information of the current environment where the mobile device is located. The current environment image is the image of the current environment where the mobile device is located. The current global positioning data is the global positioning data of the current environment where the mobile device is located. The current IMU data is the current acceleration and angular velocity of the mobile device collected by the IMU.

[0092] When the mobile device is in an outdoor environment, the current environment three-dimensional information and the current environment image are data-fused to obtain environment fusion data, and the current environment data is determined to include the environment fusion data and the current global positioning data;

[0093] When the mobile device is in an indoor environment, an extended Kalman filter is used to fuse the current environment three-dimensional information, the current environment image and the current IMU data to obtain environment fusion data; a Kalman filter algorithm is used to fuse the current global positioning data and the current IMU data to obtain global fusion data; and it is determined that the current environment data includes environment fusion data and global fusion data;

[0094] Using enhanced SLAM technology and nonlinear least squares optimization algorithm, the map is constructed and optimized based on the current environmental data to obtain the current map information to minimize the error of all edges on the map;

[0095] Navigate the mobile device based on current map information.

[0096] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0097] The present application embodiment also provides a navigation device for a mobile device. It should be noted that the navigation device for a mobile device of the present application embodiment can be used to execute the navigation method for a mobile device provided by the present application embodiment. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been described will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0098] The following is an introduction to the navigation device of the mobile device provided in the embodiment of the present application.

[0099] Figure 3 is a structural block diagram of a navigation device for a mobile device provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0100] An acquisition unit 31 is used to acquire at least current environment three-dimensional information, a current environment image and current global positioning data, wherein the current environment three-dimensional information is the three-dimensional information of the current environment where the movable device is located, the current environment image is the image of the current environment where the movable device is located, and the current global positioning data is the global positioning data of the current environment where the movable device is located;

[0101] The first processing unit 32 is used to perform data fusion on at least the current environment three-dimensional information and the current environment image to obtain environment fusion data, and determine that the current environment data includes the environment fusion data and the current global positioning data;

[0102] The second processing unit 33 is used to construct and optimize the map based on the above current environment data by using enhanced SLAM technology and nonlinear least squares optimization algorithm to obtain current map information so as to minimize the errors of all edges on the map;

[0103] The third processing unit 34 is used to navigate the movable device based on the current map information.

[0104] In the above-mentioned device, the current environment three-dimensional information, the current environment image and the current global positioning data provide different types of information. Combining the three can achieve more stable pose estimation and map construction. By using enhanced SLAM technology, rich environmental features are provided. The lidar provides accurate geometric information. The map constructed after fusion has both the richness of visual features and the accuracy of geometric structure. Compared with the existing solutions, the navigation error of this application is reduced and the navigation safety is improved, thereby solving the problem that the movable device of the existing solution is too dependent on a single navigation technology. Since there is a large error between the single navigation technology and the actual working conditions, the navigation error of the movable device is large, which leads to the problem of low navigation safety of the movable device.

[0105] In one embodiment of the present application, the first processing unit includes a first processing module and a second processing module;

[0106] The first processing module is used for fusing the current environment three-dimensional information and the current environment image to obtain the environment fusion data when the movable device is in an outdoor environment, and determining that the current environment data includes the environment fusion data and the current global positioning data;

[0107] The second processing module is used for, when the mobile device is in an indoor environment, fusing the current environment three-dimensional information with the current environment image to obtain the environment fusion data; fusing the current global positioning data with the current IMU data to obtain global fusion data; and determining that the current environment data includes the environment fusion data and the global fusion data;

[0108] Alternatively, the above-mentioned current environment three-dimensional information, the above-mentioned current environment image and the above-mentioned current IMU data are data-fused to obtain the above-mentioned environment fusion data; the above-mentioned current global positioning data and the above-mentioned current IMU data are data-fused to obtain global fusion data; it is determined that the above-mentioned current environment data includes the above-mentioned environment fusion data and the above-mentioned global fusion data, and the above-mentioned current IMU data is the current acceleration and angular velocity of the above-mentioned movable device collected by the IMU.

[0109] In one embodiment of the present application, the second processing module includes: a first processing submodule and a second processing submodule,

[0110] The first processing submodule is used to use a Kalman filter algorithm or a particle algorithm to fuse the above-mentioned current environment three-dimensional information and the above-mentioned current environment image to obtain the above-mentioned environment fusion data;

[0111] The second processing submodule is used to use the Kalman filter algorithm to fuse the above-mentioned current global positioning data and the above-mentioned current IMU data to obtain the above-mentioned global fusion data.

[0112] In one embodiment of the present application, the second processing module includes: a third processing submodule and a fourth processing submodule,

[0113] The third processing submodule is used to use an extended Kalman filter to fuse the above-mentioned current environment three-dimensional information, the above-mentioned current environment image and the above-mentioned current IMU data to obtain the above-mentioned environment fusion data;

[0114] The fourth processing submodule is used to use the Kalman filter algorithm to fuse the above-mentioned current global positioning data and the above-mentioned current IMU data to obtain the above-mentioned global fusion data.

[0115] In one embodiment of the present application, the third processing unit includes a fifth processing module and a sixth processing module, the fifth processing module is used to search for a target node based on f(n)=g(n)+h(n) in the process of navigating the above-mentioned movable device based on the above-mentioned current map information, wherein g(n) is the actual cost from the starting node to the node n, h(n) is the estimated cost from the node n to the target node, and f(n) is the total cost of the node n;

[0116] The sixth processing module is used to minimize the total cost of node n and search for the target node until the target node is determined or all reachable nodes are searched.

[0117] In one embodiment of the present application, the third processing unit includes a seventh processing module, and the seventh processing module is used to use the BWA algorithm to perform obstacle avoidance control on the movable device during navigation of the movable device based on the current map information.

[0118] The navigation device of the mobile device includes a processor and a memory, and the acquisition unit, the first processing unit, the second processing unit and the third processing unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0119] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to solve the problem that the mobile device of the existing solution is too dependent on a single navigation technology. Since the single navigation technology has a large error with the actual working conditions, the navigation error of the mobile device is large, which leads to low navigation safety of the mobile device.

[0120] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0121] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the navigation method for the movable device.

[0122] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the navigation method of the movable device is executed when the program is run.

[0123] The embodiment of the present invention provides a device, the device includes a processor, a memory and a program stored in the memory and can be run on the processor, and the processor implements at least the following steps when executing the program: at least obtain the current environment three-dimensional information, the current environment image and the current global positioning data, the current environment three-dimensional information is the three-dimensional information of the current environment where the movable device is located, the current environment image is the image of the current environment where the movable device is located, and the current global positioning data is the global positioning data of the current environment where the movable device is located; at least the current environment three-dimensional information and the current environment image are data-fused to obtain environment fusion data, and determine that the current environment data includes the environment fusion data and the current global positioning data; using enhanced SLAM technology and nonlinear least squares optimization algorithm, constructing and optimizing a map based on the current environment data to obtain current map information to minimize the errors of all edges on the map; navigating the movable device based on the current map information. The device in this article can be a server, PC, PAD, mobile phone, etc.

[0124] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes at least the following method steps: at least obtaining current environment three-dimensional information, current environment image and current global positioning data, wherein the above-mentioned current environment three-dimensional information is the three-dimensional information of the current environment where the movable device is located, the above-mentioned current environment image is the image of the current environment where the above-mentioned movable device is located, and the above-mentioned current global positioning data is the global positioning data of the current environment where the above-mentioned movable device is located; at least the above-mentioned current environment three-dimensional information and the above-mentioned current environment image are data-fused to obtain environment fusion data, and it is determined that the current environment data includes the above-mentioned environment fusion data and the above-mentioned current global positioning data; using enhanced SLAM technology and nonlinear least squares optimization algorithm, a map is constructed and optimized based on the above-mentioned current environment data to obtain current map information to minimize the errors of all edges on the map; and the above-mentioned movable device is navigated based on the above-mentioned current map information.

[0125] The present application also provides a navigation system for a mobile device, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any of the above methods. The current environment three-dimensional information, the current environment image and the current global positioning data provide different types of information. Combining the three can achieve more stable pose estimation and map construction. By using enhanced SLAM technology, rich environmental features are provided. The laser radar provides accurate geometric information. The map constructed after fusion has both the richness of visual features and the accuracy of geometric structure, so that the navigation error of the present application is reduced compared with the existing solution, and the navigation safety is improved, thereby solving the problem that the mobile device of the existing solution is too dependent on a single navigation technology. Due to the large error between the single navigation technology and the actual working conditions, the navigation error of the mobile device is large, which leads to the problem of low navigation safety of the mobile device.

[0126] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0132] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0135] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0136] 1) In the navigation method of the movable device of the present application, the current environment three-dimensional information, the current environment image and the current global positioning data provide different types of information. Combining the three can achieve more stable pose estimation and map construction. By using enhanced SLAM technology, rich environmental features are provided. The lidar provides accurate geometric information. The map constructed after fusion has both the richness of visual features and the accuracy of geometric structure. Compared with the existing solutions, the navigation error of the present application is reduced and the navigation safety is improved, thereby solving the problem that the movable device of the existing solution is too dependent on a single navigation technology. Since there is a large error between the single navigation technology and the actual working conditions, the navigation error of the movable device is large, which leads to the problem of low navigation safety of the movable device.

[0137] 2) The navigation device of the movable device of the present application, the current environment three-dimensional information, the current environment image and the current global positioning data provide different types of information. Combining the three can achieve more stable pose estimation and map construction. By using enhanced SLAM technology, rich environmental features are provided. The lidar provides accurate geometric information. The map constructed after fusion has both the richness of visual features and the accuracy of geometric structure. Compared with the existing solutions, the navigation error of the present application is reduced and the navigation safety is improved, thereby solving the problem that the movable device of the existing solution is too dependent on a single navigation technology. Since there is a large error between the single navigation technology and the actual working conditions, the navigation error of the movable device is large, which leads to the problem of low navigation safety of the movable device.

[0138] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A navigation method for a mobile device, characterized in that: include: At least obtain current environment three-dimensional information, current environment image and current global positioning data, wherein the current environment three-dimensional information is the three-dimensional information of the current environment where the movable device is located, the current environment image is the image of the current environment where the movable device is located, and the current global positioning data is the global positioning data of the current environment where the movable device is located; At least fusing the current environment three-dimensional information and the current environment image to obtain environment fusion data, and determining that the current environment data includes the environment fusion data and the current global positioning data; Using enhanced SLAM technology and nonlinear least squares optimization algorithm, a map is constructed and optimized based on the current environment data to obtain current map information so as to minimize the errors of all edges on the map; The movable device is navigated based on the current map information.

2. The method according to claim 1, characterized in that At least the current environment three-dimensional information and the current environment image are data-fused to obtain environment fusion data, and the current environment data is determined to include the environment fusion data and the current global positioning data, including: When the movable device is in an outdoor environment, the current environment three-dimensional information and the current environment image are subjected to data fusion to obtain the environment fusion data, and the current environment data is determined to include the environment fusion data and the current global positioning data; When the movable device is in an indoor environment, the current environment three-dimensional information and the current environment image are data-fused to obtain the environment fusion data; the current global positioning data and the current IMU data are data-fused to obtain global fusion data; and the current environment data is determined to include the environment fusion data and the global fusion data; Alternatively, the current environment three-dimensional information, the current environment image and the current IMU data are data-fused to obtain the environment fusion data; the current global positioning data and the current IMU data are data-fused to obtain global fusion data; it is determined that the current environment data includes the environment fusion data and the global fusion data, and the current IMU data is the current acceleration and angular velocity of the movable device collected by the IMU.

3. The method according to claim 2, characterized in that The method comprises: fusing the three-dimensional information of the current environment with the current environment image to obtain the environment fusion data, including: fusing the three-dimensional information of the current environment with the current environment image using a Kalman filter algorithm or a particle algorithm to obtain the environment fusion data; The current global positioning data and the current IMU data are subjected to data fusion to obtain global fusion data, including: adopting a Kalman filter algorithm to perform data fusion on the current global positioning data and the current IMU data to obtain the global fusion data.

4. The method according to claim 2, characterized in that: The method comprises: fusing the current three-dimensional environment information, the current environment image and the current IMU data to obtain the environment fusion data, including: using an extended Kalman filter to fuse the current three-dimensional environment information, the current environment image and the current IMU data to obtain the environment fusion data; The current global positioning data and the current IMU data are subjected to data fusion to obtain global fusion data, including: adopting a Kalman filter algorithm to perform data fusion on the current global positioning data and the current IMU data to obtain the global fusion data.

5. The method according to any one of claims 1 to 4, characterized in that In the process of navigating the movable device based on the current map information, the method further includes: Search for the target node based on f(n)=g(n)+h(n), where g(n) is the actual cost from the start node to node n, h(n) is the estimated cost from node n to the target node, and f(n) is the total cost of node n; The goal is to minimize the total cost of node n, and the target node is searched until the target node is determined or all reachable nodes are searched.

6. The method according to any one of claims 1 to 4, characterized in that In the process of navigating the movable device based on the current map information, the method further includes: The BWA algorithm is used to perform obstacle avoidance control on the movable device.

7. A navigation device for a mobile device, characterized in that: include: an acquisition unit, configured to acquire at least current environment three-dimensional information, a current environment image, and current global positioning data, wherein the current environment three-dimensional information is three-dimensional information of a current environment where the movable device is located, the current environment image is an image of the current environment where the movable device is located, and the current global positioning data is global positioning data of the current environment where the movable device is located; A first processing unit is used to perform data fusion on at least the current environment three-dimensional information and the current environment image to obtain environment fusion data, and determine that the current environment data includes the environment fusion data and the current global positioning data; A second processing unit is used to construct and optimize a map based on the current environment data using enhanced SLAM technology and a nonlinear least squares optimization algorithm to obtain current map information so as to minimize errors of all edges on the map; The third processing unit is used to navigate the movable device based on the current map information.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A navigation system for a mobile device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 6.

Images