Visual positioning method, device and storage medium

Abstract

An embodiment of the present application provides a visual positioning method, device, and storage medium. In the embodiment of the present application, during the positioning process of an autonomous mobile device, vision, inertial navigation, and semantic segmentation technologies are combined. The dynamic objects in the environmental image are identified through semantic segmentation technology, and the non-dynamic image features other than the dynamic objects are removed. Then, the non-dynamic image features in vision are combined with the collected inertial navigation data to perform joint positioning on the autonomous mobile device. During the positioning process, on the one hand, the interference of dynamic objects in the environment to positioning can be eliminated based on semantic segmentation technology. On the other hand, by introducing inertial navigation data, the motion state of the device can be reflected, which is beneficial to more accurately predicting the pose of the autonomous mobile device and improving the accuracy of the autonomous mobile device positioning and the robustness of the system.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a visual positioning method, device, and storage medium. Background Art

[0002] Robots are gradually entering people's daily work and life, bringing great convenience to people. Simultaneous Localization And Mapping (SLAM) is a key technology for solving the problem of autonomous mapping and localization of mobile robots in unstructured environments, and is the basis for robots to achieve autonomous movement.

[0003] Among them, visual SLAM is a commonly used SLAM technology. In a visual SLAM system, visual sensors such as monocular or binocular sensors are usually used. The visual sensors work like human eyes to obtain the environmental information around the robot, and based on this, mapping and localization are performed. However, in practical applications, there is a problem of inaccurate positioning in visual SLAM-based positioning. Summary of the Invention

[0004] Multiple aspects of this application provide a visual positioning method, device, and storage medium to improve the accuracy of positioning of autonomous mobile devices and the robustness of the system.

[0005] An embodiment of this application provides a visual positioning method applicable to an autonomous mobile device including a visual sensor and an inertial navigation sensor. The method includes: respectively obtaining environmental images and inertial navigation data collected by the visual sensor and the inertial navigation sensor during the movement of the autonomous mobile device; performing semantic segmentation on the environmental images collected by the visual sensor to obtain semantic labels of at least one object in the environmental images; identifying dynamic objects and non-dynamic image regions outside the image regions where the dynamic objects are located in the environmental images according to the semantic labels of at least one object in the environmental images; and performing joint positioning on the autonomous mobile device according to the non-dynamic image regions in the environmental images and the inertial navigation data.

[0006] An embodiment of the present application further provides an autonomous mobile device, including: a device body, on which a memory, a processor, a vision sensor, and an inertial navigation sensor are provided; the vision sensor is configured to collect environmental images during the movement of the autonomous mobile device; the inertial navigation sensor is configured to collect inertial navigation data during the movement of the autonomous mobile device; the memory is configured to store a computer program; the processor is coupled to the memory and is configured to execute the computer program to: perform semantic segmentation on the environmental images collected by the vision sensor to obtain semantic labels of at least one object in the environmental images; identify dynamic objects in the environmental images and non-dynamic image regions outside the image regions where the dynamic objects are located according to the semantic labels of at least one object in the environmental images; perform joint positioning on the autonomous mobile device according to the non-dynamic image regions in the environmental images and the inertial navigation data.

[0007] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the vision positioning method provided by the embodiment of the present application.

[0008] In the embodiment of the present application, during the positioning process of the autonomous mobile device, vision, inertial navigation, and semantic segmentation technologies are combined. Through the semantic segmentation technology, dynamic objects in the environmental images are identified and non-dynamic image features outside the dynamic objects are removed. Then, the non-dynamic image features in the vision and the collected inertial navigation data are combined to perform joint positioning on the autonomous mobile device. During the positioning process, on the one hand, the interference of dynamic objects in the environment to the positioning can be eliminated based on the semantic segmentation technology. On the other hand, by introducing inertial navigation data, the motion state of the device can be reflected, which is beneficial to more accurately predicting the pose of the autonomous mobile device and improving the positioning accuracy of the autonomous mobile device and the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0010] Figure 1 is a flowchart of a vision positioning method provided by an exemplary embodiment of the present application;

[0011] Figure 2 is the original image in the laboratory scenario provided by the present application and the image after semantic segmentation of the original image;

[0012] Figure 3 is a flowchart of another vision positioning method provided by an exemplary embodiment of the present application;

[0013] Figure 4Schematic structural diagram of a visual positioning device provided by an exemplary embodiment of the present application;

[0014] Figure 5 Schematic structural diagram of an autonomous mobile device provided by an exemplary embodiment of the present application;

[0015] Figure 6 Schematic structural diagram of a robot provided by an exemplary embodiment of the present application. Detailed implementation manners

[0016] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Apparently, the described embodiments are only a part rather than all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0017] Before introducing the embodiments of the present application, the autonomous mobile device in the embodiments of the present application will be introduced and described first. The autonomous mobile device in the embodiments of the present application can be any mechanical device that can highly autonomously move in a specified environmental space. For example, it can be a robot, a purifier, an unmanned handling vehicle, etc. Among them, the robot can be a household robot, such as a floor cleaning robot, a home companion robot, etc.; it can also be a commercial robot, such as a cruise robot, a guiding robot or a service robot that assists users in handling various affairs in places such as shopping malls, supermarkets, banks, hospitals, airports or railway stations, and a handling robot in various warehouses, etc. In the embodiments of the present application, the autonomous mobile device has a traveling device and can autonomously move in the current environment; in addition, the autonomous mobile device also has a visual sensor and an inertial navigation sensor, which can respectively collect the surrounding environment images and inertial navigation data reflecting the motion state of the autonomous mobile device during the movement process. Among them, the visual sensor can be a monocular camera, a binocular camera or a depth camera (Red GreenBlue-Depth map, RGB-D), etc.; the inertial navigation sensor includes but is not limited to: an inertial measurement unit (Inertial measurement unit, IMU) or an odometer. Among them, a six-axis IMU includes a three-axis gyroscope and a three-axis accelerometer, and a nine-axis IMU includes a three-axis magnetometer, a three-axis gyroscope and a three-axis accelerometer. The inertial navigation data collected by the IMU includes the angular velocity and acceleration of the autonomous mobile device when moving in three-dimensional space. In addition, it can also measure the gravitational acceleration, but is not limited thereto. The inertial navigation data collected by the odometer is, on the one hand, the pose (position and rotation angle) of the autonomous mobile device, and on the other hand, the speed (forward speed and turning speed) of the autonomous mobile device. On this basis, by combining vision, inertial navigation and semantic segmentation technologies and applying them to the SLAM system of the autonomous mobile device, the joint positioning, navigation and obstacle avoidance of the autonomous mobile device are realized, and the robustness of the SLAM system is effectively improved.

[0018] The following will, with reference to the accompanying drawings, detail the technical solutions provided by the embodiments of the present application.

[0019] Figure 1 The flowchart of a visual positioning method provided for an exemplary embodiment of the present application is applicable to an autonomous mobile device including a visual sensor and an inertial navigation sensor, as Figure 1 shown, the method includes:

[0020] 101. Respectively obtain the environmental image and inertial navigation data collected by the visual sensor and the inertial navigation sensor during the movement of the autonomous mobile device;

[0021] 102. Perform semantic segmentation on the environmental image collected by the visual sensor to obtain the semantic labels of at least one object in the environmental image;

[0022] 103. Identify dynamic objects in the environmental image and non-dynamic image regions outside the image regions where the dynamic objects are located according to the semantic labels of at least one object in the environmental image.

[0023] 104. Perform joint positioning on the autonomous mobile device according to the non-dynamic image regions in the environmental image and inertial navigation data.

[0024] In this embodiment, the vision sensor collects environmental images, and the inertial navigation sensor collects inertial navigation data. In order to obtain more accurate inertial navigation data between two frames of environmental images, in this embodiment, the vision sensor and the inertial navigation sensor are synchronously triggered by a hardware clock to ensure the real-time performance of the system. In addition, the vision sensor and the inertial navigation sensor are rigidly connected. Through the method of pose calibration, the conversion relationship between the coordinate systems of the two can be calculated. The SLAM technology that fuses vision and inertial navigation shows higher robustness compared with the traditional vision SLAM technology. For example, the data collected by both can reflect the pose of the autonomous mobile device. If there is a large difference between the pose of the autonomous mobile device predicted by the vision sensor and the pose of the autonomous mobile device predicted by the inertial navigation sensor at a certain moment, it can be considered that there is a large error in one of the data, and the environmental image or inertial navigation data collected at this moment can be discarded, which can effectively avoid problems such as the autonomous mobile device slipping and visual matching errors.

[0025] In this embodiment, on the basis of the fusion of vision and inertial navigation, further, semantic segmentation technology can also be combined. Through semantic segmentation, the dynamic objects in the environment are predicted and segmented to identify the dynamic objects and fixed objects in the environment. During the SLAM process, the dynamic objects are removed, and the fixed objects are added to the environmental map, which can be used as a priori information to make subsequent positioning and navigation based on the environmental map more accurate. Specifically, after the environmental image is collected, semantic segmentation can be performed on the environmental image to obtain at least one object included in the environmental image and its semantic label. In this embodiment, the vision sensor has a certain working frequency and can continuously collect surrounding environmental images. Optionally, semantic segmentation can be performed on each frame of the environmental image collected by the vision sensor. Of course, semantic segmentation can also be selectively performed on some environmental images, and this is not limited. In an optional embodiment, in order to reduce the computational complexity of semantic segmentation and subsequent processing, the method of performing semantic segmentation on part of the environmental images belonging to key frames is adopted.

[0026] Among them, the key frame refers to the image frame with representational significance in the environmental image, that is, the key frame can represent the ordinary frames near it. In this embodiment, the implementation manner of selecting key frames is not limited. For example, one frame of environmental image can be selected as a key frame every fixed number of frames; it is also possible to determine whether the number of feature points in the currently tracked environmental image is greater than a set feature point threshold, such as 50 feature points, and obtain the environmental image with the number of feature points greater than the set feature point threshold as the key frame; it is also possible to determine whether there is sufficient transformation in the spatial position of the currently tracked environmental image from the nearest key frame. For example, it can be determined whether the rotation angle of the currently tracked environmental image relative to the previous key frame is greater than a set threshold, such as 20 degrees. If so, it means that there is sufficient transformation in the spatial position of the currently tracked environmental image from the nearest key frame, and then this frame of environmental image is used as the key frame.

[0027] Whether semantic segmentation is performed on each frame of environmental image or on some environmental images belonging to key frames, in this embodiment, the implementation manner of performing semantic segmentation on environmental images is not limited. For example, a semantic segmentation model can be used to perform semantic segmentation on environmental images. Semantic segmentation combines image classification, object detection, and image segmentation. Semantic segmentation can divide an image into regional blocks with certain semantic meanings and identify the semantic categories of each regional block, realizing a semantic reasoning process from the bottom layer to the high layer, and finally obtaining a segmented image with pixel semantic annotations. As Figure 2 shown, the original image in the laboratory scenario and the image obtained after semantic segmentation of the original image are presented. Among them, people and the ground are divided into different regional blocks. The semantic segmentation model can be, but is not limited to: Pyramid Scene Parsing Network (PSP), Generative Adversarial Nets (GAN), or Fully Convolutional Networks (FCN), etc.

[0028] In this embodiment, after performing semantic segmentation on the environmental image to obtain the semantic labels of at least one object in the environmental image, the dynamic objects in the environmental image can be identified according to the semantic labels of the at least one object; further, the image region where the dynamic object is located in the environmental image and other image regions except the image region where the dynamic object is located can also be identified, simply referred to as the non-dynamic image region. Among them, the semantic label of an object refers to information that can be understood and interpreted by natural language about what the object is or what category it belongs to, etc. For example, the semantic labels of the objects in the environmental image obtained by semantic segmentation may be: people, counters, ATMs, etc. In this embodiment, the implementation manner of identifying the dynamic objects in the environmental image according to the semantic labels of the at least one object is not limited. In an alternative embodiment, a correspondence relationship between multiple known objects and their status information is pre-maintained, and the status information reflects the basic status attributes of the known objects, such as whether they are dynamic, fixed, or substantially fixed. The fixed state refers to a state that remains stable for a long time, and a fixed object refers to an object that remains in a state of unchanged position or shape for a long time. Among them, fixed objects can be items such as printers, water dispensers, or desks in an office; they can also be items such as columns, stairs, game machines, billboards, or flower pots in a shopping mall; they can also be items such as counters, seats, ATMs, or self-service machines in a bank; they can also be shelves, goods, etc. in an industrial handling scenario. Dynamic refers to a state that may change irregularly at any time. Dynamic objects can be, for example, people, pets, or movable vehicles, etc., and these objects may change their positions or forms at any time (for example, changes in standing, sitting, lying, or walking states). Optionally, the correspondence relationship between the known objects and the status information can be expressed as: semantic label of the object <——> status information of the object, such as people <——> dynamic; movable vehicle <——> dynamic; desk <——> solid state; printer <——> solid state; water dispenser <——> solid state; flower pot <——> solid state; stairs <——> solid state; ATM <——> solid state, etc. Based on this, the dynamic objects in the environmental image are identified according to the semantic labels of at least one object in the environmental image and the status information of whether multiple pre-maintained known objects are fixed.

[0029] Further, after identifying dynamic objects in the environmental image based on the semantic labels of at least one object in the environmental image, the image regions where the dynamic objects are located and the non-dynamic image regions other than the image regions where the dynamic objects are located in the environmental image can be further identified. Since dynamic objects may change irregularly at any time, in this embodiment, dynamic objects are not considered, and fixed objects and background features in the environmental image are mainly considered. Therefore, the autonomous mobile device can be jointly located according to the non-dynamic image regions in the environmental image and inertial navigation data. Among them, during the process of map building, positioning, or navigation, the interference of dynamic objects in the environment to positioning is eliminated, the positioning accuracy of the autonomous mobile device is improved, and the robustness of the system is improved. For fixed objects, they are used as positioning constraints, added as map points in the environmental map, and obstacle avoidance information and drivable area information are given, which can improve the robustness of positioning and navigation. On the other hand, by integrating inertial navigation data and combining the visual tracking process, such as optical flow tracking, feature matching tracking, etc., the matching speed of visual tracking can be improved, especially the matching speed of the autonomous mobile device in the rotating state can be improved, and the robustness of the system is improved.

[0030] In an alternative embodiment, the environmental images can be classified into key frames and ordinary frames, and different joint positioning methods can be adopted for key frames and ordinary frames. During the real-time positioning process of the autonomous mobile device, whenever a frame of environmental image is collected, it can be identified whether the environmental image is a key frame; if so, the autonomous mobile device is jointly located using the joint positioning method A1; if not, the autonomous mobile device is jointly located using the joint positioning method A2. The following is a detailed description of each:

[0031] Combined positioning method A1:

[0032] In the case where the current environmental image is identified as a key frame, semantic segmentation is performed on the environmental image belonging to the key frame to obtain the dynamic objects included in the environmental image and the non-dynamic image regions other than the image regions where the dynamic objects are located. Furthermore, in a tightly coupled manner, multiple consecutive key frame images including the current environmental image and the inertial navigation data between the consecutive key frame images are jointly optimized to obtain the current pose of the autonomous mobile device.

[0033] Among them, tight coupling means jointly estimating a set of variables using the raw data of the vision sensor and the inertial navigation sensor (environmental image and inertial navigation data), so as to obtain the pose of the autonomous mobile device based on this set of variables. Adopting the tight coupling method can make full use of the environmental image of the vision sensor and the inertial navigation data of the inertial navigation sensor to achieve better results.

[0034] In this embodiment, multiple consecutive key-frame images are two or more adjacent key-frame images. The multiple consecutive key-frame images can be divided into a current key frame and one or more historical key frames. For example, 10 consecutive key-frame images are respectively key-frame images K, K-1, ..., K-9, where key-frame image K is the current key frame, and key-frame images K-1, ..., K-9 are historical key frames. Whether it is the current key frame or the historical key frame, in this embodiment, it can include a non-dynamic image area, and further optionally, it can also include the image area where the dynamic object is located. For the convenience of distinction and description, the non-dynamic image area in the current key frame is called the current non-dynamic image area, and the non-dynamic image area in the historical key frame is called the historical non-dynamic image area. In this embodiment, the current pose of the autonomous mobile device can be obtained based on the environmental image currently collected by the vision sensor. For the convenience of description, the current pose of the autonomous mobile device obtained from the environmental image currently collected by the vision sensor can be called the observation value of the vision sensor; the current pose of the autonomous mobile device predicted according to the historical environmental image is called the prediction value of the vision sensor. In this embodiment, the inertial navigation sensor can collect inertial navigation data from the historical moment to the current moment. Therefore, based on the pose of the autonomous mobile device at the historical moment and combined with the inertial navigation data from the historical moment to the current moment, the pose of the autonomous mobile device at the current moment can be predicted. For the convenience of description, the pose of the autonomous mobile device predicted by the inertial navigation data is called the prediction value of the inertial navigation sensor.

[0035] Based on the above, an implementation method that uses a tightly coupled method to jointly optimize multiple consecutive key-frame images including the current environmental image and the inertial navigation data between the consecutive key-frame images includes: calculating the observation value of the vision sensor according to the feature points in the current non-dynamic image area, and combining the feature points in the historical non-dynamic image area to calculate the prediction value of the vision sensor; and calculating the prediction value of the inertial navigation sensor according to the inertial navigation data collected between the current key frame and the historical key frame; jointly optimizing the observation value, prediction value of the vision sensor and the prediction value of the inertial navigation sensor on the manifold to obtain the pose of the autonomous mobile device in the environmental map.

[0036] The above implementation method will be described in detail in four parts below.

[0037] (1) Calculating the observation value of the vision sensor:

[0038] In this embodiment, when performing semantic segmentation on the environmental map, in addition to obtaining at least one object and its semantic label, the background area in the environmental image can also be detected. Among them, the at least one object can be a dynamic object or a static object. To reduce the interference of dynamic objects on the positioning process, in this embodiment, the observation value of the vision sensor is calculated based on the static objects in the non-dynamic image area. The specific process includes: determining a candidate map area according to the semantic labels of the static objects included in the current non-dynamic image area and the semantic labels in the current existing environmental map; matching the feature points in the current non-dynamic image area in the candidate map area to obtain the observation value of the vision sensor.

[0039] In an alternative embodiment, during the feature matching of the current non-dynamic image area in the candidate map area, the feature points on the static object and the feature points in the background area (abbreviated as background feature points) can have the same weight coefficient. Based on this, the current non-dynamic image area can be directly matched in the candidate map area to obtain the observation value of the vision sensor.

[0040] In another alternative embodiment, the static object has more obvious features relative to the background, which is more conducive to matching in the candidate map area to obtain the observation value of the vision sensor. Based on this, an implementation manner of matching the feature points in the current non-dynamic image area in the candidate map area to obtain the observation value of the vision sensor includes: configuring weight coefficients for the feature points in the current non-dynamic image area, where the weight coefficient of the feature points on the static object is greater than the weight coefficient of the background feature points; matching according to the feature points and their weight coefficients in the current non-dynamic image area in the candidate map area to obtain the observation value of the vision sensor.

[0041] (2) Calculate the predicted value of the vision sensor

[0042] In an alternative embodiment, the predicted value of the vision sensor can be obtained according to the co-visible feature points between the current non-dynamic image area and the historical non-dynamic image area. Among them, the co-visible feature points refer to the feature points that exist in both the current non-dynamic image area and the historical non-dynamic image area. Among them, the historical key frame images can be one or more. Based on this, for each historical non-dynamic image area, the co-visible feature points between the current non-dynamic image area and the historical non-dynamic image area can be identified, and the relative position change of the co-visible feature points between the current non-dynamic image area and the historical non-dynamic image area can be calculated; according to the relative position change of the co-visible feature points and the pose of the autonomous mobile device at the corresponding historical moment of this historical non-dynamic image area, a predicted value of the vision sensor at the current moment can be obtained. If there are multiple historical key frame images, multiple predicted values of the vision sensor at the current moment can be obtained.

[0043] Further optionally, an implementation manner for calculating the relative position change of the co-visible feature points existing between the current non-dynamic image region and the historical non-dynamic image region includes: determining a candidate map region in the current existing environmental map according to the semantic label of the fixed object included in the current non-dynamic image region; performing feature matching on the current non-dynamic image region in the candidate map region to obtain the position coordinates of the co-visible feature points in the current non-dynamic image region in the environmental map; calculating the relative position change of the co-visible feature points according to the position coordinates of the co-visible feature points in the current non-dynamic image region in the environmental map and the position coordinates of the co-visible feature points in the historical non-dynamic image region in the environmental map.

[0044] (3) Calculate the predicted value of the inertial navigation sensor

[0045] In this embodiment, for the historical moment corresponding to the historical key frame image, the pose of the autonomous mobile device is known, that is, the observation value of the inertial navigation sensor at the historical moment corresponding to the historical key frame is known. Combining the inertial navigation data collected between the current key frame image and the historical key frame image, the predicted value of the inertial navigation sensor at the current moment can be obtained. If there are multiple historical key frame images, a predicted value of the inertial navigation sensor at the current moment can be obtained according to the inertial navigation data between each historical key frame image and the current key frame image, so as to obtain multiple predicted values of the inertial navigation sensor at the current moment.

[0046] (4) Joint optimization

[0047] In this embodiment, there is no limitation on the implementation manner of jointly optimizing the observation value, predicted value of the vision sensor and the predicted value of the inertial navigation sensor on the manifold to obtain the pose of the autonomous mobile device in the environmental map. Among them, the state of the autonomous mobile device running at time k can be described as x k = {R k , p k , u k , b gk , b ak}, where the parameters represent rotation, displacement, inertial measurement quantity, gyroscope bias, and accelerometer bias at time k in sequence. Then, the state of the autonomous mobile device at time k can be predicted according to the state of the autonomous mobile device estimated at time k - 1 and the inertial measurement quantity during the process from time k - 1 to time k, which is expressed as f(x k-1 , u k ). Among them, there is a difference between the predicted state of the autonomous mobile device at time k and the real state of the autonomous mobile device at time k, and this difference is caused by the noise of the inertial navigation sensor. The noise term is expressed as w k . Then, the prediction equation of the autonomous mobile device can be expressed as: x k = f(xk-1 , u k ) + w k , where the prediction equation can reflect the predicted value of the vision sensor. Additionally, the projection model of the vision sensor can be expressed as h(P, x k ), and the set of fixed objects in the environment where the autonomous mobile device is located is represented as P k , p k is the set of projection points of the fixed objects in the environment on the environmental image collected by the vision sensor, and the observation noise of the vision sensor is v k , then the measurement equation of the autonomous mobile device can be expressed as: p k = h(P k , x k ) + v k , where the measurement equation can reflect the observed value of the vision sensor.

[0048] In this embodiment, taking the predicted value of the inertial navigation sensor as a constraint, using the prediction equation and the measurement equation, with the minimum reprojection error as the optimization objective, joint optimization is performed on the manifold. The joint optimization formula is as follows:

[0049]

[0050] Among them, in the above formula, i represents the i-th inertial navigation data, k represents the k-th key frame, j represents the integral term of the inertial navigation data, e represents the error term, and W is the covariance matrix. The first term after the equal sign is the error term of the inertial navigation data, and the second is the error term of the reprojection of the feature points on the environmental image.

[0051] After the above optimization process, the pose of the autonomous mobile device at the current moment can be obtained; and for the next key frame image collected, the above optimization process can be continued, thereby continuously positioning the pose of the autonomous mobile device.

[0052] Combined positioning method A2:

[0053] In the case where the current environmental image is identified as a normal frame, semantic segmentation may not be performed on the current environmental image. Instead, the current environmental image can be directly used to locate the autonomous mobile device. Specifically, joint optimization can be performed based on the current environmental image, its previous environmental image, and the inertial navigation data between the two environmental images to obtain the pose of the autonomous mobile device in the environmental map. Among them, the previous environmental image adjacent to the current environmental image may be a normal frame or a key frame, and no limitation is imposed on this. In the case where the current environmental image is a normal frame, the specific joint optimization process is the same as or similar to the foregoing. The difference is that the joint optimization process uses all the image features in all adjacent two-frame environmental images, rather than only considering the non-dynamic image regions in the environmental image. For detailed content, reference can be made to the foregoing embodiments, and details will not be elaborated here.

[0054] Among them, as the autonomous mobile device moves, the vision sensor continuously acquires environmental images, and the inertial navigation sensor continuously acquires inertial navigation data. During this process, according to whether the acquired environmental image is a key frame, the joint positioning methods A1 and A2 are continuously executed, and the pose of the autonomous mobile device can be continuously located.

[0055] In an alternative embodiment, in addition to outputting the pose of the autonomous mobile device in the environmental map during the above joint optimization process, the mapped positions of the feature points in the current non-dynamic image region in the environmental map can also be obtained. Based on this, it can be determined whether there are map points corresponding to the feature points in the current non-dynamic image region in the environmental map. If there are no map points corresponding to the feature points in the environmental map, map points are marked at the mapped positions in the environmental map to construct the environmental map in real time. Further optionally, if there are map points corresponding to the feature points in the environmental map, but the positions of the map points are different from the mapped positions of the feature points in the environmental map, the map points are corrected according to the mapped positions of the feature points in the environmental map. Among them, the map point refers to the position point occupied by an obstacle recorded in the environmental map.

[0056] In an alternative embodiment, as Figure 3As shown, after the combined positioning of the autonomous mobile device, it further includes: Step 105, performing navigation or obstacle avoidance on the autonomous mobile device according to the pose of the autonomous mobile device. Among them, the navigation process includes: determining the target position that the autonomous mobile device needs to reach, calculating the navigation path from the current position to the target position based on the pose of the autonomous mobile device and in combination with the environmental map; guiding the autonomous mobile device to move along the navigation path to the target position. Among them, the obstacle avoidance process includes: the autonomous mobile device can analyze the environmental image collected by the vision sensor to obtain the positions of the stationary objects and dynamic objects in the environmental map in the forward direction of the autonomous mobile device. Further, it can also estimate the movement direction and movement speed of the dynamic objects. For stationary objects, avoidance can be performed based on the positions of the stationary objects; for dynamic objects, avoidance can be performed based on the movement direction and movement speed of the dynamic objects. During the obstacle avoidance process, if the original navigation path is deviated, the navigation path can be re-planned in combination with the target position and move along the navigation path to the target position.

[0057] In an alternative embodiment, the autonomous mobile device can perform positioning based on the constructed environmental map while constructing the environment. For example, when the autonomous mobile device is first applied to a working environment, the SLAM technology needs to be used to construct the environmental map of the working environment. During the map construction process, it is necessary to locate the pose of the autonomous mobile device in real time to achieve autonomous movement. Further optionally, the visual sensor on the autonomous mobile device is a monocular camera. The data obtained by the monocular camera each time is only single-image data, and there is a problem of uncertain scale information. Scale information can be understood as the proportional relationship between the environmental map constructed by the autonomous mobile device and the actual physical map. The uncertainty of the scale information may lead to inaccurate positioning, navigation, and map construction. Based on this, during the process of constructing the environmental map while positioning, before starting to construct the environmental map, the method of fusing vision and inertial navigation can be used to solve the problem of scale degradation of the environmental map constructed based on the visual sensor, so that the scale information of the environmental map is clear, and the coordinates of the environmental map are aligned with the coordinates of the physical world, thereby better coping with problems such as slipping and visual matching errors of the autonomous mobile device. Among them, the implementation method of fusing inertial navigation to solve the map scale information includes: selecting adjacent environmental images with co-visible feature points in the non-dynamic image area and the position change of the co-visible feature points being greater than the set threshold from the environmental images collected by the visual sensor. Specifically, feature points can be found in two adjacent environmental images respectively, and the feature points in the two adjacent environmental images can be matched to find the co-visible feature points; judging whether the relative position change of the co-visible feature points is greater than the set threshold. If the relative position change of the co-visible feature points is greater than the set threshold, it means that the pose of the autonomous mobile device has changed to a certain extent when taking these two images and has reference value, then select these two adjacent environmental images; pre-integrate the inertial data collected by the inertial sensor between the adjacent environmental images to obtain the relative position change of the visual sensor when collecting the adjacent environmental images; according to the conversion relationship between the inertial coordinate system and the visual coordinate system, convert the relative position change from the inertial coordinate system to the visual coordinate system; estimate the scale information of the environmental map according to the relative position change in the visual coordinate system to unify the coordinates of the environmental map and the actual physical world.

[0058] In an alternative embodiment, whether ordinary frames or key frames are used, during the joint optimization process, the feature points in the environmental map constructed by the autonomous mobile device through the SLAM technology can be optimized to obtain a more accurate environmental map, which is beneficial to subsequent positioning and navigation based on the environmental map and improves the robustness of the system.

[0059] The visual positioning method provided in this embodiment can be applied to various scenarios, such as industrial handling scenarios, commercial cleaning scenarios, or bank reception scenarios, etc. The application of the visual positioning method provided in the embodiments of the present application in specific application scenarios will be exemplified below:

[0060] For example, in an industrial handling scenario, for an Automated Guided Vehicle (AGV), there is no human intervention in its operation scenario. The main static objects in this scenario come from the shelves, and the main dynamic objects come from other AGVs. For any AGV, when receiving a handling instruction, it needs to move from the current position to the position where the goods are located and transport the goods from that position to the destination position. Or, when receiving an instruction to assist other AGVs, it also needs to move from the current position to the AGV to be assisted to help the AGV to be assisted transport the goods. All of these require the AGV to move from one position to another, and it is necessary to locate and navigate the AGV. In this embodiment, the AGV is provided with a camera and an IMU. During the movement, it can collect environmental images using the camera while collecting inertial navigation data reflecting the motion state of the AGV using the IMU; perform semantic segmentation on the collected environmental images to identify static objects such as shelves in the environmental images, dynamic objects such as other AGVs, and background such as drivable area lane lines existing on the ground; remove dynamic objects such as other AGVs, and use the remaining image area including the shelves and lane lines in combination with the inertial navigation data collected by the IMU to perform joint positioning and build an environmental map for the autonomous mobile device. Among them, static objects such as shelves can be added to the environmental map as a landmark for positioning, while dynamic objects such as other AGVs will not be reflected in the environmental map, which is beneficial to improving the positioning and navigation accuracy based on the environmental map. In addition, during the positioning process, only static objects such as shelves are considered, and dynamic objects such as other AGVs are not considered. Combining the environmental map containing only static objects such as shelves can avoid the interference of dynamic objects and is beneficial to improving the positioning accuracy.

[0061] Another example is in the bank reception scenario. Self-service robots often need to avoid obstacles for people entering the hall, chairs, columns, or teller machines in the hall. Then the self-service robot can collect environmental data in the bank hall using a vision sensor during the autonomous service process or the first traversal mapping process, and at the same time collect inertial navigation data reflecting the motion state of the self-service robot using an inertial navigation sensor; perform semantic segmentation on the collected environmental data to obtain static and dynamic objects in the environmental image. Among them, the static objects include chairs, columns, or teller machines, etc., and the dynamic objects include people or other autonomous mobile devices, etc.; on the one hand, the static objects can be marked in the environmental map as the basis for positioning, while dynamic objects such as people or other autonomous mobile devices will not be reflected in the environmental map, which can effectively remove dynamic interference. On the other hand, the environmental image after removing dynamic objects can be combined with the inertial navigation data to perform joint positioning on the autonomous mobile device based on the environmental map containing only static objects.

[0062] For another example, in the scenario of commercial cleaning, for a floor cleaning robot, the main fixed objects in its working scenario come from columns, chairs or doors in the hall, etc. The main dynamic objects in this scenario are consumers or staff in the mall. When the floor cleaning robot performs the cleaning task, it can collect environmental information while constructing an environmental map, and perform positioning and navigation based on the environmental map, so as to clean the whole or part of the environmental area. Specifically, the floor cleaning robot can be equipped with a camera and an IMU. During the movement, it can use the camera to collect environmental images while using the IMU to collect inertial navigation data reflecting the movement state of the floor cleaning robot; perform semantic segmentation on the collected environmental images to identify fixed objects such as columns, chairs or walls in the environmental images, and dynamic objects such as consumers or staff in the environment. Eliminate dynamic objects such as consumers or staff in the environment, and use the remaining fixed objects in the environmental images, such as columns, chairs or doors, etc., combined with the inertial navigation data collected by the IMU, to perform joint positioning and construct an environmental map for the floor cleaning robot. Among them, columns, chairs or doors, etc. can be added to the environmental map as a kind of landmark, while dynamic objects such as consumers or staff will not be reflected in the environmental map, which is beneficial to improving the positioning and navigation accuracy based on the environmental map. In addition, during the positioning process, only fixed objects such as columns, chairs or doors are considered, and dynamic objects such as consumers or staff are not considered. Combining with the environmental map containing only fixed objects can avoid the interference of dynamic objects and is beneficial to improving the positioning accuracy. In the commercial cleaning scenario, through the fusion of visual positioning and segmentation, such as an identification scheme that finally converges to a cube or an ellipse based on a bounding box or a pixel-level segmentation scheme, etc., the position and range of obstacles are made more accurate, and a more accurate cleaning plan can be obtained without relying on other ranging sensors.

[0063] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0064] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The operation numbers such as 101, 102, etc. are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are different types.

[0065] Figure 4 FIG. is a schematic structural diagram of a visual positioning device provided by an exemplary embodiment of the present application. As Figure 4 shown, the device includes: an acquisition module 41, a segmentation module 42, an identification module 43, and a positioning module 44.

[0066] The acquisition module 41 is configured to acquire environmental images and inertial navigation data collected by a visual sensor and an inertial navigation sensor during the movement of the autonomous mobile device;

[0067] The segmentation module 42 is configured to perform semantic segmentation on the environmental images collected by the visual sensor to obtain semantic labels of at least one object in the environmental images

[0068] The identification module 43 is configured to identify a dynamic object and a non-dynamic image region outside the image region where the dynamic object is located in the environmental image according to the semantic labels of at least one object in the environmental image;

[0069] The positioning module 44 is configured to perform joint positioning on the autonomous mobile device according to the non-dynamic image region in the environmental image and the inertial navigation data.

[0070] For the detailed functions of the acquisition module 41, the segmentation module 42, the identification module 43, and the positioning module 44, reference may be made to the corresponding descriptions in the foregoing embodiments, which will not be elaborated herein.

[0071] Figure 5 FIG. is a schematic structural diagram of an autonomous mobile device provided by an exemplary embodiment of the present application. As Figure 5 shown, the autonomous mobile device includes: a device body 50, and a memory 54, a processor 55, a visual sensor 51, and an inertial navigation sensor 52 are provided on the device body 50.

[0072] The visual sensor 51 is configured to collect environmental images during the movement of the autonomous mobile device.

[0073] The inertial navigation sensor 52 is configured to collect inertial navigation data during the movement of the autonomous mobile device.

[0074] A memory 54 for storing computer programs and configurable to store various other data to support operations on a computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0075] The memory 54 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks or optical disks.

[0076] A processor 55, coupled to the memory 54, for executing the computer programs in the memory 54 to: perform semantic segmentation on the environmental image acquired by the vision sensor to obtain semantic labels of at least one object in the environmental image; identify dynamic objects in the environmental image and non-dynamic image regions outside the image regions where the dynamic objects are located according to the semantic labels of at least one object in the environmental image; perform joint positioning on the autonomous mobile device according to the non-dynamic image regions in the environmental image and inertial navigation data.

[0077] In an optional embodiment, when the processor 55 performs semantic segmentation on the environmental image acquired by the vision sensor to obtain semantic labels of at least one object in the environmental image, it is specifically configured to: for each frame of the acquired environmental image, identify whether the environmental image is a key frame; if so, perform semantic segmentation on the environmental image to obtain semantic labels of at least one object in the environmental image.

[0078] In an optional embodiment, when the processor 55 performs joint positioning on the autonomous mobile device according to the non-dynamic image regions in the environmental image and inertial navigation data, it is specifically configured to: if the current environmental image is a key frame image, adopt a tightly coupled manner to jointly optimize multiple consecutive key frame images including the current environmental image and the inertial navigation data between the consecutive key frame images to obtain the current pose of the autonomous mobile device; if the current environmental image is a normal frame image, adopt a tightly coupled manner to jointly optimize the current environmental image, the previous environmental image and the inertial navigation data between the two environmental images to obtain the current pose of the autonomous mobile device.

[0079] In an alternative embodiment, the multiple consecutive key-frame images include: a current key frame and historical key frames. When the processor 55 jointly optimizes the multiple consecutive key-frame images including the current environmental image and the inertial data between the consecutive key-frame images in a tightly coupled manner, it is specifically configured to: calculate the observation value of the vision sensor according to the feature points in the current non-dynamic image region, and calculate the prediction value of the vision sensor in combination with the feature points in the historical non-dynamic image region; and calculate the prediction value of the inertial sensor according to the inertial data collected between the current key frame and the historical key frame; jointly optimize the observation value, prediction value of the vision sensor, and the prediction value of the inertial sensor on the manifold to obtain the pose of the autonomous mobile device in the environmental map; wherein, the current non-dynamic image region and the historical non-dynamic image region respectively refer to the non-dynamic image regions in the current key frame and the historical key frames.

[0080] In an alternative embodiment, when the processor 55 calculates the observation value of the vision sensor according to the feature points in the current non-dynamic image region, it is specifically configured to: determine a candidate map region according to the semantic labels of the fixed objects included in the current non-dynamic image region and the semantic labels in the current existing environmental map; match the feature points in the current non-dynamic image region in the candidate map region to obtain the observation value of the vision sensor.

[0081] In an alternative embodiment, the current non-dynamic image region includes fixed objects and a background; when the processor 55 matches the feature points in the current non-dynamic image region in the candidate map region to obtain the observation value of the vision sensor, it is specifically configured to: configure weight coefficients for the feature points in the current non-dynamic image region, wherein the weight coefficients of the feature points on the fixed objects are greater than the weight coefficients of the background feature points; match the feature points in the current non-dynamic image region and their weight coefficients in the candidate map region to obtain the observation value of the vision sensor.

[0082] In an alternative embodiment, when the processor 55 calculates the prediction value of the vision sensor in combination with the feature points in the historical non-dynamic image region, it is specifically configured to: for each historical non-dynamic image region, calculate the relative position change of the co-visible feature points existing between the current non-dynamic image region and the historical non-dynamic image region; obtain a prediction value of the vision sensor according to the relative position change of the co-visible feature points and the pose of the autonomous mobile device at the corresponding historical moment.

[0083] In an alternative embodiment, during the joint optimization process, the processor 55 is further configured to: obtain the mapped position of the feature points in the current non-dynamic image region in the environmental map; if there is no map point corresponding to the feature points in the environmental map, mark a map point at the mapped position in the environmental map to construct the environmental map in real time.

[0084] In an alternative embodiment, the processor 55 is further configured to: if there are map points corresponding to feature points in the environmental map, but the positions of the map points are different from the mapped positions of the feature points in the environmental map, correct the map points according to the mapped positions of the feature points in the environmental map.

[0085] In an alternative embodiment, after jointly positioning the autonomous mobile device, the processor 55 is further configured to: navigate or avoid obstacles for the autonomous mobile device according to the located pose of the autonomous mobile device.

[0086] In an alternative embodiment, the vision sensor is a monocular camera, and the processor 55 is further configured to: select adjacent environmental images in the non-dynamic image region of the environmental images collected by the vision sensor, where there are co-visible feature points and the position changes of the co-visible feature points are greater than a set threshold; perform pre-integration on the inertial data collected by the inertial sensor between the adjacent environmental images to obtain the relative position change of the vision sensor when collecting the adjacent environmental images; convert the relative position change from the inertial coordinate system to the vision coordinate system according to the conversion relationship between the inertial coordinate system and the vision coordinate system; estimate the scale information of the environmental map according to the relative position change in the vision coordinate system to unify the coordinates of the environmental map with the actual physical world.

[0087] Further, as Figure 5 shown, the autonomous mobile device further includes: other components such as a communication component 56, a display 57, a power supply component 58, an audio component 59, etc. Figure 5 Only some components are schematically shown, and it does not mean that the autonomous mobile device only includes Figure 5 the components shown.

[0088] Optionally, the autonomous mobile device in this embodiment may be a robot, a purifier, etc.

[0089] In an alternative embodiment, the autonomous mobile device is implemented as a robot. As Figure 6 shown, the robot 600 in this embodiment includes: a mechanical body 601, on which there are one or more processors 602, one or more memories 603 for storing computer instructions, and a communication component 604. The communication component 604 may be a Wifi module, an infrared module, a Bluetooth module, etc.

[0090] In addition to one or more processors 602, a communication component 604, and one or more memories 603, the mechanical body 601 is also provided with some basic components of the robot 600, such as a vision sensor 606, a power supply component 607, a driving component 608, or an inertial navigation sensor 609, etc. The vision sensor can be a camera, a video camera, etc. The inertial navigation sensor can be an IMU or an odometer, etc. Optionally, the driving component 608 may include driving wheels, driving motors, omnidirectional wheels, etc. Optionally, if the robot 600 is a self-service robot in a bank, the robot 600 may further include a keyboard, a display screen, a document recognition component, a document scanning component, a touch sensing component, and a printing component, etc. If the robot 600 is a handling robot, the robot 600 may further include: a cargo tray, a mobile chassis, or a manipulator, etc. If the robot 600 is a sweeping robot, the robot 600 may further include a cleaning component, and the cleaning component may include a cleaning motor, a cleaning brush, a dust-raising brush, a suction fan, etc. These basic components included in different robots 600 and the compositions of the basic components will be different. The embodiments of the present application are only partial examples. It should be noted that Figure 6 The components shown in dashed boxes are all optional components, rather than essential components.

[0091] It should be noted that one or more processors 602 and one or more memories 603 may be disposed inside the mechanical body 601 or on the surface of the mechanical body 601. The mechanical body 601 is an execution mechanism by which the robot 600 completes its operation tasks and can perform the operations specified by the processor 602 in a determined environment. Among them, the mechanical body 601 to a certain extent reflects the appearance form of the robot 600. In this embodiment, the appearance form of the robot 600 is not limited. For example, it can be humanoid, circular, etc. Regarding the processor 602 executing the computer program stored in the memory 603, the same functions as those in the foregoing embodiments (such as Figure 5 the illustrated embodiments) can be achieved, which will not be elaborated here. Refer to the descriptions in the foregoing embodiments.

[0092] Correspondingly, the embodiments of the present application further provide a computer-readable storage medium storing a computer program, and when the computer program is executed, it can implement each step executable by the autonomous mobile device in the foregoing embodiments of the vision positioning method.

[0093] The foregoing Figure 5The communication component therein is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0094] The above Figure 5 The display therein includes a screen, and the screen can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations.

[0095] The above Figure 5 The power supply component therein provides power for various components of the device where the power supply component is located. The power supply component can include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power for the device where the power supply component is located.

[0096] The above Figure 5 The audio component therein can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signals can be further stored in a memory or sent via the communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

[0097] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0098] The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block of the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart Figure 1 for one or more flows and / or blocks Figure 1 of the block or blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in the flowchart Figure 1 for one or more flows and / or blocks Figure 1 of the block or blocks.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 for one or more flows and / or blocks Figure 1 of the block or blocks.

[0101] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0102] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0103] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0105] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A visual positioning method, applicable to an autonomous mobile device including a visual sensor and an inertial navigation sensor, characterized in that The method includes: Obtaining environmental images and inertial data collected by a vision sensor and an inertial navigation sensor respectively during the movement of an autonomous mobile device; Performing semantic segmentation on the environmental images collected by the vision sensor to obtain semantic labels of at least one object in the environmental images; Identifying dynamic objects in the environmental images and non-dynamic image regions outside the image regions where the dynamic objects are located according to the semantic labels of at least one object in the environmental images; when the current environmental image is a key frame, calculating the observation value and prediction value of the vision sensor, and the prediction value of the inertial navigation sensor according to the non-dynamic image regions included in the current key frame and at least one historical key frame, and the inertial data between the current key frame and at least one historical key frame; Jointly optimizing the observation value and prediction value of the vision sensor and the prediction value of the inertial navigation sensor on a manifold to obtain the pose of the autonomous mobile device in the environmental map.

2. The method according to claim 1, wherein Performing semantic segmentation on the environmental images collected by the vision sensor to obtain semantic labels of at least one object in the environmental images, including: For each frame of environmental image collected, identifying whether the environmental image is a key frame; if so, performing semantic segmentation on the environmental image to obtain semantic labels of at least one object in the environmental image.

3. The method according to claim 2, wherein It further includes: If the current environmental image is a normal frame image, adopting a tightly coupled method to jointly optimize the current environmental image, the previous environmental image and the inertial data between the two frames of environmental images to obtain the current pose of the autonomous mobile device.

4. The method according to claim 3, wherein Calculating the observation value and prediction value of the vision sensor, and the prediction value of the inertial navigation sensor according to the non-dynamic image regions included in the current key frame and at least one historical key frame, and the inertial data between the current key frame and at least one historical key frame, including: Calculating the observation value of the vision sensor according to the feature points in the current non-dynamic image region, and calculating the prediction value of the vision sensor in combination with the feature points in the historical non-dynamic image region; and Calculating the prediction value of the inertial navigation sensor according to the inertial data collected between the current key frame and the historical key frame; Wherein, the current non-dynamic image region and the historical non-dynamic image region respectively refer to the non-dynamic image regions in the current key frame and the historical key frame.

5. The method according to claim 4, wherein Calculating the observation value of the vision sensor according to the feature points in the current non-dynamic image region, including: Determining a candidate map region according to the semantic labels of the fixed objects included in the current non-dynamic image region and the semantic labels in the current existing environmental map; Matching the feature points in the current non-dynamic image region in the candidate map region to obtain the observation value of the vision sensor.

6. The method according to claim 5, wherein The current non-dynamic image region includes fixed objects and a background; matching the feature points in the current non-dynamic image region in the candidate map region to obtain the observation value of the vision sensor, including: Configuring weight coefficients for the feature points in the current non-dynamic image region, wherein the weight coefficients of the feature points on the fixed objects are greater than the weight coefficients of the background feature points; Match in the candidate map region according to the feature points and their weight coefficients in the current non-dynamic image region to obtain the observation value of the visual sensor.

7. The method according to claim 4, wherein Combine the feature points in the historical non-dynamic image region to calculate the predicted value of the visual sensor, including: For each historical non-dynamic image region, calculate the relative position change of the co-visible feature points existing between the current non-dynamic image region and the historical non-dynamic image region; Obtain a predicted value of the visual sensor according to the relative position change of the co-visible feature points and the pose of the autonomous mobile device at the corresponding historical moment.

8. The method according to claim 4, characterized in that, In the joint optimization process, it also includes: Obtain the mapping position of the feature points in the current non-dynamic image region in the environmental map; If there is no map point corresponding to the feature point in the environmental map, mark the map point at the mapping position in the environmental map to construct the environmental map in real time.

9. The method according to claim 8, wherein It also includes: If there is a map point corresponding to the feature point in the environmental map, but the position of the map point is different from the mapping position of the feature point in the environmental map, correct the map point according to the mapping position of the feature point in the environmental map.

10. The method according to any one of claims 1-9, characterized in that, After jointly positioning the autonomous mobile device, it also includes: Navigate or avoid obstacles for the autonomous mobile device according to the located pose of the autonomous mobile device.

11. The method according to any one of claims 1-9, characterized in that, The visual sensor is a monocular camera, and the method also includes: Select adjacent environmental images in which there are co-visible feature points in the non-dynamic image region and the position change of the co-visible feature points is greater than a set threshold from the environmental images collected by the visual sensor; Pre-integrate the inertial navigation data collected by the inertial navigation sensor between the adjacent environmental images to obtain the relative position change of the visual sensor when collecting the adjacent environmental images; Convert the relative position change from the inertial navigation coordinate system to the visual coordinate system according to the conversion relationship between the inertial navigation coordinate system and the visual coordinate system; Estimate the scale information of the environmental map according to the relative position change in the visual coordinate system to unify the coordinates of the environmental map with the actual physical world.

12. An autonomous mobile device, characterized in that, It includes: The device body, on which a memory, a processor, a visual sensor and an inertial navigation sensor are provided; The visual sensor is used to collect environmental images during the movement of the autonomous mobile device; The inertial navigation sensor is used to collect inertial navigation data during the movement of the autonomous mobile device; The memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program for: Perform semantic segmentation on the environmental image collected by the visual sensor to obtain semantic labels of at least one object in the environmental image; identify dynamic objects in the environmental image and non-dynamic image regions outside the image regions where the dynamic objects are located according to the semantic labels of at least one object in the environmental image; in the case where the current environmental image is a key frame, calculate the observation value and prediction value of the visual sensor, and the prediction value of the inertial sensor according to the non-dynamic image regions included in the current key frame and at least one historical key frame, and the inertial data between the current key frame and at least one historical key frame. Jointly optimize the observation value and prediction value of the visual sensor and the prediction value of the inertial sensor on the manifold to obtain the pose of the autonomous mobile device in the environmental map.

13. The autonomous mobile device according to claim 12, wherein When the processor obtains the semantic labels of at least one object, it is specifically used for: For each frame of environmental image collected, identify whether the environmental image is a key frame; if so, perform semantic segmentation on the environmental image to obtain semantic labels of at least one object in the environmental image.

14. The autonomous mobile device according to claim 13, characterized in that, The processor is further used for: If the current environmental image is a normal frame image, adopt a tightly coupled method to jointly optimize the current environmental image, the previous frame of environmental image, and the inertial data between the two frames of environmental images to obtain the current pose of the autonomous mobile device.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to implement the steps in the method according to any one of claims 1-11.

Images