Dynamic environment SLAM (Simultaneous Localization and Mapping) method and equipment based on instance segmentation

Abstract

The invention provides a dynamic environment SLAM method and device based on instance segmentation, and belongs to the technical field of computer vision. The method comprises the following steps: acquiring a first environment image from image acquisition equipment, dividing image grid blocks and determining corresponding gradient information; determining texture division areas corresponding to the first environment image based on the gradient information; and determining a geometric feature point set and a contour feature point set of the first environment image according to each texture division region and a preset bimodal feature extraction strategy. And generating a dynamic feature filtering mask of the first environment image when determining that the dynamic feature points exist in the first environment image based on the plurality of second environment images. And based on the geometric feature point set, the contour feature point set and the dynamic feature filtering mask, determining a matching point pair set corresponding to the first environment image, determining current pose information according to the matching point pair set, and outputting the current pose information to a downstream task. Through the method, robust positioning is realized in a dynamic and weak texture environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a dynamic environment SLAM method and device based on instance segmentation. BACKGROUND

[0002] In the field of computer vision, the Simultaneous Localization and Mapping (SLAM) technology is a core technology for realizing robot autonomous navigation, augmented reality and other applications.

[0003] However, in a dynamic and complex environment, the traditional SLAM is prone to mismatching due to interference of dynamic objects, which reduces the positioning accuracy, and it is difficult to extract geometric features in a weak texture area, and the matching success rate is low. In addition, most of the current SLAM methods are limited to single feature matching, and the use of image information is not sufficient, which further increases the difficulty of realizing robust positioning in dynamic and weak texture environments. SUMMARY

[0004] The embodiments of the present application provide a dynamic environment SLAM method and device based on instance segmentation, which is used to solve the technical problem of how to provide a robust positioning method in dynamic and weak texture environments.

[0005] In one aspect, the embodiments of the present application provide a dynamic environment SLAM method based on instance segmentation, which comprises:

[0006] obtaining a first environment image from an image acquisition device, dividing the image grid sub-blocks and determining the corresponding gradient information;

[0007] determining each texture division area corresponding to the first environment image based on each gradient information;

[0008] determining a geometric feature point set and an outline feature point set of the first environment image according to each texture division area and a preset dual-modal feature extraction strategy; the geometric feature point set is obtained based on extracting geometric features of a texture-rich area, and the outline feature point set is obtained based on instance segmentation of object outline features in a weak texture area;

[0009] generating a dynamic feature filtering mask of the first environment image when it is determined that there is a dynamic feature point in the first environment image based on a plurality of second environment images; wherein the plurality of second environment images and the first environment image constitute continuous image frames; the dynamic feature filtering mask is used to mark the spatial position of a dynamic object;

[0010] Based on the geometric feature point set, the contour feature point set and the dynamic feature filtering mask, a matching point pair set corresponding to the first environment image is determined, so as to determine current pose information according to the matching point pair set, and output the current pose information to a downstream task.

[0011] In an implementation manner of the present application, the image grid sub-blocks are divided and corresponding gradient information is determined, specifically including:

[0012] According to a preset grid size, the first environment image is divided into a plurality of image grid sub-blocks;

[0013] The gradient amplitudes corresponding to each image grid sub-block are calculated respectively;

[0014] According to each gradient amplitude and the number of pixels corresponding to each image grid sub-block, the gradient variances corresponding to each image grid sub-block are determined respectively, and the gradient variances and the sub-block labels of the corresponding image grid sub-blocks are added to the gradient information.

[0015] In an implementation manner of the present application, based on each gradient information, each texture division region corresponding to the first environment image is determined, specifically including:

[0016] Each gradient variance corresponding to each gradient information is compared with a preset texture evaluation threshold;

[0017] In the case where the gradient variance is greater than the preset texture evaluation threshold, the corresponding image grid sub-block is determined as a texture-rich region;

[0018] In the case where the gradient variance is greater than the preset texture evaluation threshold, the corresponding image grid sub-block is determined as a weak texture region;

[0019] According to the texture-rich region and the weak texture region, each texture division region is determined.

[0020] In an implementation manner of the present application, according to the texture-rich region and the weak texture region, each texture division region is determined, specifically including:

[0021] It is determined whether two adjacent image grid sub-blocks are texture regions of the same type; the type includes a texture-rich region and a weak texture region;

[0022] If yes, it is determined whether the difference between two gradient variances corresponding to the two adjacent image grid sub-blocks is less than a preset merging threshold;

[0023] In the case where the difference is less than the preset merging threshold, the two adjacent image grid sub-blocks are merged into the same texture division region.

[0024] In an implementation manner of the present application, according to the texture division area and the preset bimodal feature extraction strategy, the geometric feature point set and the contour feature point set of the first environment image are determined, specifically including:

[0025] In the case that the texture division area is the texture-rich area, the geometric feature points are extracted and the geometric feature point set is constructed;

[0026] In the case that the texture division area is the weak texture area, the object contour feature points corresponding to the texture division area are extracted through the instance segmentation, and the contour feature point set is constructed according to each object contour feature point; wherein the contour feature point set includes a plurality of contour feature points corresponding to one or more preset object categories.

[0027] In an implementation manner of the present application, it is determined that there is a dynamic feature point in the first environment image based on a plurality of second environment images, specifically including:

[0028] The plurality of second environment images and the first environment image form a continuous image frame; wherein the plurality of second environment images is at least 2;

[0029] According to the continuous image frame and the device pose information, it is determined whether there is a dynamic object in the continuous image frame;

[0030] If yes, the pixel point of the dynamic object in the first environment image is determined as the dynamic feature point.

[0031] In an implementation manner of the present application, based on the geometric feature point set, the contour feature point set and the dynamic feature filtering mask, a matching point pair set corresponding to the first environment image is determined, specifically including:

[0032] The geometric feature point set is matched with a first preset feature comparison set to determine a geometric matching point pair set according to a first matching result;

[0033] The contour feature point set is matched with a second preset feature comparison set to determine a contour matching point pair set according to a second matching result;

[0034] The geometric matching point pair set and the contour matching point pair set are fused, and the dynamic area matching point pair corresponding to the dynamic feature filtering mask is removed, to determine the matching point pair set corresponding to the first environment image.

[0035] In an implementation manner of the present application, the current pose information is determined according to the matching point pair set, specifically including:

[0036] A preset pose correlation matrix in a preset database is obtained;

[0037] The set of matching point pairs is matched with a preset pose association matrix to obtain the current pose information when the image acquisition device acquires the first environment image.

[0038] In an implementation form of the present application, the downstream task at least includes one or more of the following: AGV path planning, automatic cruise of a drone, scene three-dimensional reconstruction, and surgical navigation.

[0039] In another aspect, the present application also provides a dynamic environment SLAM device based on instance segmentation, comprising:

[0040] at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the instance segmentation-based dynamic environment SLAM method as described above.

[0041] Compared with the prior art, the present application has the following significant effects:

[0042] Through the above technical solution, the present application performs grid division on the first environment image, and obtains a texture division region according to gradient information, so as to perform feature extraction of geometric features and contour features respectively for different texture division regions, thereby establishing a geometric and semantic parallel processing architecture, which not only improves the information utilization rate of weak texture regions, but also combines geometric and contour features for feature matching, so as to fully utilize image information for efficient feature extraction. Moreover, the present application performs dynamic feature recognition and dynamic feature elimination, which effectively suppresses the interference of dynamic objects on positioning when performing the SLAM method. Further, the technical problem that the traditional SLAM is difficult to fully utilize image information for robust positioning in the presence of dynamic object interference and weak texture regions is solved. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate embodiments of the present application and its description, which serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0044] Figure 1 FIG. 1 is a flowchart of a dynamic environment SLAM method based on instance segmentation according to an embodiment of the present application;

[0045] Figure 2 FIG. 2 is a structural diagram of a dynamic environment SLAM device based on instance segmentation according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0047] In a dynamic complex environment, the traditional SLAM is prone to mismatching due to the interference of dynamic objects, which reduces the positioning accuracy, and it is difficult to extract geometric features in a weak texture area, and the matching success rate is low. In addition, most of the current SLAM methods are limited to single feature matching, and the use of image information is not sufficient, which further increases the difficulty of realizing robust positioning in dynamic and weak texture environments.

[0048] Based on this, the embodiments of the present application provide a dynamic environment SLAM method and device based on instance segmentation to solve the technical problem that the traditional SLAM cannot fully utilize image information for robust positioning in the presence of dynamic object interference and weak texture area.

[0049] The various embodiments of the present application will be described in detail below in combination with the drawings.

[0050] The embodiments of the present application provide a dynamic environment SLAM method based on instance segmentation, as shown in Figure 1 The method can include steps S101-S105:

[0051] S101, the microcontroller acquires a first environment image from the image acquisition device, divides the image grid sub-block and determines the corresponding gradient information.

[0052] The above microcontroller can be understood as a microcontroller (Microcontroller Unit, MCU) that is in communication connection with the image acquisition device and can execute the dynamic environment SLAM method based on instance segmentation, and can also be a central processing unit (Central Processing Unit, CPU), which can be arranged in devices such as automated guided vehicles (Automated Guided Vehicle, AGV), drones, mechanical hands, robots, etc. It should be noted that the microcontroller as the execution subject of the dynamic environment SLAM method based on instance segmentation is only exemplary, and the execution subject is not limited to the microcontroller, which is not specifically limited by the present application.

[0053] The image acquisition device can be understood as a camera, or a depth camera, which can be set according to user demand, and is not specifically limited here.

[0054] In the embodiments of the present application, the above dividing the image grid sub-blocks and determining the corresponding gradient information specifically includes:

[0055] According to the preset grid size, the first environment image is divided into a plurality of image grid sub-blocks. The gradient amplitudes corresponding to each image grid sub-block are calculated. According to the gradient amplitudes and the number of pixels corresponding to each image grid sub-block, the gradient variances corresponding to each image grid sub-block are determined, and the gradient variances and the sub-block labels of the corresponding image grid sub-blocks are added to the gradient information.

[0056] In other words, the present application is provided with a preset grid size for grid division of the image collected by the image acquisition device, so as to process the image in parallel and small range, and improve the image analysis accuracy. The preset grid size can be set by the user according to the actual use scene, or can be set by the developer based on the expert experience when the device is shipped, which is not limited here. The present application can calculate the gradient amplitudes of each image grid sub-block, and the gradient amplitude calculation formula is as follows:

[0057]

[0058] wherein G i represents the gradient amplitude corresponding to the i-th pixel unit of the image grid sub-block, represents the gradient in the x-axis direction, represents the gradient in the y-axis direction; the gradient can be understood as the degree of change in brightness, and the gradient amplitude can be calculated by Sobel operator or Prewitt operator, which is not limited here.

[0059] Then, according to the gradient amplitudes and the number of pixels of the image grid sub-block, the gradient variance of the image grid sub-block is calculated by the following gradient variance calculation formula, and the formula is specifically as follows:

[0060]

[0061] wherein T represents the gradient variance, N represents the number of pixels, and μG represents the mean of the gradient amplitudes corresponding to the image grid sub-block. The gradient variance of each image grid sub-block obtained by the above calculation carries the sub-block label of the image grid sub-block, and generates the gradient information. That is, the gradient information at least contains the gradient variances of each image grid sub-block.

[0062] S102, the microcontroller determines the texture division areas corresponding to the first environment image based on the gradient information.

[0063] In the embodiments of the present application, the texture division areas corresponding to the first environment image are determined based on the gradient information, specifically including:

[0064] The gradient variance corresponding to each gradient information is compared with a preset texture evaluation threshold. In a case where the gradient variance is greater than the preset texture evaluation threshold, the corresponding image grid sub-block is determined as a texture-rich region. In a case where the gradient variance is greater than the preset texture evaluation threshold, the corresponding image grid sub-block is determined as a weak texture region. According to the texture-rich region and the weak texture region, each texture division region is determined.

[0065] That is, the preset texture evaluation threshold can be set based on expert experience, which is used for dividing the image grid sub-blocks into texture regions according to the gradient variance. The specific value of the preset texture evaluation threshold can be set according to the actual use scene, which is not specifically limited here. If the gradient variance is greater than the preset texture evaluation threshold, it indicates that the texture of the corresponding image grid sub-block changes dramatically and is complex, which is regarded as a texture-rich region. If the gradient variance is less than the preset texture evaluation threshold, it indicates that the texture of the corresponding image grid sub-block is weak and it is not easy to obtain geometric features, which is regarded as a weak texture region. Through the division of the texture-rich region and the weak texture region, each image grid sub-block is divided into a texture region.

[0066] Through the texture region division, the feature extraction strategy can be flexibly switched for different texture regions in the subsequent process, and the features are extracted according to the texture strength, which makes up for the poor effect of geometric feature extraction in the weak texture region, and further utilizes the effective information of the weak texture region to improve the robustness and positioning accuracy of the scheme.

[0067] Further, in an embodiment of the present application, according to the texture-rich region and the weak texture region, each texture division region is determined, specifically including:

[0068] It is determined whether the two adjacent image grid sub-blocks are the same type of texture region. The types include a texture-rich region and a weak texture region. If it is determined that the two adjacent image grid sub-blocks are the same type of texture region, it is determined whether the difference between the two gradient variances corresponding to the two adjacent image grid sub-blocks is less than a preset merging threshold. In a case where the difference is less than the preset merging threshold, the two adjacent image grid sub-blocks are merged into the same texture division region.

[0069] In other words, after the texture region division is performed on the first environment image, some adjacent image grid sub-blocks can be of the same type of texture region, for example, both are texture-rich regions. If subsequent analysis is performed on each of the regions alone, effective information can be lost from the whole, which can negatively affect the subsequent positioning accuracy. For example, if the grid division method is used, the same object is in two image grid sub-blocks. If the texture division regions corresponding to the two image grid sub-blocks are analyzed alone, subsequent pose recognition can be achieved. However, if the two image grid sub-blocks can be spliced into an object for analysis, the semantic information can be deepened, that is, the deep information correlation between the image grid sub-blocks is established.

[0070] Therefore, in the technical solution, when the two adjacent image grid sub-blocks are of the same type of texture region and the difference between the gradient variances of the two image grid sub-blocks is less than a preset merging threshold, the two image grid sub-blocks are merged into the same texture division region. The preset merging threshold is set to ensure that the two image grid sub-blocks are more likely to be correlated. The preset merging threshold can be set based on expert experience or by a user according to an actual use scenario, and is not limited herein.

[0071] In S103, the microcontroller determines a geometric feature point set and a contour feature point set of the first environment image according to each texture division region and a preset bimodal feature extraction strategy.

[0072] The geometric feature point set is obtained based on the geometric features of the texture-rich region, and the contour feature point set is obtained based on the object contour features of the weak-texture region by instance segmentation.

[0073] In the embodiment of the application, the geometric feature point set and the contour feature point set of the first environment image are determined according to each texture division region and a preset bimodal feature extraction strategy, and specifically include:

[0074] In the case where the texture division region is a texture-rich region, the geometric feature points are extracted and a geometric feature point set is constructed. In the case where the texture division region is a weak-texture region, the object contour feature points corresponding to the texture division region are extracted by instance segmentation, and a contour feature point set is constructed according to each object contour feature point. The contour feature point set includes a plurality of contour feature points corresponding to one or more preset object categories.

[0075] Specifically, in the rich texture area, geometric feature points (edges, corners, etc.) can be extracted using an AKAZE algorithm, and a geometric feature point set can be constructed. In the weak texture area, object contour features can be extracted using instance segmentation, such as a YOLO algorithm, and the object contour features can be used as semantic constraints, such as the following preset object categories: signboards, garbage cans, etc., and then a contour feature point set corresponding to the object contour features is constructed. The preset object categories can be predefined, and one or more preset object categories can be set by a user or a developer according to an actual use scenario, so as to correspond the contour features to the object categories and identify the object contour feature points.

[0076] By using geometric feature extraction in the rich texture area and using contour feature extraction in the weak texture area, a hybrid descriptor about the first environment image can be constructed, thereby realizing semantic-geometric feature complementation and ensuring the stability and accuracy of feature extraction in a complex environment.

[0077] The AKAZE algorithm and the YOLO algorithm are only exemplary and other algorithms can also be used in actual use, which is not limited in the present application. In addition, when identifying the object contour feature points by using instance segmentation, a fusion mode of local feature matching and global feature matching can be used. When the target has no category, the similarity of the features is calculated by using a Hamming distance; then, the robustness of the matching is improved by using a ratio of Lowe; when the target belongs to a predefined preset object category, the depth can be estimated by using information of the target between adjacent frames, or the depth can be estimated by using information of the feature points of the target; the pose between the adjacent frames can be tracked; and the global feature matching is performed by using a bag-of-words method, and whether each feature point belongs to the same target is considered when matching.

[0078] In S104, the microcontroller generates a dynamic feature filtering mask of the first environment image when it is determined that there is a dynamic feature point in the first environment image based on the plurality of second environment images.

[0079] The plurality of second environment images and the first environment image constitute continuous image frames. The dynamic feature filtering mask is used to mark the spatial position of a dynamic object.

[0080] In the embodiments of the present application, the determination of the dynamic feature point in the first environment image based on the plurality of second environment images specifically includes:

[0081] The plurality of second environment images and the first environment image constitute a continuous image frame. Wherein, the plurality of second environment images is at least 2. According to the continuous image frame and the device pose information, it is determined whether there is a dynamic object in the continuous image frame. In the case of determining that there is a dynamic object in the continuous image frame, the pixel point of the dynamic object in the first environment image is determined as a dynamic feature point.

[0082] In other words, the application can collect continuous image frames to identify dynamic objects. The first environment image of the application is at least the third image after starting the image acquisition task. The position change of the device is determined by the continuous image frame and the device pose information, the motion compensation of the image acquisition device is performed, the dynamic object existing in the continuous image frame is detected, and the dynamic object in the first environment image is identified. The dynamic object identification can be obtained by the displacement variance of the pixel points corresponding to the object, or by cross-frame optical flow consistency analysis, which is not limited in the application. The pixel point is a pixel unit.

[0083] For example, the position of the feature point with the same ID is tracked in the continuous image frame, the average of the position change is calculated, the displacement variance is calculated, and whether the feature point is a dynamic feature point is judged according to the comparison between the displacement variance and the first dynamic recognition threshold, so as to identify the dynamic object. The first dynamic recognition threshold can be set by the R&D personnel or the user based on expert experience according to the actual use scene, which is not limited here. For another example, a sparse optical flow point set is extracted in the first environment image, the position of the optical flow point is detected and predicted according to the sparse optical flow point set and the plurality of second environment images, so as to calculate the inconsistency rate of the detected position and the predicted position, and the inconsistency rate is compared with the second dynamic recognition threshold. If the inconsistency rate is greater than the second dynamic recognition threshold, the corresponding optical flow point is marked as a dynamic feature point. The second dynamic recognition threshold can be set by the R&D personnel or the user based on expert experience according to the actual use scene, which is not limited here.

[0084] After obtaining the dynamic feature point, the microcontroller can mark the part of the dynamic feature point as the dynamic object feature point in the first environment image, and generate a mask to mark the spatial position of the dynamic object in the first environment image as a dynamic feature filtering mask.

[0085] In the embodiment of the application, the matching point pair set corresponding to the first environment image is determined based on the geometric feature point set, the contour feature point set and the dynamic feature filtering mask, and the current pose information is determined according to the matching point pair set, and the current pose information is output to the downstream task.

[0086] In the embodiment of the application, the matching point pair set corresponding to the first environment image is determined based on the geometric feature point set, the contour feature point set and the dynamic feature filtering mask, and the current pose information is determined according to the matching point pair set, and the current pose information is output to the downstream task.

[0087] The geometric feature point set is matched with the first preset feature comparison set to determine a geometric matching point pair set according to a first matching result. The contour feature point set is matched with the second preset feature comparison set to determine a contour matching point pair set according to a second matching result. The geometric matching point pair set and the contour matching point pair set are fused, and a dynamic region matching point pair corresponding to the dynamic feature filtering mask is removed, to determine a matching point pair set corresponding to the first environment image.

[0088] That is, the present application is provided with the first preset feature comparison set and the second preset feature comparison set. The first preset feature comparison set can be a geometric feature point set of a comparison image at a corresponding position of the first environment image, which is determined according to an electronic map and a preset image comparison set after the position information of the first environment image is determined. The preset image comparison set can be maintained by a user periodically, which is not specifically limited herein. The preset image comparison set can be understood as an image set at a historical time before the first environment image is collected. Similarly, the second preset feature comparison set can be understood as a contour feature point set of an object extracted from the comparison image.

[0089] The geometric feature point set and the contour feature point set are matched with the first preset feature comparison set and the second preset feature comparison set respectively, to obtain the geometric matching point pair set and the contour matching point pair set. The matching can be performed by calculating the cosine similarity and the Euclidean distance between the feature points, for example, if the cosine similarity between two feature points is greater than a first predetermined value, it is indicated that the two feature points are matched, and a matching point pair is obtained. For another example, if the Euclidean distance between two feature points is less than a second predetermined value, it is indicated that the two feature points are matched, and a matching point pair is obtained. The specific matching mode and how to set the first predetermined value or the second predetermined value can be set by a user or an expert based on an actual scene, which is not specifically limited herein.

[0090] After the geometric matching point pair set and the contour matching point pair set are obtained, the microcontroller can further perform a set fusion operation, to determine which pixel feature points in the first environment image are matched with the comparison image. Meanwhile, the feature points of a dynamic object existing in the first environment image are removed by using the dynamic feature filtering mask, that is, a dynamic region matching point pair corresponding to the dynamic feature filtering mask is removed from the initial matching point pair set obtained through the fusion, to obtain a matching point pair set in which dynamic objects are filtered.

[0091] Through the above scheme, the influence of the dynamic object on the mismatching of the SLAM method can be removed, and the positioning accuracy is improved.

[0092] Further, in the embodiment of the present application, the current pose information is determined according to the matching point pair set, specifically including:

[0093] obtain a preset pose correlation matrix in a preset database, and match the set of matched point pairs with the preset pose correlation matrix to obtain current pose information of the image acquisition device when collecting the first environment image.

[0094] That is, the set of matched point pairs can be used to determine the pose of the image acquisition device, wherein the preset pose correlation matrix can contain pose data of the comparison image, and the relative position and attitude information between the first environment image and the comparison image can be matched by the set of matched point pairs, so as to finally obtain an optimized camera pose matrix, including a rotation matrix and a translation vector. Subsequently, the pose information can also be converted into a quaternion and an Euclidean coordinate and output to a downstream task (such as navigation, reconstruction, etc.).

[0095] Specifically, the downstream task at least includes one or more of the following: AGV path planning, unmanned aerial vehicle automatic cruising, scene three-dimensional reconstruction, and surgical navigation. According to the application device of the dynamic environment SLAM method based on instance segmentation, there can be different downstream tasks, which are not specifically limited by the application.

[0096] Through the above technical solution, the first environment image is divided into grids, and the texture division region is obtained according to the gradient information, so as to perform feature extraction of geometric features and contour features on different texture division regions respectively, thereby establishing a geometric and semantic parallel processing architecture, which improves the information utilization rate of weak texture regions, and combines geometric and contour features for feature matching, so as to fully utilize image information for efficient feature extraction. Moreover, the application performs dynamic feature recognition and dynamic feature elimination, which effectively suppresses the interference of dynamic objects on positioning when performing the SLAM method. Further, the technical problem that the traditional SLAM is difficult to fully utilize image information for robust positioning in the presence of dynamic object interference and weak texture regions is solved.

[0097] Figure 2 A structural schematic diagram of a dynamic environment SLAM device based on instance segmentation provided by an embodiment of the application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the device includes:

[0098] at least one processor; and a memory connected in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0099] The first environment image from the image acquisition device is acquired, the image grid sub-blocks are divided, and the corresponding gradient information is determined. Based on the gradient information, the texture division area corresponding to the first environment image is determined. According to the texture division area and the preset bimodal feature extraction strategy, the geometric feature point set and the contour feature point set of the first environment image are determined. The geometric feature point set is obtained based on extracting the geometric features of the texture-rich area, and the contour feature point set is obtained based on instance segmentation of the object contour features of the weak texture area. When it is determined that there is a dynamic feature point in the first environment image based on a plurality of second environment images, a dynamic feature filtering mask of the first environment image is generated. The plurality of second environment images and the first environment image constitute a continuous image frame. The dynamic feature filtering mask is used to mark the spatial position of the dynamic object. Based on the geometric feature point set, the contour feature point set and the dynamic feature filtering mask, a matching point pair set corresponding to the first environment image is determined, so as to determine the current pose information according to the matching point pair set, and output the current pose information to the downstream task.

[0100] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0101] The device and the method provided by the embodiments of the present application are one-to-one correspondence, therefore, the device also has the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device will not be described here.

[0102] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or other elements inherent in such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0103] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A dynamic environment SLAM method based on instance segmentation, characterized in that, The method comprises: obtaining a first environment image from an image acquisition device, dividing an image grid sub-block and determining corresponding gradient information; based on each of the gradient information, determining each texture division area corresponding to the first environment image; According to each of the texture division area and the preset double modal feature extraction strategy, the geometric feature point set and the contour feature point set of the first environment image are determined; The geometric feature point set is obtained based on the extraction of the geometric feature of the texture rich area, and the contour feature point set is obtained based on the object contour feature of the instance segmentation weak texture area; When it is determined that there is a dynamic feature point in the first environment image based on a plurality of second environment images, a dynamic feature filtering mask of the first environment image is generated; wherein the plurality of second environment images and the first environment image constitute a continuous image frame; The dynamic feature filtering mask is used to mark the spatial position of the dynamic object; Based on the geometric feature point set, the contour feature point set and the dynamic feature filtering mask, a matching point pair set corresponding to the first environment image is determined, so as to determine the current pose information according to the matching point pair set, and output the current pose information to the downstream task.

2. The dynamic environment SLAM method based on instance segmentation according to claim 1, characterized in that, Divide the image grid sub-block and determine the corresponding gradient information, specifically including: According to the preset grid size, the first environment image is divided into a plurality of image grid sub-blocks; Calculate the gradient amplitude corresponding to each image grid sub-block respectively; According to each of the gradient amplitude and the pixel number corresponding to each of the image grid sub-block, the gradient variance corresponding to each of the image grid sub-block is determined, and the gradient variance and the sub-block label of the corresponding image grid sub-block are added to the gradient information.

3. The dynamic environment SLAM method based on instance segmentation according to claim 2, characterized in that, Based on each of the gradient information, each texture division area corresponding to the first environment image is determined, specifically including: Compare each of the gradient variances corresponding to each of the gradient information with a preset texture evaluation threshold value; In the case where the gradient variance is greater than the preset texture evaluation threshold value, the corresponding image grid sub-block is determined as a texture rich area; In the case where the gradient variance is greater than the preset texture evaluation threshold value, the corresponding image grid sub-block is determined as a weak texture area; According to the texture rich area and the weak texture area, each of the texture division areas is determined.

4. The dynamic environment SLAM method based on instance segmentation according to claim 3, characterized in that, According to the texture rich area and the weak texture area, each of the texture division areas is determined, specifically including: Determine whether two adjacent image grid sub-blocks are the same type of texture area; The type includes a texture rich area and a weak texture area; If yes, determine whether the difference between the two gradient variances corresponding to the two adjacent image grid sub-blocks is less than a preset merging threshold value; In the case where the difference is less than the preset merging threshold value, the two adjacent image grid sub-blocks are merged into the same texture division area.

5. The dynamic environment SLAM method based on instance segmentation according to claim 1 or 3, characterized in that, According to each of the texture division area and the preset double modal feature extraction strategy, the geometric feature point set and the contour feature point set of the first environment image are determined, specifically including: In the case where the texture division area is the texture rich area, the geometric feature point is extracted and the geometric feature point set is constructed; In a case where the texture division region is the weak texture region, object contour feature points corresponding to the texture division region are extracted by the instance segmentation, and a contour feature point set is constructed according to each object contour feature point; the contour feature point set includes a plurality of contour feature points corresponding to one or more preset object categories.

6. The instance segmentation-based dynamic environment SLAM method of claim 1, wherein, The determination that the dynamic feature point exists in the first environment image based on a plurality of second environment images specifically includes: The plurality of second environment images and the first environment image are combined into a continuous image frame; the plurality of second environment images are at least two. Whether a dynamic object exists in the continuous image frame is determined according to the continuous image frame and device pose information. If yes, a pixel point of the dynamic object in the first environment image is determined as the dynamic feature point.

7. The instance segmentation-based dynamic environment SLAM method of claim 1, wherein, Based on the geometric feature point set, the contour feature point set and the dynamic feature point filtering mask, a matching point pair set corresponding to the first environment image is determined, specifically including: The geometric feature point set is matched with a first preset feature comparison set to determine a geometric matching point pair set according to a first matching result. The contour feature point set is matched with a second preset feature comparison set to determine a contour matching point pair set according to a second matching result. The geometric matching point pair set and the contour matching point pair set are fused, and a dynamic region matching point pair corresponding to the dynamic feature filtering mask is removed, to determine the matching point pair set corresponding to the first environment image.

8. The instance segmentation-based dynamic environment SLAM method of claim 1, wherein, The current pose information is determined according to the matching point pair set, specifically including: A preset pose correlation matrix in a preset database is obtained. The matching point pair set and the preset pose correlation matrix are matched to obtain the current pose information when the image acquisition device acquires the first environment image.

9. The instance segmentation-based dynamic environment SLAM method of claim 1, wherein, The downstream task at least includes one or more of the following: AGV path planning, unmanned aerial vehicle automatic cruising, scene three-dimensional reconstruction, surgical navigation.

10. A dynamic environment SLAM device based on instance segmentation, characterized in that, The device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a dynamic environment SLAM method based on instance segmentation according to any one of claims 1-9.

Images