Dynamic environment positioning and mapping method based on binocular vision and related device

Abstract

The invention discloses a binocular vision-based dynamic environment positioning and mapping method and a related device, and relates to the technical field of robot environment perception, and the method comprises the following steps: determining a key frame in a left eye image set based on an ORB feature matching algorithm, determining a corresponding right eye image based on a timestamp, and determining a target image based on the key frame and the right eye image; determining the spatial position of each feature point by adopting a triangulation method; clustering each feature point, and determining each object and a spatial position thereof; and determining depth change based on spatial positions of the object in the key frame and a plurality of continuous frames before the key frame, further determining a final dynamic attribute, removing corresponding feature points when the spatial positions are greater than a dynamic attribute threshold value, and finally performing global mapping. Compared with a traditional SLAM method in a dynamic environment, the scheme of the invention considers the situation of evaluating the dynamic attribute of the target object in real time instead of assuming that a movable object is a dynamic object, and is more suitable for a wide practical application environment.

Description

Technical Field

[0001] The present invention relates to the field of robot environment perception, and in particular to a dynamic environment positioning and mapping method based on binocular vision and related devices. Background Art

[0002] Robots with environmental adaptability and motion robustness have broad application prospects in the real world. The demand for robots in unknown environment detection, autonomous task execution, adaptive trajectory planning, etc. is increasing, and environmental perception is an important basis for the practical application of robots. Among them, simultaneous localization and mapping (SLAM) is becoming more and more important as the first step in exploring the environment.

[0003] In the past few years, some research institutions have combined perception methods including vision and lidar technology with SLAM frameworks to improve the perception of complex environments, thereby achieving precise positioning and high-fidelity mapping of unknown environments. The lidar-based SLAM framework can capture simple distance information and create point clouds. Due to the advanced hardware settings, it has the characteristics of high efficiency, strong versatility, and rich structural information. However, most lidar SLAMs usually require relatively large equipment and a large amount of sensor information, which in turn incurs inevitable costs in terms of acquisition and processing power consumption. These problems are particularly unacceptable in situations where resources are limited, such as when using small robots or performing long-term missions.

[0004] To address the SLAM problem in small spaces and the fact that small robots cannot carry large sensing equipment, some methods based on deep reinforcement learning have been proposed and combined with feature extraction in visual SLAM. However, most of the work incorporates static environment assumptions into the SLAM process without considering the impact of erroneous data due to the presence of dynamic moving objects. As a result, these frameworks perform well in ideal and limited scenarios, but will experience significant positioning errors or failures when there are moving objects in real scenarios. Therefore, how to accurately locate and map in challenging dynamic environments has become a difficult problem that needs to be solved in this field. Summary of the invention

[0005] The purpose of the present invention is to provide a dynamic environment positioning and mapping method and related devices based on binocular vision, which can realize high-precision positioning and mapping of robots in actual dynamic environments.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] On the one hand, the present invention provides a method for positioning and mapping a dynamic environment based on binocular vision, comprising the following steps:

[0008] A binocular video image set is obtained; the binocular video image set includes a left-eye image set and a right-eye image set; timestamps of several left-eye images in the left-eye image set correspond to timestamps of several right-eye images in the right-eye image set.

[0009] Based on the ORB feature matching algorithm, multiple key frames are determined in the left-eye image set; the key frame is the left-eye image with the later timestamp among the two left-eye images after ORB feature matching is performed on two adjacent left-eye images, the number of feature matching points is less than the feature matching threshold.

[0010] For any key frame, the corresponding right eye image is determined in the right eye image set based on the timestamp of the key frame, and the spatial position of each feature point is determined based on the key frame and the right eye image using the triangulation method to obtain the local mapping result of the key frame.

[0011] Based on the initial dynamic attributes of each pixel in the key frame, the feature points in the local mapping results are clustered to determine multiple objects and the spatial position of each object; the initial dynamic attributes are the attribute values ​​assigned to each pixel after target recognition and image segmentation are performed on each left-eye image and each right-eye image through the target recognition and image segmentation model.

[0012] For any object, a depth change data set of the object is determined based on the spatial position of the object in a key frame and the spatial position of the object in a number of consecutive image frames before the key frame; the depth change data set includes a number of depth change data.

[0013] Based on the depth change dataset of the object and the initial dynamic properties of the object, the final dynamic properties of the object are determined. When the final dynamic properties of the object are greater than the dynamic property threshold, the feature points corresponding to the object are removed from the local mapping result of the key frame to obtain the local mapping update result of the key frame.

[0014] According to the local mapping update results of each key frame, global mapping is performed to obtain the global mapping result.

[0015] Optionally, the target recognition and image segmentation model adopts the yolov8_seg_onnx target detection model.

[0016] Optionally, based on the initial dynamic properties of each pixel in the key frame, each feature point in the local mapping result is clustered to determine multiple objects and the spatial position of each object, specifically including the following steps:

[0017] Based on the initial dynamic properties of each pixel in the key frame, the feature points in the local mapping result are clustered to determine multiple objects.

[0018] For any feature point of any object, the modulus of the feature point is determined according to the spatial position of the feature point.

[0019] Based on the modulus lengths of the feature points of the object, the average modulus length of the object is calculated, and the average modulus length of the object is used as the spatial position of the object.

[0020] Optionally, the modulus of the feature point is determined according to the following formula:

[0021]

[0022] Among them, ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point.

[0023] The average modulus length of the object is calculated according to the following formula:

[0024]

[0025] in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

[0026] Optionally, determining a depth change dataset of the object based on the spatial position of the object in the key frame and the spatial position of the object in a number of consecutive image frames before the key frame specifically includes the following steps:

[0027] For any image frame before the key frame, the depth change distance of the image frame is determined based on the spatial position of the object in the image frame and the spatial position of the object in the image frame before the key frame.

[0028] Depth change data of the image frame is determined based on the depth change distance and the depth change threshold of the image frame.

[0029] Based on the depth change data of each image frame, a depth change data set of the object is determined.

[0030] Optionally, a triangulation method is used to determine the spatial position of each feature point based on the key frame and the right eye image, specifically including the following steps:

[0031] For any pixel point at the same position of the key frame and the right eye image, two intersecting straight lines are determined based on the camera optical center of the key frame and the camera optical center of the right eye image.

[0032] The intersection point of the two intersecting straight lines is taken as a feature point, and the spatial position of the intersection point is taken as the spatial position of the feature point.

[0033] Optionally, the final dynamic properties of the object are determined according to the following formula:

[0034]

[0035] Among them, V is the final dynamic attribute of the object, v0 is the initial dynamic attribute of the object, and v i is the i-th depth change data in the depth change data set of the object, and n is the size of the depth change data set of the object.

[0036] On the other hand, the present invention provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-mentioned methods for dynamic environment positioning and mapping based on binocular vision.

[0037] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for dynamic environment positioning and mapping based on binocular vision.

[0038] On the other hand, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for dynamic environment positioning and mapping based on binocular vision.

[0039] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The present invention provides a method and related device for positioning and mapping a dynamic environment based on binocular vision. The method comprises the following steps: obtaining a left-eye image set and a right-eye image set with corresponding timestamps, determining a plurality of key frames in the left-eye image set based on an ORB feature matching algorithm; for any key frame, determining the corresponding right-eye image in the right-eye image set based on the timestamp of the key frame, and using a triangulation method to determine the spatial position of each feature point based on the key frame and the right-eye image, and obtaining a local mapping result of the key frame; then performing local mapping on each feature point based on the initial dynamic attributes of each pixel point. Clustering is performed to determine the spatial positions of multiple objects and each object; for any of the objects, the depth change data set of the object is determined based on the spatial position of the object in the key frame and the spatial position of the object in several consecutive image frames before the key frame; based on the depth change data set of the object and the initial dynamic properties of the object, the final dynamic properties of the object are determined, and when the final dynamic properties of the object are greater than the dynamic property threshold, the feature points corresponding to the object are removed from the local mapping results of the key frame to obtain the local mapping update results of the key frame; according to the local mapping update results of each key frame, global mapping is performed to obtain the global mapping result. The present invention adopts a method of integrating target detection and segmentation with visual SLAM. Compared with the traditional SLAM method in a dynamic environment, the scheme of the present invention considers the situation of real-time evaluation of the dynamic properties of the target object, rather than assuming that the movable object is a dynamic object, that is, assuming that a person is a dynamic object and a water cup is a static object. It is more suitable for a wide range of practical application environments, and the scheme designed by the present invention has a small amount of calculation, low computing power requirements, and a light load on the robot's computing platform, which has the significance of improving the accuracy of the robot's positioning and mapping in an actual dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flowchart of a dynamic environment positioning and mapping method based on binocular vision provided in Example 1 of the present invention;

[0043] Figure 2 A specific flow chart of step A4 in a dynamic environment positioning and mapping method based on binocular vision provided in Example 1 of the present invention;

[0044] Figure 3 A specific flow chart of step A5 in a dynamic environment positioning and mapping method based on binocular vision provided in Example 1 of the present invention;

[0045] Figure 4 A specific flow chart of step A6 in a dynamic environment positioning and mapping method based on binocular vision provided in Example 1 of the present invention;

[0046] Figure 5 A schematic diagram of a triangulation method in a dynamic environment positioning and mapping method based on binocular vision provided in Example 1 of the present invention;

[0047] Figure 6 This is a diagram of the internal structure of a computer device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The purpose of the present invention is to provide a dynamic environment positioning and mapping method and related devices based on binocular vision, aiming to achieve accurate positioning and mapping in challenging dynamic environments.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] like Figure 1 As shown in the flowchart, a dynamic environment positioning and mapping method based on binocular vision in this embodiment includes the following steps:

[0053] A1. Obtain a binocular video image set; the binocular video image set includes a left-eye image set and a right-eye image set; timestamps of several left-eye images in the left-eye image set correspond to timestamps of several right-eye images in the right-eye image set.

[0054] A2. Based on the ORB feature matching algorithm, multiple key frames are determined in the left-eye image set; the key frame is the left-eye image with the later timestamp among the two left-eye images after ORB feature matching is performed on two adjacent left-eye images, and the number of feature matching points is less than the feature matching threshold.

[0055] A3. For any key frame, based on the timestamp of the key frame, determine the corresponding right-eye image in the right-eye image set.

[0056] A4. Using triangulation method, based on key frame and right eye image, determine the spatial position of each feature point and obtain the local mapping result of key frame. Figure 2 In the flowchart shown, step A4 specifically includes the following steps:

[0057] A41. For any pixel point at the same position of the key frame and the right eye image, two intersecting straight lines are determined based on the camera optical center of the key frame and the camera optical center of the right eye image.

[0058] A42. The intersection point of two intersecting straight lines is taken as a feature point, and the spatial position of the intersection point is taken as the spatial position of the feature point.

[0059] A5. Based on the initial dynamic attributes of each pixel in the key frame, cluster each feature point in the local mapping result to determine multiple objects and the spatial position of each object; the initial dynamic attribute is the attribute value assigned to each pixel after performing target recognition and image segmentation on each left eye image and each right eye image through the target recognition and image segmentation model. Specifically, in this embodiment, the target recognition and image segmentation model adopts the yolov8_seg_onnx target detection model.

[0060] like Figure 3 In the flowchart shown, step A5 specifically includes the following steps:

[0061] Step A51: Based on the initial dynamic attributes of each pixel in the key frame, cluster each feature point in the local mapping result to determine multiple objects.

[0062] Step A52: for any feature point of any object, determine the modulus of the feature point according to the spatial position of the feature point.

[0063] Determine the modulus length of the feature point according to formula (1):

[0064]

[0065] Among them, ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point.

[0066] Step A53: Based on the modulus lengths of the feature points of the object, the average modulus length of the object is calculated, and the average modulus length of the object is used as the spatial position of the object.

[0067] According to formula (2), the average modulus length of the object is calculated:

[0068]

[0069] in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

[0070] A6. For any object, based on the spatial position of the object in the key frame and the spatial position of the object in a number of consecutive image frames before the key frame, determine the depth change data set of the object; the depth change data set includes a number of depth change data. Figure 4 In the flowchart shown, step A6 specifically includes the following steps:

[0071] A61. For any image frame before the key frame, determine a depth change distance of the image frame based on a spatial position of the object in the image frame and a spatial position of the object in an image frame before the image frame.

[0072] A62. Determine depth change data of the image frame based on the depth change distance and the depth change threshold of the image frame.

[0073] A63. Determine a depth change data set of the object based on the depth change data of each image frame.

[0074] A7. Based on the depth change data set of the object and the initial dynamic properties of the object, determine the final dynamic properties of the object. Determine the final dynamic properties of the object according to formula (3):

[0075]

[0076] Among them, V is the final dynamic attribute of the object, v0 is the initial dynamic attribute of the object, and v i is the i-th depth change data in the depth change data set of the object, and n is the size of the depth change data set of the object.

[0077] A8. When the final dynamic attribute of the object is greater than the dynamic attribute threshold, the feature points corresponding to the object are removed from the local mapping result of the key frame to obtain the local mapping update result of the key frame.

[0078] A9. Perform global mapping based on the local mapping update results of each key frame to obtain a global mapping result.

[0079] The above content gradually introduces in detail the steps of a dynamic environment positioning and mapping method based on binocular vision provided in this embodiment. The steps of a dynamic environment positioning and mapping method based on binocular vision provided in this embodiment are described in a positive order below, which mainly includes the following steps:

[0080] The first step is to extract image frames from the binocular video and put them into the image buffer. The image frames entering the image buffer are subjected to target recognition and image segmentation through the yolov8 target detection and segmentation model. According to the name of the recognized object, the initial dynamic attributes of the pixel points of the object and its image frame are assigned.

[0081] The ideas of target recognition and image segmentation are as follows:

[0082] 1) First, generate multiple regions of interest (ROI) where the target to be detected may appear for the image frame entering the image buffer. Because a picture contains a lot of background information, a feasible method is to first generate the approximate area where the target may exist, and then fine-tune these regions of interest.

[0083] 2) For the region of interest (ROI), features are extracted through a convolutional neural network for subsequent classification and regression.

[0084] 3) After completing the first two steps, a series of candidate boxes containing the target will be obtained, but these boxes have a lot of overlap (that is, for the same target, there may be multiple boxes to frame it), so they need to be screened and reorganized, and the non-maximum suppression method (NMS) is applied so that each target is framed by a single box.

[0085] 4) At this point, the target framed by a single rectangular box has been obtained. Next, the convolutional neural network is used to extract the features of the image in the box and put it into the segmentation model trained by yolov8. With the goal of minimizing the intersection-over-union loss, multiple iterative calculations are performed, and finally the image can be segmented according to its contour.

[0086] The specific process of target recognition and image segmentation is as follows:

[0087] 1) Generate region of interest ROI:

[0088] To generate ROI, the most direct way is to scan line by line, such as using a sliding window, scanning the image with windows of different aspect ratios and different proportions relative to the original image size, but most of the area in an image is the background, so this method is very time-consuming. Therefore, selective search is used in this example. Its idea is based on the relationship between pixels, such as color, texture, size, etc., and clustering is performed based on this information to generate ROI.

[0089] 2) Feature extraction:

[0090] The purpose of feature extraction is to find a representative representation of the region of interest, and further use it for tasks such as classification and regression. In this embodiment, the existing technology is used, that is, the fastest and most stable feature point detection and extraction algorithm, the ORB feature point detection algorithm (Oriented FAST and Rotated BRIEF is the abbreviation of Oriented FAST+RotatedBRIEF).

[0091] 3) Non-maximum suppression:

[0092] Non-maximum suppression still uses existing technology. Its idea is to reduce the number of predicted candidate boxes to the number of real targets in the picture, that is, the same target does not want to be framed by multiple boxes.

[0093] The specific steps of initial dynamic attribute "assignment" are:

[0094] For the 80 common objects that can be detected by the Yolov8 target detection and segmentation model, an initial value of the dynamic attribute is added to these 80 objects, ranging from 0 to 100. For example, the initial dynamic attribute assigned to the "table" class is 20, and the initial dynamic attribute assigned to the "person" class is 80. The greater the mobility of the object, the higher the initial dynamic attribute value.

[0095] The second step is to read the left eye image from the image buffer, and only extract the ORB features of the left eye image. Through the calculation of FAST corner detection and BRIEF descriptor, the image corners with scaling invariance and rotation invariance are detected, so as to perform ORB feature matching on adjacent key frames. The purpose is to calculate the camera movement of the current frame relative to the previous frame, that is, to set the initial image frame as the pose origin, and to estimate the pose of the image frame that matches the ORB features through the triangulation method, and calculate the pose of the current frame. Here, triangulation is to calculate the pose relationship between different frames through the left eye images of different frames.

[0096] Triangulation is the process of determining the distance of a point by observing the angle between two points. Figure 5 Take the example shown below: Assume that the adjacent keyframes are I1 and I2, and the optical centers of the camera at these two positions are O1 and O2. With the left picture as a reference, the transformation matrix of the right picture is T (including the rotation matrix R and the translation matrix t). Assume that there is a feature point p1 in I1, which corresponds to the feature point p2 in I2. At this time, due to the position invariance of the three-dimensional points, the rays O1p1 and O2p2 will intersect and intersect at point P, which is the position of the map points corresponding to the two feature points in the three-dimensional scene. However, due to the influence of noise, these two straight lines often cannot intersect. Therefore, the least squares method can be used to solve the point with the shortest distance as the intersection point.

[0097] According to the definition in epipolar geometry, if x1 and x2 are the normalized coordinates of two feature points (i.e., x1 = [u1, v1, 1], x2 = [u2, v2, 1], u and v are the width and height positions of the description point in the phase plane, respectively), then according to the definition of epipolar geometry, we have the relationship shown in formula (4):

[0098] s1x1=s2Rx2+t (4).

[0099] In formula (4), x1 and x2 are known quantities, and R and t are required quantities. Here, the initial values ​​are randomly given, and R and t are considered to be known quantities. Their least squares solutions are then found through iteration. Now what is required is the depths s1 and s2 of the two feature points, which can be found separately. First, s2 is found.

[0100] Multiply both sides of equation (4) by The ^ is the antisymmetric symbol of the vector, the purpose is to convert a vector into an antisymmetric matrix In this way, the vector outer product is written as the multiplication of the matrix and the vector, and the result is still a vector, which is perpendicular to the two vectors, so The size is 0.

[0101] So we can get formula (5):

[0102]

[0103] The left side of this equation is zero, and the right side can be seen as an equation of s2, from which s2 can be directly obtained. With s2, s1 is also very easy to obtain. Thus, the depth of the points in the two frames is obtained, and their spatial coordinates are determined. Due to the presence of noise, the estimated R and t may not accurately make the above equation zero, so the common practice is to find the least squares solution. Through the idea of ​​iteration in the least squares solution, the pose change matrix R and t of the latter frame relative to the previous frame are made variables, and the intersection of ray O1p1 and ray O2p2 at one point is taken as the goal. After multiple iterations, the pose change matrix R and t that finally meet the threshold requirements are obtained.

[0104] In the third step, the number of ORB feature matches in the previous step is calculated and compared with the matching threshold. If the number of matches is less than the matching threshold, the current frame is set as the key frame, and the target recognition and segmentation results of the image and the initial dynamic attributes corresponding to the segmented image are found from the results calculated in the first step.

[0105] In the fourth step, the key frame with recognition, segmentation and dynamic attributes is used to find the corresponding right eye image in the image buffer through timestamp consistency. In the previous step, the ORB feature points of the frame have been obtained through the ORB feature extraction algorithm, and its attributes include the position of the feature points in the image and the relationship between the feature points and the surrounding pixels. Applying the ORB feature extraction operation to the right eye image at the same time can obtain the position of the feature point pixels in the binocular, and perform ORB feature matching on the two frames of the current binocular image. According to the image recognition and segmentation attributes, the attribution category and dynamic attributes of each ORB feature point are obtained.

[0106] The fifth step is to perform triangulation measurement by matching the left and right target ORB features in the key frame to restore the three-dimensional map points of all feature points in the current frame.

[0107] Step 6: Determine the category of each feature point based on its pixel coordinates. Cluster the feature points of the same category and calculate the modulus of their corresponding map points. Take the average modulus of all map points in the category to represent the average position of the object in the map.

[0108] Determine the modulus length of the feature point according to formula (1):

[0109]

[0110] Among them, ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point.

[0111] According to formula (2), the average modulus length of the object is calculated:

[0112]

[0113] in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

[0114] Step 7: Through the above method, by extracting multiple key frames, the depth of the same object in different frames (n = 1, 2, 3, ..., n) can be calculated. Next, by calculating the depth difference of the same object in adjacent frames The depth changes of the same object in different frames can be obtained.

[0115] In the eighth step, the final dynamic properties of the object are determined by weighted calculation based on the initial value of the dynamic properties of each feature point obtained in the fourth step and the depth change calculated in this step. Take an ORB feature point as an example.

[0116] First, according to the attribution category and dynamic attribute calculated in the fourth step, the initial value v0 of the dynamic attribute is attached to the ORB feature point. The depth change calculated in this step, if the depth change distance in this frame (the i-th frame) is greater than 0.1m (the preset value), the target is considered to be a dynamic target, and the depth change v of the target object in this frame is recorded. i =0.1, otherwise it is recorded as v i =-0.1, after recording multiple frames, the final dynamic properties are calculated by formula (3):

[0117]

[0118] The final dynamic attribute V of the object is obtained. Next, according to the preset dynamic threshold, the dynamic feature points whose dynamic attributes are greater than the threshold are removed, and their map points in the map are deleted at the same time. It has been changed to depth change v i .

[0119] In the ninth step, the map points deleted in the previous step are compared with the map points in the fifth step, the dynamic map points in the local map are optimized, and the local map is updated.

[0120] In the tenth step, by performing the above process on all video frames, the update of all local maps can be optimized and a global map with dynamic feature points removed can be generated.

[0121] In this embodiment, the neural network model with high-precision image segmentation function of Yolov8-seg is used to replace the SLAM method based on the traditional target detection model, which can avoid removing features that are located within the detection frame but outside the actual dynamic target contour, and help improve the removal accuracy of dynamic features. At the same time, the yolov8-seg.onnx model is used to obtain detection results containing information such as image detection bounding box, mask and depth, and the dynamic properties of the feature points in the scene are determined based on the dynamic properties of each object and the coordinates of the pixel position of the point calculated by the SLAM method in the actual three-dimensional world. The object dynamic property acquisition method and dynamic judgment method will be introduced below.

[0122] Different types of objects have different dynamic characteristics. People and cars in the scene are more likely to move, while tables, monitors, etc. are less likely to move. In addition, a cup placed on a table may be moved by a person. Therefore, its motion characteristics need to be evaluated. The core idea of ​​the previous method is to predetermine the dynamic attribute value of the object, which lacks practical connection. However, the inventors hope to make judgments based on the actual behavior of the object, similar to large-scale data training. Based on the idea of ​​the target detection network, the dynamic attribute value is set for each object according to the actual situation. Subsequent dynamic judgments are then made.

[0123] When further dynamic judgment is performed, it is considered that the map point represents the specific 3D position (x, y, z) of the object in the world coordinate system, and maintains the position unchanged when the camera moves. However, if the object is a dynamic object, that is, the position moves, the mapping points recovered from the ORB features in the object mask will change in consecutive frames. When creating new map points, ORB-SLAM3 will consider parallax, reprojection error, scale consistency, and evaluate the depth of each new map point. This process will detect and remove unqualified map points. On this basis, this embodiment introduces a new map point classification method, which can determine whether each new map point corresponds to the detected target object, and then determine whether each type of map point is displaced in the world coordinate system.

[0124] Map points recovered by ORB features of the same object belong to the same category. First, the target object to which each feature point belongs is determined by the pixel position. If the coordinate P(x, y) is within a target mask, the map point can be placed in the map point container of the category object in the format of (category name, all related map points) for processing by using binocular stereo vision and reprojection. Calculate the average module length of all map points in each object in the current frame, and the module length is used to represent the average position of the map points belonging to the object. By comparing the module length change values ​​of different objects in multiple consecutive frames, objects with inconsistent depth changes can be identified. In this way, the dynamic calculation value of each target in three-dimensional space is obtained. If the calculated value exceeds the dynamic limit threshold, the probability that the target has dynamic properties increases. Based on the above judgment, it is possible to accurately determine whether the target has dynamic characteristics.

[0125] Once the features of a dynamic object are identified, they are removed during the ORB feature point extraction process. The dynamic 3D world points corresponding to the dynamic object are removed from the map to ensure the accuracy of the SLAM results. In the subsequent ORB feature point extraction process, if a feature point is within the mask range of the dynamic object, the feature point will be removed and no related 3D map points will be generated, thereby achieving the removal of dynamic targets.

[0126] In summary, this embodiment provides a dynamic environment positioning and mapping method based on binocular vision. The present invention adopts a method of fusion of target detection and segmentation and visual SLAM. Compared with the traditional SLAM method in a dynamic environment, the scheme of the present invention takes into account the situation of real-time evaluation of the dynamic properties of the target object, rather than assuming that objects that can move are dynamic objects, that is, assuming that people are dynamic objects and water cups are static objects. It is more suitable for a wide range of practical application environments. The scheme designed by the present invention has low computational complexity, low computing power requirements, and light load on the robot's computing platform, which is of great significance for improving the accuracy of the robot's positioning and mapping in an actual dynamic environment.

[0127] Example 2

[0128] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a binocular vision-based dynamic environment positioning and mapping method in Example 1.

[0129] Example 3

[0130] A computer program product includes a computer program, which, when executed by a processor, implements the steps of a binocular vision-based dynamic environment positioning and mapping method in Example 1.

[0131] Example 4

[0132] A computer device, which may be a database, may have an internal structure as shown in Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a dynamic environment positioning and mapping method based on binocular vision in Example 1 is implemented.

[0133] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A dynamic environment positioning and mapping method based on binocular vision, characterized in that: include: Acquire a binocular video image set; the binocular video image set includes a left-eye image set and a right-eye image set; timestamps of a plurality of left-eye images in the left-eye image set correspond to timestamps of a plurality of right-eye images in the right-eye image set; Based on the ORB feature matching algorithm, multiple key frames are determined in the left-eye image set; the key frame is a left-eye image with a later timestamp among the two left-eye images, in which ORB feature matching is performed on two adjacent left-eye images, the number of feature matching points is less than a feature matching threshold, and the key frame is a left-eye image with an ORB feature matching algorithm. For any key frame, based on the timestamp of the key frame, determine the corresponding right-eye image in the right-eye image set, and use triangulation to determine the spatial position of each feature point based on the key frame and the right-eye image to obtain a local mapping result of the key frame; Based on the initial dynamic attributes of each pixel in the key frame, clustering each feature point in the local mapping result to determine multiple objects and the spatial position of each object; the initial dynamic attribute is an attribute value assigned to each pixel after target recognition and image segmentation are performed on each left-eye image and each right-eye image through a target recognition and image segmentation model; For any object, based on the spatial position of the object in a key frame and the spatial position of the object in a plurality of consecutive image frames before the key frame, a depth change data set of the object is determined; the depth change data set includes a plurality of depth change data; Based on the depth change data set of the object and the initial dynamic attribute of the object, a final dynamic attribute of the object is determined, and when the final dynamic attribute of the object is greater than a dynamic attribute threshold, feature points corresponding to the object are removed from the local mapping result of the key frame to obtain a local mapping update result of the key frame; According to the local mapping update results of each key frame, global mapping is performed to obtain the global mapping result.

2. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: The target recognition and image segmentation model adopts the yolov8_seg_onnx target detection model.

3. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: Based on the initial dynamic properties of each pixel in the key frame, clustering each feature point in the local mapping result to determine multiple objects and the spatial position of each object specifically includes: Based on the initial dynamic properties of each pixel in the key frame, clustering each feature point in the local mapping result to determine multiple objects; For any feature point of any object, determine the modulus of the feature point according to the spatial position of the feature point; Based on the modulus lengths of the feature points of the object, an average modulus length of the object is calculated, and the average modulus length of the object is used as the spatial position of the object.

4. The method for dynamic environment positioning and mapping based on binocular vision according to claim 3, characterized in that: Determine the modulus length of the feature point according to the following formula: Among them, ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point; The average modulus length of the object is calculated according to the following formula: in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

5. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: Determining a depth change dataset of the object based on the spatial position of the object in the key frame and the spatial position of the object in a plurality of consecutive image frames before the key frame specifically includes: For any image frame before the key frame, determining a depth change distance of the image frame based on a spatial position of the object in the image frame and a spatial position of the object in an image frame before the image frame; Determining depth change data of the image frame based on a depth change distance and a depth change threshold of the image frame; Based on the depth change data of each image frame, a depth change data set of the object is determined.

6. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: Using triangulation method, based on the key frame and the right eye image, the spatial position of each feature point is determined, specifically including: For any pixel point at the same position of the key frame and the right-eye image, two intersecting straight lines are determined based on the camera optical center of the key frame and the camera optical center of the right-eye image; The intersection point of the two intersecting straight lines is taken as a feature point, and the spatial position of the intersection point is taken as the spatial position of the feature point.

7. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: The final dynamic properties of the object are determined according to the following formula: Among them, V is the final dynamic attribute of the object, v0 is the initial dynamic attribute of the object, and v i is the i-th depth change data in the depth change data set of the object, and n is the size of the depth change data set of the object.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a dynamic environment positioning and mapping method based on binocular vision as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a dynamic environment positioning and mapping method based on binocular vision described in any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of a dynamic environment positioning and mapping method based on binocular vision described in any one of claims 1 to 7 are implemented.

Images