A dynamic environment positioning and mapping method based on binocular vision and related devices

Abstract

The present invention discloses a dynamic environment positioning and mapping method and related devices based on binocular vision, which relate to the field of robot environment perception technology. The method includes the following steps: based on the ORB feature matching algorithm, a key frame is determined in the left eye image set, and based on the timestamp, the corresponding right eye image is determined, and based on the key frame and the right eye image, the spatial position of each feature point is determined by triangulation; then, each feature point is clustered to determine each object and its spatial position; and based on the spatial position of the object in the key frame and several consecutive frames before it, the depth change is determined, and then the final dynamic attribute is determined. When the dynamic attribute threshold is greater than the dynamic attribute threshold, the corresponding feature point is eliminated, and finally global mapping is performed. Compared with the traditional SLAM method in a dynamic environment, the solution of the present invention takes into account the situation of real-time evaluation of the dynamic attributes of the target object, rather than assuming that an object that can move is a dynamic object, and is more suitable for a wide range of practical application environments.

Description

Technical Field

[0001] The present invention relates to the field of robot environment perception, and in particular to a dynamic environment positioning and mapping method based on binocular vision and related devices. Background Art

[0002] Robots with environmental adaptability and robust motion have broad application prospects in the real world. The demand for robots in areas such as unknown environment detection, autonomous task execution, and adaptive trajectory planning is increasing. Environmental perception is a crucial foundation for the practical application of robots. Simultaneous localization and mapping (SLAM), as the first step in environmental exploration, is becoming increasingly important.

[0003] Over the past few years, several research institutions have combined perception methods, including vision and LiDAR technologies, with SLAM frameworks to improve perception capabilities in complex environments, enabling precise positioning and high-fidelity mapping of unknown environments. LiDAR-based SLAM frameworks can capture simple distance information and create point clouds. Due to advanced hardware configurations, they offer high efficiency, versatility, and rich structural information. However, most LiDAR SLAMs typically require relatively large equipment and a large amount of sensor information, which in turn incurs unavoidable costs in terms of acquisition and processing power consumption. These issues are particularly unacceptable in resource-constrained environments, such as when using small robots or performing long-term missions.

[0004] To address the SLAM problem in confined spaces and the fact that small robots cannot carry large sensor equipment, several methods based on deep reinforcement learning have been proposed, incorporating feature extraction into visual SLAM. However, most work incorporates the static environment assumption into the SLAM process, without considering the impact of erroneous data generated by the presence of dynamic moving objects. As a result, these frameworks perform well in ideal and limited scenarios, but suffer significant positioning errors or even failure in the presence of moving objects in real-world scenarios. Therefore, achieving accurate positioning and mapping in challenging dynamic environments has become a pressing challenge in this field. Summary of the Invention

[0005] The purpose of the present invention is to provide a dynamic environment positioning and mapping method and related devices based on binocular vision, which can achieve high-precision positioning and mapping of robots in actual dynamic environments.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] In one aspect, the present invention provides a method for positioning and mapping a dynamic environment based on binocular vision, comprising the following steps:

[0008] A binocular video image set is obtained; the binocular video image set includes a left-eye image set and a right-eye image set; and timestamps of several left-eye images in the left-eye image set correspond to timestamps of several right-eye images in the right-eye image set.

[0009] Based on the ORB feature matching algorithm, multiple key frames are determined in the left-eye image set. A key frame is a left-eye image with a later timestamp and the number of feature matching points is less than the feature matching threshold when ORB feature matching is performed on two adjacent left-eye images.

[0010] For any key frame, the corresponding right eye image is determined in the right eye image set based on the timestamp of the key frame, and the spatial position of each feature point is determined based on the key frame and the right eye image using triangulation to obtain the local mapping result of the key frame.

[0011] Based on the initial dynamic attributes of each pixel in the key frame, the feature points in the local mapping results are clustered to determine multiple objects and their spatial positions. The initial dynamic attributes are the attribute values ​​assigned to each pixel after target recognition and image segmentation are performed on each left and right image using the target recognition and image segmentation model.

[0012] For any object, a depth change dataset of the object is determined based on the spatial position of the object in a key frame and the spatial position of the object in several consecutive image frames before the key frame; the depth change dataset includes several depth change data.

[0013] Based on the depth change dataset of the object and the initial dynamic properties of the object, the final dynamic properties of the object are determined. When the final dynamic properties of the object are greater than the dynamic property threshold, the feature points corresponding to the object are removed from the local mapping result of the key frame to obtain the local mapping update result of the key frame.

[0014] Based on the local mapping update results of each key frame, global mapping is performed to obtain the global mapping result.

[0015] Optionally, the target recognition and image segmentation model adopts the yolov8_seg_onnx target detection model.

[0016] Optionally, based on the initial dynamic properties of each pixel in the key frame, each feature point in the local mapping result is clustered to determine multiple objects and the spatial position of each object, specifically including the following steps:

[0017] Based on the initial dynamic properties of each pixel in the key frame, the feature points in the local mapping result are clustered to determine multiple objects.

[0018] For any feature point of any object, the modulus of the feature point is determined according to the spatial position of the feature point.

[0019] Based on the modulus of each feature point of the object, the average modulus of the object is calculated and used as the spatial position of the object.

[0020] Optionally, the modulus of the feature point is determined according to the following formula:

[0021]

[0022] Among them, ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point.

[0023] The average modulus of the object is calculated according to the following formula:

[0024]

[0025] in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

[0026] Optionally, determining a depth change dataset of the object based on the spatial position of the object in the key frame and the spatial position of the object in a plurality of consecutive image frames before the key frame specifically includes the following steps:

[0027] For any image frame before the key frame, a depth change distance of the image frame is determined based on the spatial position of the object in the image frame and the spatial position of the object in an image frame before the image frame.

[0028] Depth change data of the image frame is determined based on the depth change distance and the depth change threshold of the image frame.

[0029] A depth change dataset of the object is determined based on the depth change data of each image frame.

[0030] Optionally, a triangulation method is used to determine the spatial position of each feature point based on the key frame and the right eye image, specifically including the following steps:

[0031] For any pixel point at the same position in the key frame and the right eye image, two intersecting straight lines are determined based on the camera optical center of the key frame and the camera optical center of the right eye image.

[0032] The intersection of the two intersecting straight lines is taken as a feature point, and the spatial position of the intersection is taken as the spatial position of the feature point.

[0033] Optionally, the final dynamic properties of the object are determined according to the following formula:

[0034]

[0035] Among them, V is the final dynamic property of the object, v0 is the initial dynamic property of the object, and v i is the i-th depth change data in the depth change data set of the object, and n is the size of the depth change data set of the object.

[0036] On the other hand, the present invention provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the steps of a binocular vision-based dynamic environment positioning and mapping method described above.

[0037] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for dynamic environment positioning and mapping based on binocular vision.

[0038] On the other hand, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for positioning and mapping a dynamic environment based on binocular vision.

[0039] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The present invention provides a method and related device for positioning and mapping a dynamic environment based on binocular vision. The method comprises the following steps: obtaining a left-eye image set and a right-eye image set with corresponding timestamps, determining a plurality of key frames in the left-eye image set based on an ORB feature matching algorithm; for any key frame, determining the corresponding right-eye image in the right-eye image set based on the timestamp of the key frame, and using a triangulation method to determine the spatial position of each feature point based on the key frame and the right-eye image, thereby obtaining a local mapping result of the key frame; and then performing a local mapping on each feature point in the local mapping result based on the initial dynamic attributes of each pixel point. The present invention adopts a method of integrating target detection and segmentation with visual SLAM. Compared with the traditional SLAM method in a dynamic environment, the scheme of the present invention takes into account the situation of evaluating the dynamic properties of the target in real time, rather than assuming that a movable object is a dynamic object, that is, assuming that a person is a dynamic object and a water cup is a static object. The scheme is more suitable for a wide range of practical application environments. The scheme designed by the present invention has low computational complexity, low computing power requirements, and light load on the robot's computing platform, which has the significance of improving the accuracy of the robot's positioning and mapping in a practical dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flowchart of a binocular vision-based dynamic environment positioning and mapping method provided in Example 1 of the present invention;

[0043] Figure 2 This is a specific flow chart of step A4 in a binocular vision-based dynamic environment positioning and mapping method provided in Example 1 of the present invention;

[0044] Figure 3 This is a specific flow chart of step A5 in a binocular vision-based dynamic environment positioning and mapping method provided in Example 1 of the present invention;

[0045] Figure 4 This is a specific flow chart of step A6 in a binocular vision-based dynamic environment positioning and mapping method provided in Example 1 of the present invention;

[0046] Figure 5 A schematic diagram of a triangulation method in a binocular vision-based dynamic environment positioning and mapping method provided in Example 1 of the present invention;

[0047] Figure 6 This is a diagram of the internal structure of a computer device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The purpose of the present invention is to provide a dynamic environment positioning and mapping method and related devices based on binocular vision, aiming to achieve accurate positioning and mapping in challenging dynamic environments.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] like Figure 1 As shown in the flowchart, a dynamic environment positioning and mapping method based on binocular vision in this embodiment includes the following steps:

[0053] A1. Obtain a binocular video image set; the binocular video image set includes a left-eye image set and a right-eye image set; the timestamps of several left-eye images in the left-eye image set correspond to the timestamps of several right-eye images in the right-eye image set.

[0054] A2. Based on the ORB feature matching algorithm, multiple key frames are determined in the left-eye image set. A key frame is a left-eye image with a later timestamp and the number of feature matching points is less than the feature matching threshold when ORB feature matching is performed on two adjacent left-eye images.

[0055] A3. For any key frame, based on the timestamp of the key frame, determine the corresponding right-eye image in the right-eye image set.

[0056] A4. Using triangulation method, based on key frame and right eye image, determine the spatial position of each feature point and obtain the local mapping result of key frame. Figure 2 In the flowchart shown, step A4 specifically includes the following steps:

[0057] A41. For any pixel point at the same position in the key frame and the right eye image, determine two intersecting straight lines based on the camera optical center of the key frame and the camera optical center of the right eye image.

[0058] A42. The intersection of two intersecting straight lines is used as a feature point, and the spatial position of the intersection is used as the spatial position of the feature point.

[0059] A5. Based on the initial dynamic attributes of each pixel in the keyframe, cluster the feature points in the local mapping results to determine multiple objects and their spatial locations. The initial dynamic attributes are attribute values ​​assigned to each pixel after performing target recognition and image segmentation on each left and right image using an object recognition and image segmentation model. Specifically, in this embodiment, the object recognition and image segmentation model uses the yolov8_seg_onnx object detection model.

[0060] like Figure 3 In the flowchart shown, step A5 specifically includes the following steps:

[0061] Step A51: Based on the initial dynamic attributes of each pixel in the key frame, cluster the feature points in the local mapping result to determine multiple objects.

[0062] Step A52: For any feature point of any object, determine the modulus of the feature point according to the spatial position of the feature point.

[0063] Determine the modulus of the feature point according to formula (1):

[0064]

[0065] Among them, ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point.

[0066] Step A53: Based on the modulus of each feature point of the object, calculate the average modulus of the object, and use the average modulus of the object as the spatial position of the object.

[0067] The average modulus of the object is calculated according to formula (2):

[0068]

[0069] in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

[0070] A6. For any object, based on the spatial position of the object in the key frame and the spatial position of the object in several consecutive image frames before the key frame, determine the depth change data set of the object; the depth change data set includes several depth change data. Figure 4 In the flowchart shown, step A6 specifically includes the following steps:

[0071] A61. For any image frame before the key frame, determine a depth change distance of the image frame based on a spatial position of the object in the image frame and a spatial position of the object in an image frame preceding the image frame.

[0072] A62. Determine depth change data of the image frame based on the depth change distance and the depth change threshold of the image frame.

[0073] A63. Determine a depth change dataset of the object based on the depth change data of each image frame.

[0074] A7. Based on the depth change dataset of the object and the initial dynamic properties of the object, determine the final dynamic properties of the object. The final dynamic properties of the object are determined according to formula (3):

[0075]

[0076] Among them, V is the final dynamic property of the object, v0 is the initial dynamic property of the object, and v i is the i-th depth change data in the depth change data set of the object, and n is the size of the depth change data set of the object.

[0077] A8. When the final dynamic attribute of the object is greater than the dynamic attribute threshold, the feature points corresponding to the object are removed from the local mapping result of the key frame to obtain the local mapping update result of the key frame.

[0078] A9. Perform global mapping based on the local mapping update results of each key frame to obtain a global mapping result.

[0079] The above content gradually introduces in detail the steps of a dynamic environment positioning and mapping method based on binocular vision provided by this embodiment. The steps of a dynamic environment positioning and mapping method based on binocular vision provided by this embodiment are described in a forward order below. The method mainly includes the following steps:

[0080] The first step is to extract image frames from the binocular video and store them in the image buffer. Using the Yolov8 object detection and segmentation model, these frames are then subjected to object recognition and image segmentation. Based on the object's name, initial dynamic attributes are assigned to the object and the pixels in the image frame it is in.

[0081] The ideas of target recognition and image segmentation are as follows:

[0082] 1) First, multiple regions of interest (ROIs) are generated for each image frame entering the image buffer, where the target to be detected may appear. Because an image contains a large amount of background information, a feasible approach is to first generate the approximate areas where the target may be located, and then fine-tune these ROIs.

[0083] 2) For the region of interest (ROI), features are extracted through a convolutional neural network for subsequent classification and regression.

[0084] 3) After completing the first two steps, a series of candidate boxes containing the target will be obtained. However, these boxes have a lot of overlap (that is, for the same target, multiple boxes may appear to frame it). Therefore, they need to be screened and reorganized, and the non-maximum suppression method (NMS) is applied to ensure that each target is framed by a single box.

[0085] 4) At this point, the target is framed by a single rectangular box. Next, the convolutional neural network is used to extract features from the image in the box and put it into the segmentation model trained by yolov8. With the goal of minimizing the intersection-over-union loss, multiple iterative calculations are performed, and finally the image can be segmented according to its contours.

[0086] The specific process of target recognition and image segmentation is as follows:

[0087] 1) Generate region of interest (ROI):

[0088] The most straightforward way to generate an ROI is to scan the image row by row, for example using a sliding window with varying aspect ratios and proportions relative to the original image size. However, this approach is time-consuming because most of an image is background. Therefore, this example employs selective search, which relies on relationships between pixels, such as color, texture, and size, to cluster and generate an ROI.

[0089] 2) Feature extraction:

[0090] The purpose of feature extraction is to find a representative representation of the region of interest for further use in tasks such as classification and regression. This embodiment uses existing technology, namely the fastest and most stable feature point detection and extraction algorithm, the ORB feature point detection algorithm (Oriented FAST and Rotated BRIEF, which is the abbreviation of Oriented FAST + Rotated BRIEF).

[0091] 3) Non-maximum suppression:

[0092] Non-maximum suppression still uses existing technology. Its idea is to reduce the number of predicted candidate boxes to the number of real targets in the picture, that is, it does not want the same target to be framed by multiple boxes.

[0093] The specific steps for initial dynamic attribute "assignment" are:

[0094] For the 80 common object types that the Yolov8 object detection and segmentation model can detect, we add an initial value for the dynamic attribute for each of these 80 types of objects, ranging from 0 to 100. For example, the "table" class is assigned an initial dynamic attribute of 20, and the "person" class is assigned an initial dynamic attribute of 80. The greater the object's mobility, the higher the initial dynamic attribute value.

[0095] The second step is to read the left eye image from the image buffer and perform ORB feature extraction on this image only. Using FAST corner detection and BRIEF descriptor calculations, image corners with scale and rotation invariance are detected, which are then used to perform ORB feature matching on adjacent keyframes. The goal is to calculate the camera motion of the current frame relative to the previous frame. This involves setting the initial image frame as the pose origin, and using triangulation to estimate the pose of the image frames that match the ORB features to calculate the pose of the current frame. Triangulation here involves calculating the pose relationship between the left eye images of different frames.

[0096] Triangulation is the process of determining the distance of a point by observing the angle between two points. Figure 5 Take the example shown below: Assume that the adjacent keyframes are I1 and I2, and the optical centers of the camera at these two positions are O1 and O2. Using the left image as a reference, the transformation matrix in the right image is T (including the rotation matrix R and the translation matrix t). Assume that there is a feature point p1 in I1 that corresponds to the feature point p2 in I2. At this time, due to the position invariance of the three-dimensional points, the rays O1p1 and O2p2 will intersect and intersect at point P, which is the position of the map point corresponding to the two feature points in the three-dimensional scene. However, due to the influence of noise, these two lines often cannot intersect. Therefore, the least squares method can be used to solve the point with the closest distance as the intersection point.

[0097] According to the definition of epipolar geometry, if x1 and x2 are the normalized coordinates of the two feature points (i.e., x1 = [u1, v1, 1], x2 = [u2, v2, 1], where u and v are the width and height of the description point in the phase plane, respectively), then according to the definition of epipolar geometry, we have the relationship shown in formula (4):

[0098] s1x1=s2Rx2+t (4).

[0099] In equation (4), x1 and x2 are known quantities, while R and t are required quantities. Here, we randomly assign initial values ​​and assume that R and t are known quantities. Their least-squares solution is then found through iteration. Now, we need the depths s1 and s2 of the two feature points, which can be found separately. We start by finding s2.

[0100] Multiply both sides of formula (4) by Where ^ is the antisymmetric symbol of the vector, the purpose is to convert a vector into the form of an antisymmetric matrix In this way, the vector outer product is written as the multiplication of the matrix and the vector, and the result is still a vector, which is perpendicular to the two vectors, so The size is 0.

[0101] So we can get formula (5):

[0102]

[0103] The left side of this equation is zero, and the right side can be viewed as an equation for s2, which can be used to directly calculate s2. With s2, s1 is also very easy to calculate. Thus, the depth of the points in both frames is obtained, and their spatial coordinates are determined. Due to the presence of noise, the estimated R and t may not accurately make the above equation zero, so a common practice is to find a least squares solution. Using the iterative concept of the least squares solution, the pose change matrices R and t of the subsequent frame relative to the previous frame are used as variables, and the point where rays O1p1 and O2p2 intersect is the target. After multiple iterations, the pose change matrices R and t that ultimately meet the threshold requirements are found.

[0104] In the third step, the number of ORB feature matches in the previous step is calculated and compared with the matching threshold. If the number of matches is less than the matching threshold, the current frame is set as the key frame, and the target recognition and segmentation results of the image and the initial dynamic attributes corresponding to the segmented image are found from the results calculated in the first step.

[0105] In the fourth step, the keyframes with recognition, segmentation, and dynamic attributes are used to find the corresponding right-eye image in the image buffer using timestamp consistency. In the previous step, the ORB feature extraction algorithm was used to extract the ORB feature points for that frame. These features include their location within the image and their relationship to surrounding pixels. Applying this ORB feature extraction step to the right-eye image at the same time simultaneously determines the locations of the feature point pixels in both binoculars and performs ORB feature matching on the two frames of the current binocular image. Based on the image recognition and segmentation attributes, the category and dynamic attributes of each ORB feature point are determined.

[0106] The fifth step is to perform triangulation measurement by matching the left and right target ORB features in the key frame to restore the 3D map points of all feature points in the current frame.

[0107] The sixth step is to determine the category of each feature point based on its pixel coordinates. Feature points of the same category are clustered together, and the modulus of their corresponding map points is calculated. The modulus of all map points in the category is averaged to represent the average position of the object on the map.

[0108] Determine the modulus of the feature point according to formula (1):

[0109]

[0110] Among them, ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point.

[0111] The average modulus of the object is calculated according to formula (2):

[0112]

[0113] in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

[0114] Step 7: By extracting multiple key frames using the above method, the depth of the same object in different frames (n=1, 2, 3, ..., n) can be calculated. Next, by calculating the depth difference of the same object in adjacent different frames The depth changes of the same object in different frames can be obtained.

[0115] In the eighth step, the final dynamic properties of the object are determined by weighted calculation based on the initial dynamic property values ​​of each feature point obtained in the fourth step and the depth change calculated in this step. Take an ORB feature point as an example.

[0116] First, according to the attribute category and dynamic attribute calculated in the fourth step, the initial value v0 of the dynamic attribute is attached to the ORB feature point. The depth change calculated in this step, if the depth change distance in this frame (i-th frame) is greater than 0.1m (artificial preset value), then the target is considered to be a dynamic target, and the depth change v of the target object in this frame is recorded. i =0.1, otherwise it is recorded as v i =-0.1, after recording multiple frames, the final dynamic properties are calculated by formula (3):

[0117]

[0118] Obtain the final dynamic attribute V of the object. Next, according to the preset dynamic threshold, remove the dynamic feature points whose dynamic attributes are greater than the threshold, and delete their map points in the map. i .

[0119] In the ninth step, the map points deleted in the previous step are compared with the map points in the fifth step, the dynamic map points in the local map are optimized, and the local map is updated.

[0120] In the tenth step, by performing the above process on all video frames, the updates of all local maps can be optimized and a global map with dynamic feature points removed can be generated.

[0121] In this embodiment, the Yolov8-seg neural network model with high-precision image segmentation function is used to replace the SLAM method based on the traditional target detection model. This can avoid removing features that are within the detection frame but outside the actual dynamic target outline, which helps to improve the accuracy of removing dynamic features. At the same time, the yolov8-seg.onnx model is used to obtain detection results containing information such as image detection bounding box, mask, and depth, and the dynamic nature of the feature points in the scene is determined based on the dynamic properties of each object and the coordinates of the pixel position of the point in the actual three-dimensional world calculated by the SLAM method. The following will introduce the object dynamic property acquisition method and dynamic judgment method.

[0122] Different types of objects have different dynamic characteristics. People and cars in a scene are more likely to move, while tables, monitors, etc. are less likely to move. Furthermore, a cup placed on a table might be moved by a person. Therefore, its motion characteristics need to be evaluated. The core idea of ​​previous methods is to predetermine the dynamic attribute values ​​of an object, which lacks practical relevance. However, the inventors hope to make judgments based on the actual behavior of the object, similar to large-scale data training. Based on the concept of target detection network, dynamic attribute values ​​are set for each object according to the actual situation. This allows for subsequent dynamic judgments.

[0123] When further dynamic judgment is performed, it is considered that the map point represents the specific 3D position (x, y, z) of the object in the world coordinate system, and maintains the same position when the camera moves. However, if the object is a dynamic object, that is, the position moves, the mapping point recovered from the ORB feature within the object mask will change in consecutive frames. When creating a new map point, ORB-SLAM3 will take into account parallax, reprojection error, scale consistency, and evaluate the depth of each new map point. This process will detect and remove unqualified map points. On this basis, this embodiment introduces a new map point classification method, which can determine whether each new map point corresponds to the detected target object, and then determine whether each type of map point is displaced in the world coordinate system.

[0124] Map points recovered from ORB features of the same object belong to the same category. First, the target object to which each feature point belongs is determined by the pixel position. If the coordinate P(x,y) is within a target mask, the map point can be placed into the map point container of the category object in the format of (category name, all related map points) through binocular stereo vision and reprojection. The average module length of all map points in each object in the current frame is calculated. This module length is used to represent the average position of the map points belonging to the object. By comparing the module length changes of different objects in multiple consecutive frames, objects with inconsistent depth changes can be identified. In this way, the dynamic calculation value of each target in three-dimensional space is obtained. If the calculated value exceeds the dynamic limit threshold, the probability that the target has dynamic properties increases. Based on the above judgment, it is possible to accurately determine whether the target has dynamic characteristics.

[0125] Once the features of a dynamic object are identified, they are removed during the ORB feature point extraction process. Dynamic 3D world points corresponding to dynamic objects are removed from the map to ensure the accuracy of the SLAM results. During the subsequent ORB feature point extraction process, if a feature point falls within the mask range of a dynamic object, it is removed and no associated 3D map points are generated, thus removing the dynamic object.

[0126] In summary, this embodiment provides a dynamic environment positioning and mapping method based on binocular vision. The present invention adopts a method of target detection and segmentation and visual SLAM fusion. Compared with the traditional SLAM method in a dynamic environment, the scheme of the present invention takes into account the situation of real-time evaluation of the dynamic properties of the target object, rather than assuming that objects that can move are dynamic objects, that is, assuming that people are dynamic objects and water cups are static objects. It is more suitable for a wide range of practical application environments. The scheme designed by the present invention has low computational complexity, low computing power requirements, and light load on the robot's computing platform, which is of great significance for improving the accuracy of the robot's positioning and mapping in actual dynamic environments.

[0127] Example 2

[0128] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a binocular vision-based dynamic environment positioning and mapping method in Example 1.

[0129] Example 3

[0130] A computer program product includes a computer program, which, when executed by a processor, implements the steps of a binocular vision-based dynamic environment positioning and mapping method in embodiment 1.

[0131] Example 4

[0132] A computer device, which may be a database, may have an internal structure as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a dynamic environment positioning and mapping method based on binocular vision in Example 1 is implemented.

[0133] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0134] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0135] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for positioning and mapping a dynamic environment based on binocular vision, characterized in that: include: Acquire a binocular video image set; the binocular video image set includes a left-eye image set and a right-eye image set; timestamps of a plurality of left-eye images in the left-eye image set correspond to timestamps of a plurality of right-eye images in the right-eye image set; Based on the ORB feature matching algorithm, multiple key frames are determined in the left eye image set; the key frame is the left eye image with the later timestamp among the two left eye images, which has the number of feature matching points less than the feature matching threshold after ORB feature matching is performed on two adjacent left eye images; For any key frame, based on the timestamp of the key frame, determine the corresponding right eye image in the right eye image set, and use triangulation to determine the spatial position of each feature point based on the key frame and the right eye image to obtain a local mapping result of the key frame; Clustering the feature points in the local mapping result based on the initial dynamic attributes of each pixel in the key frame to determine multiple objects and the spatial position of each object; the initial dynamic attributes are attribute values ​​assigned to each pixel after performing target recognition and image segmentation on each left image and each right image using a target recognition and image segmentation model; For any object, based on the spatial position of the object in a key frame and the spatial position of the object in a plurality of consecutive image frames before the key frame, a depth change dataset of the object is determined; the depth change dataset includes a plurality of depth change data; Determining a final dynamic attribute of the object based on the depth change dataset of the object and the initial dynamic attribute of the object, and when the final dynamic attribute of the object is greater than a dynamic attribute threshold, removing feature points corresponding to the object from the local mapping result of the key frame to obtain an updated local mapping result of the key frame; Based on the local mapping update results of each key frame, global mapping is performed to obtain the global mapping result.

2. A method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: The target recognition and image segmentation model adopts the yolov8_seg_onnx target detection model.

3. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: Based on the initial dynamic properties of each pixel in the key frame, each feature point in the local mapping result is clustered to determine multiple objects and the spatial position of each object, specifically including: Clustering the feature points in the local mapping result based on the initial dynamic attributes of each pixel in the key frame to determine multiple objects; For any feature point of any object, determine the modulus of the feature point according to the spatial position of the feature point; Based on the modulus of each feature point of the object, an average modulus of the object is calculated, and the average modulus of the object is used as the spatial position of the object.

4. The method for dynamic environment positioning and mapping based on binocular vision according to claim 3, characterized in that: Determine the modulus of the feature point according to the following formula: Where ||P1|| is the modulus of the first feature point, x1, y1 and z1 are the x-axis coordinate, y-axis coordinate and z-axis coordinate of the first feature point respectively, and (x1, y1, z1) is the spatial position of the first feature point; The average modulus of the object is calculated according to the following formula: in, is the average modulus of the object, ||P i || is the modulus of the i-th feature point, and p is the number of feature points of the object.

5. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: Determining a depth change dataset of the object based on the spatial position of the object in the key frame and the spatial positions of the object in a plurality of consecutive image frames before the key frame, specifically comprising: For any image frame before the key frame, determining a depth change distance of the image frame based on a spatial position of the object in the image frame and a spatial position of the object in an image frame preceding the image frame; Determining depth change data of the image frame based on a depth change distance and a depth change threshold of the image frame; A depth change dataset of the object is determined based on the depth change data of each image frame.

6. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: Using triangulation, based on the key frame and the right eye image, the spatial position of each feature point is determined, specifically including: For any pixel point at the same position of the key frame and the right eye image, determine two intersecting straight lines based on the camera optical center of the key frame and the camera optical center of the right eye image; The intersection of the two intersecting straight lines is taken as a feature point, and the spatial position of the intersection is taken as the spatial position of the feature point.

7. The method for dynamic environment positioning and mapping based on binocular vision according to claim 1, characterized in that: The final dynamic properties of the object are determined according to the following formula: Among them, V is the final dynamic property of the object, v0 is the initial dynamic property of the object, and v i is the i-th depth change data in the depth change dataset of the object, and n is the size of the depth change dataset of the object.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a dynamic environment positioning and mapping method based on binocular vision according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a binocular vision-based dynamic environment positioning and mapping method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of a binocular vision-based dynamic environment positioning and mapping method according to any one of claims 1 to 7 are implemented.

Images