A method, apparatus, and electronic device for map optimization.

Abstract

This application provides a method, apparatus, and electronic device for map optimization, relating to the field of robotics. The map optimization method includes: in response to multiple cameras simultaneously acquiring frames of images, extracting feature points; constructing a current feature point set based on the extracted feature points; calculating the residual of each feature point in the current feature point set with respect to each camera group; accumulating the obtained residuals to obtain the accumulated value corresponding to the feature point; identifying whether the feature point belongs to a dynamic feature point based on the accumulated value; determining a target feature point set in response to all feature points in the current feature point set being identified; and updating the map data and the robot's pose based on the feature points in the target feature point set. Therefore, this solution can improve the accuracy of the map constructed by the robot and the robot's localization.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a method, apparatus, and electronic device for map optimization. Background Technology

[0002] In recent years, various types of robots have developed rapidly. Robots are automated machines that perform tasks, relying on their own power and control capabilities to achieve various functions. Map building (or simply mapping) is one of the functions performed by robots. Robots can collect the data needed for mapping and, based on this data, complete the map construction. Map building is usually achieved using SLAM (Simultaneous Localization and Mapping) technology. Furthermore, SLAM technology relies on feature points in the environment during the mapping process to enable the robot to locate itself.

[0003] However, if the mapping is based on the feature points of moving objects in the environment, it will cause errors in the robot's localization and ghosting in the constructed map; thus, the accuracy of the map constructed by the robot and the robot's localization will be low. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, and electronic device for optimizing maps, thereby improving the accuracy of maps constructed by robots and robot localization. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a map optimization method applied to a robot, wherein the robot is equipped with a multi-view camera, the multi-view camera having at least three cameras, and the at least three cameras sharing a common field of view; the method includes:

[0006] After initializing the map data of the target environment scene and the robot's pose based on the images acquired by the multi-view camera, in response to the multi-view camera acquiring each frame of images at the same time, feature points are extracted based on the currently acquired frames of images.

[0007] Based on the extracted feature points, a current feature point set is constructed; wherein, the current feature point set is the set of feature points located in the common viewing region;

[0008] For each feature point in the current feature point set, calculate the residual of that feature point with respect to each camera group; accumulate the obtained residuals to obtain the accumulated value corresponding to that feature point, and based on the accumulated value corresponding to that feature point, identify whether that feature point belongs to a dynamic feature point; wherein, each camera group includes any two cameras from the at least three cameras; the dynamic feature point is the feature point of a moving object in the target environment scene;

[0009] In response to the fact that all feature points in the current feature point set have been identified, a target feature point set is determined; wherein, the target feature point set is: the set obtained by removing target feature points that belong to dynamic feature points from the current feature point set;

[0010] Based on the feature points in the target feature point set, the map data of the target environment scene and the pose of the robot are updated.

[0011] Secondly, embodiments of this application provide a map optimization device applied to a robot, the robot being equipped with a multi-view camera, the multi-view camera having at least three cameras, and the at least three cameras sharing a common field of view; the device includes:

[0012] The feature point extraction module is used to initialize the map data of the target environment scene and the robot's pose based on the images acquired by the multi-view camera, and then extract feature points based on the currently acquired frames of images acquired by the multi-view camera at the same time.

[0013] A construction module is used to construct a current feature point set based on the extracted feature points; wherein, the current feature point set is a set of feature points located in the common viewing region;

[0014] The calculation module is used to calculate the residual of each feature point in the current feature point set with respect to each camera group; accumulate the obtained residuals to obtain the accumulated value corresponding to the feature point; and identify whether the feature point belongs to the dynamic feature point based on the accumulated value corresponding to the feature point; wherein, each camera group includes any two cameras from the at least three cameras; the dynamic feature point is the feature point of the moving object in the target environment scene;

[0015] The determination module is used to determine the target feature point set in response to the fact that all feature points in the current feature point set have been identified; wherein, the target feature point set is: the set obtained by removing target feature points belonging to dynamic feature points from the current feature point set;

[0016] The update module is used to update the map data of the target environment scene and the pose of the robot based on the feature points in the target feature point set.

[0017] Thirdly, embodiments of this application provide an electronic device, including:

[0018] Memory, used to store computer programs;

[0019] The processor, when executing a program stored in memory, implements any of the map optimization methods described above.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the aforementioned map optimization methods.

[0021] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the aforementioned map optimization methods.

[0022] Beneficial effects of the embodiments in this application:

[0023] The map optimization method provided in this application embodiment, in response to multi-camera acquiring frames of images simultaneously, extracts feature points based on the currently acquired frames, constructs a current feature point set based on the extracted feature points, calculates the residual of each feature point with respect to each camera group for each feature point in the current feature point set, accumulates the obtained residuals to obtain the accumulated value corresponding to the feature point, and identifies whether the feature point belongs to a dynamic feature point based on the accumulated value corresponding to the feature point. In response to all feature points in the current feature point set being identified, a target feature point set is determined. The determined target feature point set has already removed target feature points that belong to dynamic feature points. To ensure that the set of target feature points used to update the map data and robot pose of the target environment scene does not contain target feature points that are dynamic feature points, compared to related technologies, in this application, whenever multiple cameras acquire images of each frame at the same time, it can determine whether each feature point in the shared viewing area is a dynamic feature point and remove target feature points that are dynamic feature points, thereby continuously updating the current feature point set. Therefore, this application embodiment does not build a map based on the feature points of moving objects in the environment, reducing the probability of errors in robot localization and the probability of ghost images appearing in the constructed map, thereby improving the accuracy of the map constructed by the robot and robot localization. Furthermore, by using the accumulated value corresponding to the feature point to determine whether the feature point is a dynamic feature point, it can also avoid the problem of small residuals caused by the movement direction of the moving object being parallel to the direction of the epipolar line under a single camera group, thereby reducing the probability of misjudgment when judging feature points.

[0024] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0026] Figure 1 A schematic diagram illustrating the principle of epipolar constraint provided for an embodiment of this application;

[0027] Figure 2 A flowchart illustrating a map optimization method provided in an embodiment of this application;

[0028] Figure 3A flowchart illustrating another map optimization method provided in this application embodiment;

[0029] Figure 4 A flowchart illustrating yet another map optimization method provided in this application embodiment;

[0030] Figure 5 A schematic diagram of a map optimization device provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0033] First, a brief introduction to some technical terms used in the embodiments of this application:

[0034] Epipolar constraint: When the same point is projected onto images from two different viewpoints, the constraints formed by the image point and the optical center of the camera in the projection model. For example... Figure 1 As shown, the optical center line O1O2 connecting the two cameras is called the baseline. The intersection points e1 and e2 of the baseline with pixel plane 1 and pixel plane 2 (which can also be considered as two images) are called base points. Plane O1O2P can be called the epipolar plane. The intersection lines e1p1 and e2p2 of the epipolar plane with pixel plane 1 and pixel plane 2, respectively, can both be called epipolar lines. The image point (i.e., pixel) of point P on pixel plane 1 is p1, and the image point (i.e., pixel) of point P on pixel plane 2 is p2. The image point p2 must lie on the intersection line e2p2 of plane O1O2P and pixel plane 2. This can also be called epipolar constraint. The image point can also be called a feature point.

[0035] Dynamic map: A map that can continuously update the position of objects, and the position of objects can be updated in real time during the map update process;

[0036] EKF filter: Extended Kalman Filter, a high-efficiency recursive filter (autoregressive filter), also known as an extended Kalman filter.

[0037] ORB: Oriented FAST and Rotated BRIEF, a computer vision algorithm designed for real-time feature extraction and matching.

[0038] Optical flow, also known as optical flow or optic flow, is a concept related to motion detection of objects in the field of view; it is used to describe the motion of the observed target, surface, or edge caused by the motion relative to the observer.

[0039] Of course, the polar constraint described above can also be expressed in the form of a formula, which will be explained below using the above formula. Figure 1 The image points P1 and P2 introduced in the text illustrate the polar constraint in formula form:

[0040]

[0041] Where p1 is image point p1, whose coordinates are represented as (u1, v1), p2 is image point p2, whose coordinates are represented as (u2, v2), F 12 Let T be the transformation matrix between camera 1 and camera 2, where camera 1 can be a camera with pixel plane 1 and camera 2 can be a camera with pixel plane 2, and T is the transpose. The vector used to characterize the projection of image point p1 onto camera 2 is 0, meaning that image point p1 lies on the epipolar line; and, K2 is the intrinsic parameter matrix of camera 2, t 12 R represents the displacement (i.e., spatial displacement) of camera 1 and camera 2 relative to the position when the previous frame was captured during the current image acquisition. 12 Let K1 be the rotation angle (i.e., the rotation angle in space) of camera 1 and camera 2 relative to when the previous frame was captured during the current image acquisition. K1 is the intrinsic parameter matrix of camera 1, and -T represents the inverse transpose. Of course, the above is only an exemplary description, and the embodiments of this application do not specifically limit it.

[0042] Furthermore, to better understand this solution, the relevant technologies are introduced below:

[0043] In related technology 1, multiple cameras can be set up in the environment to increase the camera field of view and make the constructed map wider; however, this related technology does not take into account the impact of dynamic obstacles in the environment on pose estimation, and the estimated pose may be inaccurate.

[0044] In related technology 2, AI (Artificial Intelligence) target detection technology can be used to determine whether any feature point belongs to a dynamic feature point by recognizing facial features. However, the detection object of this related technology can only be a living organism, that is, it can only recognize the facial features of living organisms. When the detection object is an object, it cannot recognize the object, nor can it construct a map for the object.

[0045] Based on the problems described above, embodiments of this application provide a method, apparatus, and electronic device for optimizing maps.

[0046] Secondly, the map optimization method provided in the embodiments of this application will be introduced.

[0047] The map optimization method provided in this application can be applied to robots with mobility capabilities and equipped with multi-view cameras. For example, the robot can be a robotic vacuum cleaner, a drone, or an AGV (Automated Guided Vehicle) with multi-view cameras; this application does not specifically limit this. For example, in one implementation, the execution entity of this application can also be the robot's main control module / control module; this application does not specifically limit this. Furthermore, the multi-view camera equipped on the robot in this application has at least three cameras, and these three cameras share a common field of view. Of course, any multi-view camera with at least three cameras can be used as the multi-view camera in this application; the multi-view camera in this application refers to a vision system (also called a vision component or vision sensor component) containing at least three cameras; this application does not specifically limit this. Additionally, the map optimization method provided in this application does not require specific detection objects in the map; the detection objects can be objects or living organisms; this application does not specifically limit this.

[0048] Furthermore, the map optimization method provided in this application can be applied to various map construction scenarios. Any scenario in which a robot constructs a map is applicable to this application, and this application does not impose any specific limitations on it. For example, the map optimization method can be applied to scenarios where a robotic vacuum cleaner constructs a map, or it can be applied to scenarios where a drone robot constructs a map. In addition, the constructed map can be a dynamic map, and this application does not impose any specific limitations on it.

[0049] One of the map optimization methods is applied to a robot equipped with a multi-view camera system. The multi-view camera system has at least three cameras that share a common field of view. The method includes:

[0050] After initializing the map data of the target environment scene and the robot's pose based on the images acquired by the multi-view camera, in response to the multi-view camera acquiring each frame of images at the same time, feature points are extracted based on the currently acquired frames of images.

[0051] Based on the extracted feature points, a current feature point set is constructed; wherein, the current feature point set is the set of feature points located in the common viewing region;

[0052] For each feature point in the current feature point set, calculate the residual of that feature point with respect to each camera group; accumulate the obtained residuals to obtain the accumulated value corresponding to that feature point, and based on the accumulated value corresponding to that feature point, identify whether that feature point belongs to a dynamic feature point; wherein, each camera group includes any two cameras from the at least three cameras; the dynamic feature point is the feature point of a moving object in the target environment scene;

[0053] In response to the fact that all feature points in the current feature point set have been identified, a target feature point set is determined; wherein, the target feature point set is: the set obtained by removing target feature points that belong to dynamic feature points from the current feature point set;

[0054] Based on the feature points in the target feature point set, the map data of the target environment scene and the pose of the robot are updated.

[0055] The map optimization method provided in this application embodiment, in response to multi-camera acquiring frames of images simultaneously, extracts feature points based on the currently acquired frames, constructs a current feature point set based on the extracted feature points, calculates the residual of each feature point with respect to each camera group for each feature point in the current feature point set, accumulates the obtained residuals to obtain the accumulated value corresponding to the feature point, and identifies whether the feature point belongs to a dynamic feature point based on the accumulated value corresponding to the feature point. In response to all feature points in the current feature point set being identified, a target feature point set is determined. The determined target feature point set has already removed target feature points that belong to dynamic feature points. To ensure that the set of target feature points used to update the map data and robot pose of the target environment scene does not contain target feature points that are dynamic feature points, compared to related technologies, in this application, whenever multiple cameras acquire images of each frame at the same time, it can determine whether each feature point in the shared viewing area is a dynamic feature point and remove target feature points that are dynamic feature points, thereby continuously updating the current feature point set. Therefore, this application embodiment does not build a map based on the feature points of moving objects in the environment, reducing the probability of errors in robot localization and the probability of ghost images appearing in the constructed map, thereby improving the accuracy of the map constructed by the robot and robot localization. Furthermore, by using the accumulated value corresponding to the feature point to determine whether the feature point is a dynamic feature point, it can also avoid the problem of small residuals caused by the movement direction of the moving object being parallel to the direction of the epipolar line under a single camera group, thereby reducing the probability of misjudgment when judging feature points.

[0056] The following describes a map optimization method provided by an embodiment of this application, with reference to the accompanying drawings.

[0057] like Figure 2 As shown in the embodiment of this application, a map optimization method is provided; the method includes:

[0058] S201: After initializing the map data of the target environment scene and the robot's pose based on the images acquired by the multi-view camera, in response to the images acquired by the multi-view camera at the same time, feature points are extracted based on the currently acquired images.

[0059] It is understood that initialization can be considered as a process of initially constructing map data of the target environment scene based on images acquired by multi-view cameras, and initially determining the robot's pose. For example, in one implementation, initialization may include: acquiring frames of images with the same timestamp acquired by the multi-view cameras to ensure time consistency; performing image correction on the acquired frames using the calibration parameters of the multi-view cameras (intrinsic parameter matrix, extrinsic parameter matrix, distortion coefficients, etc.), wherein image correction may include image distortion removal, epipolar correction, etc., which are not specifically limited in this embodiment; for each camera in the multi-view cameras, extracting feature points from each frame of images acquired by that camera, matching the feature points of the current frame with the feature points of the previous frame, thereby estimating... The fundamental matrix between adjacent frames is used to obtain the rotation and translation vectors of the camera. Based on the extrinsic parameter matrix of the camera and the body coordinate system (a coordinate system with the robot as the origin), the camera pose is transformed to the body coordinate system, completing the initialization of the camera. After initializing each camera, the poses of each camera and map points can be jointly optimized to complete the overall pose initialization and map construction for the multi-camera system. The robot's initial pose can also be determined based on the determined poses of the multi-camera system. Feature points can also be called ORB feature points. Feature point matching can be used to match feature points belonging to the same object. The extraction and matching of feature points can be implemented using ORB technology, which is not specifically limited in this application embodiment. Of course, the above is only an illustrative description, and any initialization method in related technologies can be used as the initialization method for implementing this application embodiment, which is not specifically limited in this application embodiment.

[0060] Furthermore, after initialization, the initial map data of the target environment and the initial pose of the robot can be determined. Then, in response to the multi-camera acquiring images of each frame simultaneously (frames with the same timestamp acquired by the multi-camera), feature point extraction can be performed on each currently acquired frame. Feature point extraction can be considered as the process of extracting feature points of objects in each frame. In addition, the extracted feature points can also be referred to as the feature points of the current frame acquired by the multi-camera. Of course, any feature point extraction method in related technologies is applicable to the embodiments of this application, and the embodiments of this application do not specifically limit it.

[0061] S202, Based on the extracted feature points, construct the current feature point set;

[0062] Wherein, the current set of feature points is the set of feature points located in the common viewing region;

[0063] It is understandable that after extracting feature points based on the currently acquired frames, considering that the extracted feature points may include feature points of moving objects, this application does not directly use the extracted feature points to update the map data of the target environment scene and the robot's pose. Instead, after extracting feature points, it determines the feature points located in the common viewing area from the extracted feature points and constructs the current feature point set. Therefore, all feature points in the current feature point set are located in the common viewing area of ​​the multi-camera system. Furthermore, the current feature point set may include feature points of moving objects located in the common viewing area of ​​the multi-camera system, and may also include feature points of non-moving objects located in the common viewing area of ​​the multi-camera system. This application does not specifically limit this aspect. For example, based on the currently acquired frames, 100 feature points can be extracted, and 50 feature points are located in the common viewing area. The current feature point set Q can be constructed based on these 50 feature points located in the common viewing area.

[0064] For clarity of layout, the process of constructing the current feature point set based on the extracted feature points will be described in other embodiments and will not be elaborated on here.

[0065] S203, for each feature point in the current feature point set, calculate the residual of the feature point with respect to each camera group; accumulate the obtained residuals to obtain the accumulated value corresponding to the feature point, and based on the accumulated value corresponding to the feature point, identify whether the feature point belongs to a dynamic feature point;

[0066] Each camera group includes any two of the at least three cameras, and different camera groups may contain different cameras; the dynamic feature points are feature points of moving objects in the target environment scene.

[0067] It is understood that the camera groups can be pre-set, and multiple camera groups can be determined by traversing the multi-camera system. Each camera group contains two cameras, and the number of cameras in different camera groups may vary. This application does not specifically limit this. For example, if the multi-camera system includes three cameras, namely camera 1, camera 2, and camera 3, camera 1 and camera 2 can form one camera group, camera 1 and camera 3 can form another camera group, and camera 2 and camera 3 can form yet another camera group, thus obtaining three camera groups. Furthermore, if the residual of the calculated feature point with respect to any camera group is small, the feature point may not belong to a dynamic feature point (feature point of a moving object). If the residual of the calculated feature point with respect to any camera group is large, the feature point belongs to a dynamic feature point.

[0068] It is important to emphasize that when the direction of motion of any object is parallel to the direction of the epipolar line of a camera in a camera group, the calculated residual of the feature point for that object with respect to that camera group may be relatively small. Therefore, judging whether a feature point belongs to a dynamic feature point based solely on the residual of a single camera group may result in misjudgment. Therefore, this embodiment of the application can divide the multi-camera system into multiple camera groups and calculate the residual of the feature point with respect to each camera group separately. The obtained residuals can then be accumulated to obtain the accumulated value corresponding to the feature point. Based on the accumulated value corresponding to the feature point, it can be identified whether the feature point belongs to a dynamic feature point, thereby reducing the probability of misjudgment when judging the feature point. For example, if the residual of feature point a with respect to camera group 1 is 0, the residual of feature point a with respect to camera group 2 is 1, and the residual of feature point a with respect to camera group 3 is 0.6, then the accumulated value corresponding to feature point a can be obtained as 1.6.

[0069] Furthermore, the residual of the feature point with respect to any camera group is used to characterize the difference between the position of the feature point predicted based on the images of that camera group and the actual position of the feature point acquired by that camera group. For example, in one implementation, if the calculated residual is small, it can be considered that the difference between the position of the feature point predicted by the images of that camera group and the actual position of the feature point acquired by that camera group is small; if the calculated residual is large, it can be considered that the difference between the position of the feature point predicted by the images of that camera group and the actual position of the feature point acquired by that camera group is large. This application does not specifically limit this aspect.

[0070] For clarity, the process of calculating the residual of the feature point with respect to each camera group, accumulating the obtained residuals to obtain the accumulated value corresponding to the feature point, and identifying whether the feature point belongs to a dynamic feature point based on the accumulated value corresponding to the feature point are all described in other embodiments and will not be elaborated here.

[0071] S204, in response to the fact that all feature points in the current feature point set have been identified, determine the target feature point set;

[0072] The target feature point set is the set obtained by removing target feature points that belong to dynamic feature points from the current feature point set.

[0073] It is understood that once all feature points in the current feature point set have been identified, it can be considered that all target feature points belonging to dynamic feature points in the current feature point set have been removed. The feature point set after removing target feature points is taken as the target feature point set. The target feature point set only contains feature points belonging to static feature points (feature points of non-moving objects in the target environment scene). This application embodiment does not specifically limit this. For example, if there are 50 feature points in the current feature point set, after identification, 20 target feature points belonging to dynamic feature points are removed, and the remaining 30 feature points can be determined as the target feature point set.

[0074] S205 updates the map data of the target environment scene and the robot's pose based on the feature points in the target feature point set.

[0075] Understandably, the process of updating the map data of the target environment and the robot's pose can include: using optical flow to correlate and match feature points in the target feature point set with the previous frame image captured by the camera group, and projecting and matching feature points in the target feature point set with visible map points (3D points) in the map data of the target environment to establish a correspondence; based on the correspondence between visible map points and feature points in the target feature point set, calculating the pose of the current frame captured by any camera in the camera group, and determining the pose of the current frame captured by the other camera in the camera group; and eliminating mismatched feature point pairs. The system optimizes the pose of the current frame; based on time interval, motion amplitude, number of new feature points, etc., it determines whether the current frame can be used as a keyframe. If so, it is designated as the current keyframe, and feature matching is performed between the current keyframe and co-view keyframes (keyframes that have a co-view relationship with the current keyframe). Based on the unmatched feature points, new feature points are determined; from the determined feature points, feature points located behind the cameras in the camera group are removed. Based on the removed feature points, the pose of the current frame is updated, and based on the feature points extracted that are not in the co-view area, the map data of the target environment scene is updated.

[0076] Of course, any method in the related technologies that updates map data and robot pose based on feature points is applicable to the embodiments of this application, and the embodiments of this application do not specifically limit it.

[0077] In addition, upon receiving a user's command to stop updating the robot, the robot can control the multi-camera to stop image acquisition, thereby ceasing the execution of the map optimization method provided in this application embodiment. This application embodiment does not specifically limit this.

[0078] The map optimization method provided in this application embodiment, in response to multi-camera acquiring frames of images simultaneously, extracts feature points based on the currently acquired frames, constructs a current feature point set based on the extracted feature points, calculates the residual of each feature point with respect to each camera group for each feature point in the current feature point set, accumulates the obtained residuals to obtain the accumulated value corresponding to the feature point, and identifies whether the feature point belongs to a dynamic feature point based on the accumulated value corresponding to the feature point. In response to all feature points in the current feature point set being identified, a target feature point set is determined. The determined target feature point set has already removed target feature points that belong to dynamic feature points. To ensure that the set of target feature points used to update the map data and robot pose of the target environment scene does not contain target feature points that are dynamic feature points, compared to related technologies, in this application, whenever multiple cameras acquire images of each frame at the same time, it can determine whether each feature point in the shared viewing area is a dynamic feature point and remove target feature points that are dynamic feature points, thereby continuously updating the current feature point set. Therefore, this application embodiment does not build a map based on the feature points of moving objects in the environment, reducing the probability of errors in robot localization and the probability of ghost images appearing in the constructed map, thereby improving the accuracy of the map constructed by the robot and robot localization. Furthermore, by using the accumulated value corresponding to the feature point to determine whether the feature point is a dynamic feature point, it can also avoid the problem of small residuals caused by the movement direction of the moving object being parallel to the direction of the epipolar line under a single camera group, thereby reducing the probability of misjudgment when judging feature points.

[0079] In addition, the principle of the map optimization method provided in this application embodiment is relatively simple. It can be implemented by adding a small amount of logic code to the robot, which is easy to implement, low in cost and quick to deploy.

[0080] Alternatively, in another embodiment, such as Figure 3 As shown, the residual of the feature point with respect to each camera group is calculated, including:

[0081] S301, map the feature point to the pixel plane of each camera to obtain the position of the feature point in the currently acquired image of each camera;

[0082] It is understood that mapping the feature point to the pixel plane of each camera can convert the feature point from a three-dimensional point to a two-dimensional point, that is, transform the feature point in the three-dimensional coordinate system to the pixel coordinate system, so as to obtain the position point of the feature point in the currently captured image of each camera; and, after obtaining the position point of the feature point in the currently captured image of each camera, the coordinate information of the feature point in the currently captured image of each camera can be determined. Of course, any conversion method in the related technology is applicable to the embodiments of this application. For example, the three-dimensional coordinates of the feature point can be input into the neural network model so that the neural network model outputs the position point of the feature point in the currently captured image of each camera. The embodiments of this application do not specifically limit this. For example, if the camera group includes camera 1 and camera 2, mapping any feature point P to the pixel plane of each camera can obtain the image point p1 of feature point P mapped to camera 1, and the coordinate information of p1 is (u1, v1). It can also obtain the image point p2 of feature point P mapped to camera 2, and the coordinate information of p2 is (u2, v2).

[0083] S302, Based on the position of the feature point in the currently acquired image of each camera, determine the residual of the feature point with respect to each camera group.

[0084] It is understandable that, based on the location of the feature point in the current image captured by each camera, the residual of the feature point with respect to each camera group can be calculated using a preset residual calculation formula; wherein, the residual of the feature point with respect to each camera group can characterize the difference between the position of the feature point predicted based on the image of the camera group and the actual position of the feature point captured by the camera group.

[0085] As can be seen, the embodiments of this application can map the feature point to the pixel plane of each camera to obtain the position of the feature point in the current image captured by each camera. Based on the position of the feature point in the current image captured by each camera, the residual of the feature point with respect to each camera group can be determined. Subsequently, based on the calculated residual, it can be determined whether the feature point belongs to a dynamic feature point, which provides a basis for updating the map data of the target environment scene and the robot's pose, thereby improving the accuracy of the map constructed by the robot and the robot's localization.

[0086] For example, in one implementation, the residual of the feature point with respect to any camera group is calculated in method A1:

[0087] Method A1: Based on a preset residual calculation formula, calculate the residual of the feature point with respect to the camera group; wherein, the preset residual calculation formula is:

[0088]

[0089] Where, r ij p is the residual of this feature point with respect to the camera group containing camera i and camera j. j ′ Let p be the predicted location of the feature point in the image currently captured by camera j. j ′ =p j +Δp j p j Let Δp be the position of this feature point in the previous frame image captured by camera j. j p is the pixel distance determined by the displacement that occurred when the previous frame was captured, based on the image currently being captured by camera j. i Let F be the position of this feature point in the previous frame image captured by camera i. ij Let F be the transformation matrix between camera i and camera j. ij p i For feature point p i The projection on camera j, (F) ij p i ) x For feature point p i The displacement of the projection onto camera j relative to the optical center of camera j in the x-direction, (F ij p i ) y For feature point p i The displacement of the projection onto camera j relative to the optical center of camera j in the y-direction. Characteristic point p i The absolute value of the vector projected onto camera j; T represents the transpose.

[0090] It is understandable that camera i and camera j can be a group of cameras. The numerator of the formula can represent the position of the feature point predicted by the image of the camera group, and the denominator of the formula can represent the actual position of the feature point acquired by the camera group. Then, the residual of the feature point with respect to each camera group can represent the difference between the position of the feature point predicted based on the image of the camera group and the actual position of the feature point acquired by the camera group.

[0091] And, for the numerator of this formula, p j ′ =p j +Δp jThis can be considered as camera j shifting after capturing a position point in the previous frame of the image. The distance of this shift is the displacement of camera j relative to the position when the previous frame was captured, and the corresponding pixel distance, Δp, can be determined based on this displacement. j The predicted position of the feature point in the current image captured by camera j can be obtained by adding the determined pixel distance to the position of the feature point in the previous frame captured by camera j. Therefore, when the feature point is not a dynamic feature point, the displacement of the feature point between its current position in the captured image and its position in the previous frame is consistent with the pixel distance determined based on the displacement between the current position in the captured image and its position in the previous frame. That is, the object corresponding to the feature point has not moved; the displacement is solely due to the camera's movement, causing the feature point to also move in adjacent image frames. This embodiment does not specifically limit this. Furthermore, if the feature point is not a dynamic feature point, the predicted position of the feature point in the current image captured by camera j can be considered to be located on the epipolar line. It can characterize feature point p i The absolute value of the vector projected onto camera j; if the feature point is not a dynamic feature point, then... Therefore, the calculated residual can also be 0; of course, there may be errors in actual calculations, and the calculated residual may vary. Not equal to 0, but if polar constraints are satisfied, The result should also be extremely small. In addition, based on the calculation result of the numerator of the formula, the magnitude of the residual of the feature point with respect to the camera group can be determined. If the calculation result of the numerator of the formula is large, it can be considered that the feature point does not satisfy the epipolar constraint. Correspondingly, the calculated residual of the feature point with respect to the camera group is also large. Conversely, if the calculation result of the numerator of the formula is small, it can be considered that the feature point satisfies the epipolar constraint. Correspondingly, the calculated residual of the feature point with respect to the camera group is also small. This application does not make specific limitations in this regard.

[0092] Correspondingly, for the denominator of this formula, F ij p i For feature point p i The projection on camera j, (F) ij p i ) x For feature point p i The displacement of the projection onto camera j relative to the optical center of camera j in the x-direction, (F ii p i ) y For feature point p iThe displacement of the projection onto camera j relative to the optical center of camera j in the y-direction, then... It can be considered as F ij p i Normalization is performed to calculate p. i The actual distance projected onto camera j is not specifically limited in this embodiment.

[0093] In addition, F ij It can be considered as the transformation matrix between camera i and camera j, specifically represented as K j Let t be the intrinsic parameter matrix of camera j. ij Let R be the displacement (i.e., spatial displacement) of camera i and camera j relative to when the previous frame was captured during the current image acquisition. ij Let K be the angle of rotation (i.e., the angle of rotation in space) between camera i and camera j when the current image is being captured, relative to when the previous frame was captured. i Let be the intrinsic parameter matrix of camera i. This embodiment of the application does not specifically limit this. Of course, the calculation formulas described above are merely illustrative examples. A neural network model can also be used to calculate the residual of the feature point with respect to any camera group. This embodiment of the application does not specifically limit this.

[0094] As can be seen, based on the preset residual calculation formula introduced above, the residual of the feature point with respect to the camera group can be calculated. When the epipolar constraint is not satisfied, the residual of the feature point with respect to the camera group will be larger, and when the epipolar constraint is satisfied, the residual of the feature point with respect to the camera group will be smaller. Based on the calculated residual, it can be determined whether the feature point belongs to a dynamic feature point, which provides a basis for subsequent updates to the map data of the target environment scene and the robot's pose, thereby improving the accuracy of the map constructed by the robot and the robot's localization.

[0095] Alternatively, in another embodiment, such as Figure 4 As shown, the step of accumulating the obtained residuals to obtain the accumulated value corresponding to the feature point includes:

[0096] S401, determine the weight corresponding to each camera group; wherein, the weight corresponding to any camera group is positively correlated with the baseline between the two cameras in the camera group;

[0097] It is understandable that the weights corresponding to each camera group can be predetermined. The weight of any camera group is positively correlated with the baseline between the two cameras in that group. The longer the baseline between the two cameras in a camera group, the greater the weight of that camera group. For example, if there are camera groups 1, 2, and 3, and the baseline between cameras 1 and 2 in camera group 1 is 15cm, then the weight for camera group 1 can be set to 0.5; the baseline between cameras 1 and 3 in camera group 2 is 10cm, then the weight for camera group 2 can be set to 0.33; and the baseline between cameras 2 and 3 in camera group 3 is 5cm, then the weight for camera group 3 can be set to 0.17. It should be emphasized that the larger the baseline between two cameras, the larger the acquisition range (field of view) of the two cameras. Therefore, a larger weight can be set for camera groups containing these two cameras. Of course, the above is only an illustrative example, and the embodiments of this application do not specifically limit the weights corresponding to each camera group.

[0098] S402, according to the weights corresponding to each camera group, the residuals of the feature point with respect to each camera group are weighted and accumulated to obtain the accumulated value corresponding to the feature point.

[0099] Understandably, based on the weights corresponding to each camera group, the residuals of the feature point with respect to each camera group can be weighted and calculated, and the results can be accumulated to obtain the accumulated value for the feature point. For example, if camera group 1 has a weight of 0.5, the residual of feature point a with respect to camera group 1 is 0.2; camera group 2 has a weight of 0.33, the residual of feature point a with respect to camera group 2 is 0.3; and camera group 3 has a weight of 0.17, the residual of feature point a with respect to camera group 1 is 0.1, resulting in an accumulated value of 0.216 for feature point a.

[0100] To better understand the process of weighted summation of the residuals for each camera group, the following explanation uses the formula as an example:

[0101]

[0102] Where, r total r is the accumulated value corresponding to this feature point. ij w is the residual of this feature point with respect to the camera group containing camera i and camera j. ij Let N be the weight of the feature point with respect to the camera group containing camera i and camera j, where N is the total number of cameras in the multi-camera system.

[0103] As can be seen, the embodiments of this application can set weights for each camera group, and according to the weights corresponding to each camera group, the residuals of the feature point with respect to each camera group are weighted and accumulated to obtain the accumulated value corresponding to the feature point. Subsequently, the accumulated value corresponding to the feature point is used to determine whether the feature point belongs to a dynamic feature point. This can avoid the problem of small residuals caused by the movement direction of the moving object being parallel to the direction of the epipolar line under a single camera group, thereby reducing the probability of misjudgment when judging the feature point.

[0104] Optionally, in another embodiment, identifying whether a feature point belongs to a dynamic feature point based on the accumulated value corresponding to the feature point includes steps B1-B3:

[0105] Step B1: Detect the relationship between the accumulated value corresponding to the feature point and the predetermined residual threshold; wherein, the predetermined residual threshold is a residual threshold determined based on the sparsity of objects in the target environment scene;

[0106] Understandably, the predetermined residual value is a residual threshold determined based on the sparsity of objects in the target environment. The sparser the objects in the target environment, the larger the predetermined residual threshold; conversely, the denser the objects in the target environment, the smaller the predetermined residual threshold. Specifically, the predetermined residual threshold can also be expressed as Υ. dynamic The empirical value of the predetermined residual threshold can be 0.1, and this embodiment does not specifically limit it. Furthermore, after calculating the accumulated value corresponding to the feature point, the relationship between the accumulated value corresponding to the feature point and the predetermined residual threshold can be detected. If the accumulated value corresponding to the feature point is greater than the predetermined residual threshold, then step B2 is executed; otherwise, step B3 is executed.

[0107] Step B2: If the accumulated value corresponding to the feature point is greater than the predetermined residual threshold, then the feature point is determined to be a dynamic feature point.

[0108] It is understandable that if the accumulated value corresponding to a feature point is greater than a predetermined residual threshold, then the feature point can be determined to be a dynamic feature point, that is, a feature point belonging to a moving object in the target environment scene; and subsequently, the feature point can be removed from the current feature point set. For example, if the accumulated value corresponding to feature point a is 0.216 and the predetermined residual threshold is 0.1, then the accumulated value corresponding to feature point a is greater than the predetermined residual threshold, so feature point a can be determined to be a dynamic feature point, and subsequently, feature point a can be removed from the current feature point set. Furthermore, the fact that the accumulated value corresponding to this feature point is greater than the predetermined residual threshold can also be represented as r total >Υ dynamic This application does not impose specific limitations on this aspect.

[0109] Step B3: If the accumulated value corresponding to the feature point is not greater than the residual threshold, then the feature point is determined not to be a dynamic feature point.

[0110] It is understandable that if the accumulated value corresponding to a feature point is not greater than a predetermined residual threshold, then it can be determined that the feature point does not belong to dynamic feature points, that is, it does not belong to the feature points of moving objects in the target environment scene; and, subsequently, there is no need to remove the feature point from the current feature point set. For example, if the accumulated value corresponding to feature point b is 0.1 and the predetermined residual threshold is 0.1, then the accumulated value corresponding to feature point b is not greater than the predetermined residual threshold, and it can be determined that feature point b does not belong to dynamic feature points. Furthermore, the fact that the accumulated value corresponding to this feature point is not greater than the predetermined residual threshold can also be represented as r total ≤Υ dynamic This application does not impose specific limitations on this aspect.

[0111] As can be seen, the embodiments of this application can determine whether a feature point belongs to a dynamic feature point based on the relationship between the accumulated value corresponding to the feature point and the predetermined residual threshold. Compared with related technologies, the embodiments of this application can make a judgment based on the accumulated value corresponding to the feature point, which can reduce the probability of misjudgment when judging the feature point and improve the accuracy of identifying whether the feature point belongs to a dynamic feature point.

[0112] Optionally, in another embodiment, based on the extracted feature points, a current feature point set is constructed, including steps C1-C2:

[0113] Step C1: Determine the feature points located in the common viewing region among the extracted feature points to obtain multiple initial feature points;

[0114] Step C2: From the multiple initial feature points obtained, a specified feature point is removed, and the remaining initial feature points are used to construct the current feature point set; wherein, the position of the specified feature point matches the predicted position obtained after position prediction based on any auxiliary feature point; the auxiliary feature point is: a feature point belonging to the dynamic feature point in the auxiliary feature point set; the auxiliary feature point set is: a feature point set constructed based on the extracted feature points after extracting feature points from images before the current frame image at the acquisition time;

[0115] Accordingly, the method further includes step D1:

[0116] Step D1: For each feature point in the current feature point set that belongs to dynamic feature points, predict the position of the next frame based on the position of the feature point to obtain the predicted position corresponding to the feature point.

[0117] Understandably, for step C1, feature points located in the shared viewing area can be determined from the extracted feature points to obtain multiple initial feature points. For example, based on the currently acquired frames of images, 100 feature points are extracted, and from these 100 feature points, 50 feature points located in the shared viewing area can be determined to obtain 50 initial feature points.

[0118] Understandably, regarding step C2, in response to the multi-camera acquiring images of each frame simultaneously, after extracting feature points from images acquired before the current frame, the feature point set constructed based on the extracted feature points is determined as an auxiliary feature point set. This auxiliary feature point set can include feature points belonging to dynamic feature points as well as feature points not belonging to dynamic feature points. Feature points belonging to dynamic feature points can also be called auxiliary feature points. Then, from the multiple initial feature points obtained, specified feature points are removed, and the remaining initial feature points are used to construct the current feature point set. The position of the specified feature point matches the predicted position obtained after position prediction based on any auxiliary feature point; that is, the predicted position obtained after position prediction based on any auxiliary feature point can be the same as the position of the specified feature point. When it is detected that the position of any feature point among the multiple initial feature points matches (is the same as) the predicted position obtained after position prediction based on auxiliary feature points, it can be determined that the feature point belongs to a dynamic feature point, and this feature point is used as the specified feature point. The specified feature point is then removed from the multiple initial feature points. For example, the initial feature points include four feature points, namely feature point a to feature point d. Feature point a is determined as the specified feature point. Therefore, feature point a is removed from the initial feature points, and the remaining feature points b to feature point d are used to construct the current feature point set.

[0119] Understandably, for step D1, for each feature point belonging to the dynamic feature points in the current feature point set, the position prediction for the next frame can be performed based on the position of that feature point to obtain the predicted position corresponding to that feature point. When the multi-view camera acquires new frames of images, the current feature point set in step D1 can be determined as an auxiliary feature point set, each feature point belonging to the dynamic feature points can be determined as an auxiliary feature point, and based on the predicted positions corresponding to the auxiliary feature points, it can be determined whether a specified feature point exists among the feature points extracted from the acquired new frames of images. If it exists, the specified feature point is removed. For example, if the dynamic feature points in the current feature point set include feature point a, the position prediction for the next frame can be performed based on the position of feature point a to obtain the predicted position corresponding to feature point a. In addition, if any feature point belonging to the dynamic feature points in the current feature point set is located at the edge of the image, it is not necessary to perform position prediction for that feature point in the next frame.

[0120] For example, in one implementation, a prediction model can be established for each feature point belonging to the dynamic feature points in the current feature point set. This prediction model can be a prediction model within an EKF filter. Any feature point belonging to the dynamic feature points is input into the prediction model, causing the prediction model to output the position of that feature point in the next frame, thus obtaining the predicted position corresponding to that feature point. Of course, any method for position prediction in related technologies is applicable to the embodiments of this application, and the embodiments of this application do not specifically limit this method.

[0121] As can be seen, the embodiments of this application can determine the feature points located in the common viewing area among the extracted feature points to obtain multiple initial feature points. From the multiple initial feature points obtained, specified feature points are removed, and the remaining initial feature points are used to construct the current feature point set. Subsequently, for each feature point in the current feature point set that belongs to the dynamic feature point, the position prediction of the next frame can be performed based on the position of the feature point to obtain the predicted position corresponding to the feature point. Therefore, for a feature point that matches the predicted position obtained after position prediction of any auxiliary feature point, it can be directly determined as a feature point belonging to the dynamic feature point and removed from the multiple initial feature points without having to judge the feature point again. This allows for real-time adjustment of the current feature point set, avoids multiple identifications of feature points belonging to the dynamic feature point, improves the efficiency of feature point identification, and also improves the efficiency of map optimization.

[0122] Based on the above method embodiments, this application embodiment also provides a map optimization device applied to a robot, wherein the robot is equipped with a multi-view camera, the multi-view camera has at least three cameras, and the at least three cameras have a common field of view.

[0123] Figure 5 This application provides a schematic diagram of the structure of a map optimization device, as shown in the embodiments of the present application. Figure 5 As shown, the device includes:

[0124] The feature point extraction module 510 is used to perform feature point extraction based on the currently acquired frames of images after initializing the map data of the target environment scene and the robot's pose based on the images acquired by the multi-view camera at the same time.

[0125] The construction module 520 is used to construct a current feature point set based on the extracted feature points; wherein, the current feature point set is a set of feature points located in the common viewing region;

[0126] The calculation module 530 is used to calculate the residual of each feature point in the current feature point set with respect to each camera group; accumulate the obtained residuals to obtain the accumulated value corresponding to the feature point; and identify whether the feature point belongs to the dynamic feature point based on the accumulated value corresponding to the feature point; wherein, each camera group includes any two cameras from the at least three cameras; the dynamic feature point is the feature point of the moving object in the target environment scene;

[0127] The determination module 540 is used to determine the target feature point set in response to the fact that all feature points in the current feature point set have been identified; wherein, the target feature point set is: the set obtained after removing target feature points that belong to dynamic feature points from the current feature point set;

[0128] The update module 550 is used to update the map data of the target environment scene and the pose of the robot based on the feature points in the target feature point set.

[0129] Optionally, the computing module is specifically used for:

[0130] The feature point is mapped onto the pixel plane of each camera to obtain the position of the feature point in the currently captured image of each camera;

[0131] Based on the location of the feature point in the currently acquired image of each camera, the residual of the feature point with respect to each camera group is determined.

[0132] Optionally, the computing module is further configured to:

[0133] Determine the weight corresponding to each camera group; wherein, the weight corresponding to any camera group is positively correlated with the baseline between the two cameras in that camera group;

[0134] According to the weights corresponding to each camera group, the residuals of the feature point with respect to each camera group are weighted and accumulated to obtain the accumulated value corresponding to the feature point.

[0135] Optionally, the computing module is further configured to:

[0136] The relationship between the accumulated value corresponding to the feature point and a predetermined residual threshold is detected; wherein, the predetermined residual threshold is a residual threshold determined based on the sparsity of objects in the target environment scene;

[0137] If the accumulated value corresponding to the feature point is greater than the predetermined residual threshold, then the feature point is determined to be a dynamic feature point;

[0138] If the accumulated value corresponding to the feature point is not greater than the residual threshold, then the feature point is determined not to be a dynamic feature point.

[0139] Optionally, the building module is specifically used for:

[0140] Identify the feature points located in the common viewing region among the extracted feature points to obtain multiple initial feature points;

[0141] From the multiple initial feature points obtained, specified feature points are removed, and the remaining initial feature points are used to construct the current feature point set; wherein, the position of the specified feature point matches the predicted position obtained after position prediction based on any auxiliary feature point; the auxiliary feature point is: the feature point belonging to the dynamic feature point in the auxiliary feature point set; the auxiliary feature point set is: the feature point set constructed based on the extracted feature points after extracting feature points from images before the current frame image at the acquisition time;

[0142] The device further includes:

[0143] The position prediction module is used to predict the position of each feature point in the current set of dynamic feature points in the next frame based on the position of that feature point, and obtain the predicted position of that feature point.

[0144] Optionally, the residual of the feature point with respect to any camera group can be calculated in the following ways:

[0145] Based on a preset residual calculation formula, the residual of the feature point with respect to the camera group is calculated; wherein, the preset residual calculation formula is:

[0146]

[0147] Where, r ij p is the residual of this feature point with respect to the camera group containing camera i and camera j.j ′ Let p be the predicted location of the feature point in the image currently captured by camera j. j ′ =p j +Δp j p j Let Δp be the position of this feature point in the previous frame image captured by camera j. j p is the pixel distance determined based on the displacement between the position of the currently captured image and the position of the previously captured image by camera j. i Let F be the position of this feature point in the previous frame image captured by camera i. ij Let F be the transformation matrix between camera i and camera j. ij p i For feature point p i The projection on camera j, (F) ij p i ) x For feature point p i The displacement of the projection onto camera j relative to the optical center of camera j in the x-direction, (F ij p i ) y For feature point p i The displacement of the projection onto camera j relative to the optical center of camera j in the y-direction. Characteristic point p i The absolute value of the vector projected onto camera j; T represents the transpose.

[0148] In the technical solution of this application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0149] This application also provides an electronic device, such as... Figure 6 As shown, it includes:

[0150] Memory 601 is used to store computer programs;

[0151] The processor 602, when executing the program stored in the memory 601, implements any of the above-mentioned map optimization methods.

[0152] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 602, communication interface, and memory 601 communicating with each other via the communication bus.

[0153] Optionally, the electronic device further includes a multi-view camera, wherein the number of cameras in the multi-view camera is at least three, and the at least three cameras have a common field of view.

[0154] Optionally, the electronic device is an autonomous mobile device. For example, the autonomous mobile device can be the robot described above.

[0155] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0156] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0157] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0158] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0159] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described map optimization methods.

[0160] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the map optimization methods described in the above embodiments.

[0161] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0162] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0164] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A map optimization method, characterized in that, The method is applied to a robot equipped with a multi-view camera, wherein the multi-view camera has at least three cameras and the at least three cameras share a common field of view; the method includes: After initializing the map data of the target environment scene and the robot's pose based on the images acquired by the multi-view camera, in response to the multi-view camera acquiring each frame of images at the same time, feature points are extracted based on the currently acquired frames of images. Based on the extracted feature points, a current feature point set is constructed; wherein, the current feature point set is the set of feature points located in the common viewing region; For each feature point in the current feature point set, calculate the residual of that feature point with respect to each camera group; accumulate the obtained residuals to obtain the accumulated value corresponding to that feature point, and based on the accumulated value corresponding to that feature point, identify whether that feature point belongs to a dynamic feature point; wherein, each camera group includes any two cameras from the at least three cameras; the dynamic feature point is the feature point of a moving object in the target environment scene; In response to the fact that all feature points in the current feature point set have been identified, a target feature point set is determined; wherein, the target feature point set is: the set obtained by removing target feature points that belong to dynamic feature points from the current feature point set; Based on the feature points in the target feature point set, the map data of the target environment scene and the pose of the robot are updated.

2. The method according to claim 1, characterized in that, Calculate the residual of the feature point with respect to each camera group, including: The feature point is mapped onto the pixel plane of each camera to obtain the position of the feature point in the currently captured image of each camera; Based on the location of the feature point in the currently acquired image of each camera, the residual of the feature point with respect to each camera group is determined.

3. The method according to claim 1, characterized in that, The step of accumulating the obtained residuals to obtain the accumulated value corresponding to the feature point includes: Determine the weight corresponding to each camera group; wherein, the weight corresponding to any camera group is positively correlated with the baseline between the two cameras in that camera group; According to the weights corresponding to each camera group, the residuals of the feature point with respect to each camera group are weighted and accumulated to obtain the accumulated value corresponding to the feature point.

4. The method according to claim 1, characterized in that, The step of identifying whether a feature point belongs to a dynamic feature point based on the accumulated value corresponding to the feature point includes: The relationship between the accumulated value corresponding to the feature point and a predetermined residual threshold is detected; wherein, the predetermined residual threshold is a residual threshold determined based on the sparsity of objects in the target environment scene; If the accumulated value corresponding to the feature point is greater than the predetermined residual threshold, then the feature point is determined to be a dynamic feature point; If the accumulated value corresponding to the feature point is not greater than the residual threshold, then the feature point is determined not to be a dynamic feature point.

5. The method according to claim 1, characterized in that, Based on the extracted feature points, construct the current feature point set, including: Identify the feature points located in the common viewing region among the extracted feature points to obtain multiple initial feature points; From the multiple initial feature points obtained, specified feature points are removed, and the remaining initial feature points are used to construct the current feature point set; wherein, the position of the specified feature point matches the predicted position obtained after position prediction based on any auxiliary feature point; the auxiliary feature point is: the feature point belonging to the dynamic feature point in the auxiliary feature point set; the auxiliary feature point set is: the feature point set constructed based on the extracted feature points after extracting feature points from images before the current frame image at the acquisition time; The method further includes: For each feature point in the current set that belongs to dynamic feature points, the position of the next frame is predicted based on the position of the feature point to obtain the predicted position corresponding to the feature point.

6. The method according to claim 1, characterized in that, The residual of this feature point with respect to any camera group is calculated as follows: Based on a preset residual calculation formula, the residual of the feature point with respect to the camera group is calculated; wherein, the preset residual calculation formula is: Where, r ij p is the residual of this feature point with respect to the camera group containing camera i and camera j. j ′ Let p be the predicted location of the feature point in the image currently captured by camera j. j ′ =p j +Δp j p j Let Δp be the position of this feature point in the previous frame image captured by camera j. j p is the pixel distance determined based on the displacement between the position of the currently captured image and the position of the previously captured image by camera j. i Let F be the position of this feature point in the previous frame image captured by camera i. ij Let F be the transformation matrix between camera i and camera j. ij p i For feature point p i The projection on camera j, (F) ij p i ) x For feature point p i The displacement of the projection onto camera j relative to the optical center of camera j in the x-direction, (F ij p i ) y For feature point p i The displacement of the projection onto camera j relative to the optical center of camera j in the y-direction. Characteristic point p i The absolute value of the vector projected onto camera j; T represents the transpose.

7. A map optimization device, characterized in that, An application to a robot equipped with a multi-view camera, wherein the multi-view camera has at least three cameras and the at least three cameras share a common field of view; the device includes: The feature point extraction module is used to initialize the map data of the target environment scene and the robot's pose based on the images acquired by the multi-view camera, and then extract feature points based on the currently acquired frames of images acquired by the multi-view camera at the same time. A construction module is used to construct a current feature point set based on the extracted feature points; wherein, the current feature point set is a set of feature points located in the common viewing region; The calculation module is used to calculate the residual of each feature point in the current feature point set with respect to each camera group; accumulate the obtained residuals to obtain the accumulated value corresponding to the feature point; and identify whether the feature point belongs to the dynamic feature point based on the accumulated value corresponding to the feature point; wherein, each camera group includes any two cameras from the at least three cameras; the dynamic feature point is the feature point of the moving object in the target environment scene; The determination module is used to determine the target feature point set in response to the fact that all feature points in the current feature point set have been identified; wherein, the target feature point set is: the set obtained by removing target feature points belonging to dynamic feature points from the current feature point set; The update module is used to update the map data of the target environment scene and the pose of the robot based on the feature points in the target feature point set.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6.

9. The electronic device according to claim 8, characterized in that, Also includes: A multi-view camera, wherein the number of cameras in the multi-view camera is at least three, and the at least three cameras have a common field of view.

10. The electronic device according to claim 9, characterized in that, The electronic device is an autonomous mobile device.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images