Plane fusion positioning method, device, electronic equipment and storage medium

Abstract

The present application provides a plane fusion positioning method, device, electronic device and storage medium, wherein the plane fusion positioning method includes: constructing a sparse plane using a sparse point cloud obtained based on a currently input image frame, obtaining first association information between a target three-dimensional grid and a sparse plane generated by a front end based on the image frame; obtaining a first point-surface constraint condition according to the first association information and the current first pose of the carrier; obtaining an optimized value of the first pose according to the first point-surface constraint condition, a reprojection constraint condition and a depth constraint condition obtained based on the image frame and the previous frame of the image frame. The embodiments of the present application are conducive to improving the positioning accuracy in a SLAM system.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a planar fusion positioning method, device, electronic device and storage medium. Background Art

[0002] With the development of computer vision technology, the research work of SLAM (Simultaneous Localization And Mapping) technology has achieved outstanding results and has been widely used in AR (Augmented Reality) or VR (Virtual Reality) products, drones, robots and autonomous driving. The construction and optimization of structured planes are key links in SLAM technology. At present, in terms of structured plane optimization, the accuracy and stability of the generated planes are limited due to the influence of the selected feature points and plane constraints, which leads to low positioning accuracy. Summary of the invention

[0003] The embodiments of the present application provide a planar fusion positioning method, device, electronic device and storage medium.

[0004] A first aspect of an embodiment of the present application provides a planar fusion positioning method, the method comprising:

[0005] A sparse plane is constructed by using a sparse point cloud obtained based on a currently input image frame, and first correlation information between a target three-dimensional grid generated by a front end based on the image frame and the sparse plane is obtained;

[0006] Obtaining a first point-surface constraint condition according to the first association information and the first current pose of the carrier;

[0007] According to the first point-surface constraint, the reprojection constraint obtained based on the image frame and the previous frame of the image frame, and the depth constraint, an optimized value of the first pose is obtained.

[0008] In combination with the first aspect, in a possible implementation, obtaining a first point-surface constraint condition according to the first association information and the first current pose of the carrier includes:

[0009] Obtaining second association information between a historical sparse plane constructed based on the historical sparse point cloud obtained in the previous frame and a historical three-dimensional grid generated by the front end based on the previous frame, as well as a second position and posture of the carrier in the previous frame;

[0010] The first association information, the first pose, the second association information and the second pose are used to construct a point-surface optimization model to obtain a first point-surface constraint condition.

[0011] In combination with the first aspect, in a possible implementation, obtaining an optimized value of the first pose according to the first point-surface constraint, the reprojection constraint obtained based on the image frame and the previous frame of the image frame, and the depth constraint includes:

[0012] Acquire at least one second point-surface constraint condition, the at least one second point-surface constraint condition includes the first point-surface constraint condition, the point-surface constraint condition other than the first point-surface constraint condition in the at least one second point-surface constraint condition is obtained based on target association information, and the target association information is association information obtained based on a first historical frame of a preset number of image frames;

[0013] The first pose is optimized, and a pose estimate satisfying at least one second point-surface constraint, a reprojection constraint, and a depth constraint is used as an optimized value.

[0014] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0015] The marginalization is based on the point-surface constraints obtained from the second historical frame before the first historical frame.

[0016] In combination with the first aspect, in a possible implementation manner, before constructing the sparse plane using the sparse point cloud obtained based on the currently input image frame, the method further includes:

[0017] Uniformly sampling the image frame to obtain at least one two-dimensional feature point;

[0018] Determine a point of at least one two-dimensional feature point in a depth map corresponding to the image frame;

[0019] The point of at least one two-dimensional feature point in the depth map corresponding to the image frame is mapped to the three-dimensional space to obtain a sparse point cloud.

[0020] In combination with the first aspect, in a possible implementation manner, after obtaining at least one two-dimensional feature point, the method further includes:

[0021] Constructing a two-dimensional grid using at least one two-dimensional feature point;

[0022] The sparse plane is constructed using the sparse point cloud obtained based on the current input image frame, including:

[0023] Based on the correspondence between the two-dimensional feature points in the two-dimensional grid and the three-dimensional points in the sparse point cloud, the three-dimensional points are used to construct a three-dimensional grid corresponding to the two-dimensional grid;

[0024] The 3D mesh is clustered based on the first pose to obtain a sparse plane.

[0025] In combination with the first aspect, in a possible implementation manner, uniformly sampling the image frame to obtain at least one two-dimensional feature point includes:

[0026] Remove edge pixels of the image frame;

[0027] The sampling distance is calculated according to the size of the image frame and the preset sampling number;

[0028] The pixels that have not been removed in the image frame are sampled according to the sampling distance to obtain at least one two-dimensional feature point.

[0029] In combination with the first aspect, in a possible implementation manner, obtaining first association information between a target three-dimensional grid and a sparse plane generated by a front end based on an image frame includes:

[0030] Obtain the difference between the direction of the target three-dimensional grid and the sparse plane;

[0031] When the difference is less than or equal to the preset value and at least one three-dimensional point constituting the target three-dimensional grid is on the sparse plane, the target three-dimensional grid is associated with the sparse plane to obtain first association information.

[0032] A second aspect of the embodiments of the present application provides a planar fusion positioning device, the device comprising an acquisition unit and a processing unit;

[0033] An acquisition unit, configured to construct a sparse plane using a sparse point cloud obtained based on a currently input image frame, and acquire first association information between a target three-dimensional grid generated by a front end based on the image frame and the sparse plane;

[0034] A processing unit, configured to obtain a first point-surface constraint condition according to the first association information and the first current pose of the carrier;

[0035] The processing unit is further used to obtain the optimized value of the first pose according to the first point-surface constraint, the reprojection constraint obtained based on the image frame and the previous frame of the image frame, and the depth constraint.

[0036] A third aspect of an embodiment of the present application provides an electronic device, which includes an input device and an output device, and also includes a processor suitable for implementing one or more instructions; and a computer storage medium, wherein the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by the processor and executing the steps in the method described in the first aspect above.

[0037] A fourth aspect of an embodiment of the present application provides a computer storage medium, which stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the steps in the method described in the first aspect above.

[0038] A fifth aspect of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the steps in the method described in the first aspect. The computer program product may be a software installation package.

[0039] It can be seen that in the embodiment of the present application, a sparse plane is constructed by using a sparse point cloud obtained based on the currently input image frame, and first association information between the target three-dimensional grid and the sparse plane generated by the front end based on the image frame is obtained; the first point-surface constraint condition is obtained according to the first association information and the current first pose of the carrier; the optimized value of the first pose is obtained according to the first point-surface constraint condition, the reprojection constraint condition and the depth constraint condition obtained based on the image frame and the previous frame of the image frame. Since the current input image frame is an RGBD image acquired by an RGBD image acquisition device, and an RGBD image usually contains observed two-dimensional data, such as data provided by a depth sensor, and the observed two-dimensional data is relatively accurate, then the sparse point cloud obtained based on the two-dimensional data is more accurate than the estimated three-dimensional data, thereby making the constructed sparse plane more accurate, which is beneficial to improving the accuracy of the point-to-surface association (i.e., the first association information). In addition, on the basis of the reprojection constraints and depth constraints given at the front end of the SLAM system, point-to-surface constraints based on point-to-surface association and the first pose (i.e., the first point-to-surface constraint condition) are added. By continuously optimizing the first pose so that the pose of the carrier satisfies the above three constraints, it is beneficial to reduce the deviation between the estimated and observed poses, thereby improving the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 An architectural diagram of an application environment provided for an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a flow chart of a planar fusion positioning method provided in an embodiment of the present application;

[0043] Figure 3 A visual schematic diagram of point-to-surface association provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a process for obtaining a first point-surface constraint condition provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a process for obtaining an optimized value of the first pose provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of feature point sampling provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a flow chart of another plane fusion positioning method provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a planar fusion positioning device provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0051] The terms "including" and "having" and any variations thereof that appear in the specification, claims and drawings of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices. In addition, the terms "first", "second" and "third" are used to distinguish different objects, rather than to describe a specific order.

[0052] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0053] The present application embodiment proposes a plane fusion positioning method, which can be based on Figure 1 The application environment shown is implemented as Figure 1As shown, the application environment includes an electronic device 101, an image acquisition device 102 and a terminal device 103 that are communicatively connected to the electronic device 101. Among them, the electronic device 101 supports the operation of the front-end and back-end of the SLAM system. The electronic device 101 involved in the embodiment of the present application may include various devices with program code running capabilities and communication capabilities. For example, the electronic device 101 may be an independent physical server, or a server cluster or a distributed system, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms, or the electronic device 101 may also be various robots, vehicle-mounted devices, wearable devices, and the like.

[0054] Among them, the image acquisition device 102 can be an RGBD image acquisition device. In some scenarios, it can specifically be an RGBD image acquisition device set on the electronic device 101. The image acquisition device 102 is used to send the collected RGBD image data to the electronic device 101, so that the electronic device 101 can construct a sparse plane based on the image data, and associate the three-dimensional points in the three-dimensional mesh (3Dmesh) constructed by the front end with the sparse plane, and construct a point-surface optimization model based on the point-surface association information of adjacent frames and the pose of the carrier, so as to obtain the point-surface constraint conditions, and continuously optimize the pose of the carrier to make it meet the point-surface constraint conditions, the reprojection constraint conditions and the depth constraint conditions given by the SLAM system front end, thereby improving the positioning accuracy in the SLAM system.

[0055] The terminal device 103 can be used to visualize the image data collected by the image acquisition device 102, the two-dimensional mesh (2Dmesh), 3Dmesh, map and positioning results in the SLAM system constructed by the electronic device 101. The terminal device 103 can be a device with a display function, such as a computer, AR (Augmented Reality) or VR (VirtualReality) glasses, or an independent display.

[0056] The plane fusion positioning method provided in the embodiment of the present application is described in detail below in conjunction with the relevant drawings.

[0057] See also Figure 2 , Figure 2 A schematic diagram of a flow chart of a planar fusion positioning method provided in an embodiment of the present application, the method is applied to electronic devices, such as Figure 2 As shown, steps 201-204 are included:

[0058] Step 201: construct a sparse plane using a sparse point cloud obtained based on a currently input image frame, and obtain first association information between a target 3D mesh generated by a front end based on the image frame and the sparse plane.

[0059] In the embodiment of the present application, the currently input image frame refers to the latest RGBD image acquired by the image acquisition device. For example, for the image frame, the electronic device uniformly samples it to obtain at least one two-dimensional (2D) feature point, such as Figure 3 As shown in Figure b, it should be understood that the RGBD image usually has a corresponding depth map, and the at least one 2D feature point can determine its point in the corresponding depth map, and the point of the at least one 2D feature point in the corresponding depth map is mapped to a three-dimensional (3D) space, that is, a sparse point cloud for constructing a sparse plane is obtained. In this embodiment, since the points in the depth map are data observed by the depth sensor, they are relatively accurate. The sparse plane constructed by the sparse point cloud obtained from these points has higher accuracy and stability than the plane constructed by the dense point cloud.

[0060] Exemplarily, after obtaining at least one 2D feature point, the method further includes: constructing a 2D mesh using the at least one 2D feature point. For example, the at least one 2D feature point may be triangulated to construct the 2D mesh, or the 2D mesh may be constructed by the right-angled sampling distance of the 2D mesh determined during sampling. For example, if two right-angled edges are known from three 2D feature points, then a 2D mesh is formed by adding the hypotenuse.

[0061] Exemplarily, constructing a sparse plane using a sparse point cloud obtained based on a currently input image frame includes:

[0062] Based on the correspondence between the 2D feature points in the 2D mesh and the 3D points in the sparse point cloud, the 3D points are used to construct the 3D mesh corresponding to the 2D mesh;

[0063] Cluster the 3D mesh based on the first pose to obtain a sparse plane.

[0064] In the embodiment of the present application, for each 2D feature point in the 2Dmesh, there is a corresponding 3D point in the sparse point cloud, such as Figure 3 The corresponding relationship (or mapping relationship) between the points shown in Figure b and the points shown in Figure f can be reflected by the same identifier. For example, if 2D feature points A, B, and C constitute a 2D mesh, the 3D points corresponding to A, B, and C can be used to constitute a 3D mesh. It should be understood that the SLAM system can estimate the distance between any two 3D meshes in the constructed 3D mesh based on the carrier's position and posture. The electronic device can cluster the 3D meshes whose distance is less than or equal to a certain threshold to obtain the final constructed sparse plane, which can be shown in the 3D space as Figure 3 As shown in Figure d, in 2D space, Figure 3 As shown in Figure f in the middle. In this implementation, a sparse plane is constructed based on observed 2D feature points with higher accuracy, which is beneficial to improving the stability and accuracy of the plane.

[0065] Among them, the target 3Dmesh is the 3Dmesh constructed by the SLAM front end based on the selected 3D points, which can be Figure 3 As shown in Figure c, in 2D space, Figure 3 As shown in Figure e, the front end of the SLAM system can extract features from the image frame to obtain the front end 2D feature points, such as Figure 3 As shown in Figure a, these 2D feature points are then mapped to 3D points. The front end of the SLAM system can be a visual odometer (Visual Odometry).

[0066] Exemplarily, obtaining first association information between a target 3D mesh and a sparse plane generated by a front end based on an image frame includes:

[0067] Get the difference between the direction of the target 3Dmesh and the sparse plane;

[0068] When the difference is less than or equal to the preset value and at least one 3D point constituting the target 3D mesh is on the sparse plane, the target 3D mesh is associated with the sparse plane to obtain first association information.

[0069] In the embodiment of the present application, the association of the plane needs to meet two conditions: one is that the direction is similar. When the difference between the direction of the target 3Dmesh and the sparse plane is less than or equal to the preset value, it can be determined that the two directions are similar, wherein the preset value can be set according to the empirical value; the second is that at least one 3D point of the target 3Dmesh is on the sparse plane, that is, at least one 3D point of the target 3Dmesh satisfies the plane equation of the sparse plane. When the above two conditions are met, it is considered that the target 3Dmesh and the sparse plane belong to the same plane, then the 3D points in the target 3Dmesh can be associated with the sparse plane to obtain the first association information, which can be understood as the plane obtained by expanding the sparse plane. In this implementation, since the 2D feature points selected by the front end of the SLAM system have the characteristic of fast speed, the 3D points corresponding to such feature points are used to expand the sparse plane, which is conducive to improving the speed of plane association or expansion.

[0070] Step 202: Obtain a first point-surface constraint condition according to the first association information and the first current pose of the carrier.

[0071] In the present application embodiment, Figure 4As shown, according to the first association information and the first current pose of the carrier, the first point-surface constraint condition is obtained, including steps 401-402:

[0072] Step 401: obtaining second association information between a historical sparse plane constructed based on the historical sparse point cloud obtained in the previous frame and the historical 3Dmesh generated by the front end based on the previous frame, and a second posture of the carrier in the previous frame;

[0073] Step 402: construct a point-surface optimization model using the first association information, the first pose, the second association information, and the second pose to obtain a first point-surface constraint condition.

[0074] Among them, the historical sparse point cloud refers to the sparse point cloud obtained based on at least one 2D feature point in the previous frame, the historical sparse plane refers to the sparse plane constructed based on the historical sparse point cloud, the historical 3D mesh refers to the 3D mesh constructed by the front end based on the previous frame, and the second associated information refers to the associated information between the historical sparse plane and the historical 3D mesh. The acquisition method is the same as the first associated information, that is, the sparse point cloud is used to construct the sparse plane, and then the 3D mesh constructed by the front end is associated with the sparse plane. Among them, the first pose refers to the pose given by the front end at the current moment, and the second pose refers to the pose given by the front end at the moment of the previous frame. In the SLAM system, the front end can obtain the pose at each moment. Since a point on a plane may be observed by multiple adjacent frames, for example, a point in the first associated information may also be observed in its previous frame, such as Figure 3 The observation position of point P in the current image frame is Py, and the observation position in the previous frame is px. Based on this relationship, the electronic device uses the first association information, the first pose, the second association information and the second pose to construct the point-surface optimization initial model shown below:

[0075]

[0076] in, and Respectively represent the 3D coordinates of point h in the coordinate system of the previous frame c1 and the current image frame c2. The point can be the same point observed in the first association information and the second association information. and Respectively represent the rotation of the previous frame c1 and the current image frame c2 in the sub-world coordinate system W, and They respectively represent the positions of the previous frame c1 and the current image frame c2 in the world coordinate system W, which can be obtained from the first pose and the second pose respectively. n and d represent the parameters of the sparse plane constructed based on the current image frame. n represents the direction of the plane, d represents the distance of the plane, E represents the unit matrix, and T represents the transpose.

[0077] It should be understood that the points in adjacent frames usually satisfy the homography transformation. For the above initial model, a homography matrix H can be obtained to simplify it. The simplified point-surface optimization model is as follows:

[0078]

[0079] Then the first point-surface constraint can be expressed as:

[0080]

[0081] Wherein, Error represents the position error observed by point h in the previous frame c1 and the current image frame c2 in the plane coordinate system of the carrier. The electronic device can continuously optimize the initial pose (i.e., the first pose) given by the front end of the SLAM system to make the estimated value of the pose of the carrier make the error Error smaller (closer to 0). In this implementation, adding homography Homograph constraints (i.e., homography matrix H) to the constructed point-surface constraints is conducive to improving the accuracy of the constraints and can reduce the dependence on 3D point accuracy in algorithm positioning or tracking.

[0082] Step 203: Obtain an optimized value of the first pose according to the first point-surface constraint, the reprojection constraint obtained based on the image frame and the previous frame of the image frame, and the depth constraint.

[0083] In an embodiment of the present application, similar to the first point-surface constraint condition, the electronic device also obtains a reprojection constraint condition and a depth constraint condition based on the current image frame and the previous frame. The front end of the SLAM system can estimate the posture of the carrier through a series of observations and movements. These observations and movements can provide constraints so that the posture satisfies these constraints at the same time. Among them, these constraints include the front end of the SLAM system providing a depth constraint condition and a reprojection constraint condition. The depth constraint condition is used to make the depth error of the observation point of the same point in adjacent frames smaller, and the reprojection constraint condition is used to make the reprojection error of the observation point of the same point in adjacent frames smaller.

[0084] For example, Figure 5 As shown, according to the first point-surface constraint, the reprojection constraint and the depth constraint obtained based on the image frame and the previous frame of the image frame, the optimization value of the first pose is obtained, including steps 501-502:

[0085] Step 501: obtaining at least one second point-surface constraint condition, wherein the at least one second point-surface constraint condition includes the first point-surface constraint condition, wherein the point-surface constraint condition other than the first point-surface constraint condition in the at least one second point-surface constraint condition is obtained based on target association information, and the target association information is association information obtained based on a first historical frame of a preset number of image frames;

[0086] Step 502: Optimize the first pose, and use the pose estimate that satisfies at least one second point-surface constraint, a reprojection constraint, and a depth constraint as the optimized value.

[0087] Among them, at least one second point-surface constraint condition is obtained based on the current image frame and a preset number of first historical frames closest to the current image frame. For example, if the current image frame is the 11th frame, then at least one second point-surface constraint condition is obtained based on the 1st to 11th frames, and the first historical frame refers to the previous 1st to 10th frames, wherein the target association information refers to the association information obtained based on the 1st frame, the 2nd frame...the 10th frame. For example, a second point-surface constraint condition can be constructed based on the point-surface association information of the first frame and the posture at the first frame, the point-surface association information of the second frame and the posture at the second frame, a second point-surface constraint condition can be constructed based on the point-surface association information of the second frame and the posture at the second frame, the point-surface association information of the third frame and the posture at the third frame… A second point-surface constraint condition (i.e., the first point-surface constraint condition) can be constructed based on the point-surface association information of the tenth frame (i.e., the previous frame of the current image frame) and the posture at the tenth frame (i.e., the second posture), the point-surface association information of the eleventh frame (i.e., the current image frame) (i.e., the first association information) and the posture at the eleventh frame (i.e., the first posture).

[0088] In the SLAM system, the previous preset frames of the current image frame are usually considered to be newer frames, and the point-surface constraints constructed based on these frames will still have an impact on the current positioning. Therefore, when optimizing the first pose, the estimated value of the pose should be made to satisfy the at least one second point-surface constraint, the reprojection constraint and the depth constraint as much as possible. Of course, among the at least one second point-surface constraint, the first point-surface constraint must be satisfied, and finally the optimized value that minimizes the errors in the first point-surface constraint, the reprojection constraint and the depth constraint is obtained.

[0089] Exemplarily, the method further includes: marginalizing the point-surface constraint conditions obtained based on the second historical frame before the first historical frame. The second historical frame is an earlier frame than the first historical frame. For example, the current image frame is the 11th frame, and the first historical frame is the 3rd to 10th frames, then the frames before the 3rd frame are all the second historical frames. Since the second historical frame is a relatively old frame, the point-surface constraint conditions obtained based on these frames have little effect on the current positioning or tracking. Therefore, they can be marginalized to ensure the timeliness of the point-surface constraint conditions.

[0090] Exemplarily, uniformly sampling the image frame to obtain at least one 2D feature point includes:

[0091] Remove edge pixels of the image frame;

[0092] The sampling distance is calculated according to the size of the image frame and the preset sampling number;

[0093] The pixels that have not been removed in the image frame are sampled according to the sampling distance to obtain at least one 2D feature point.

[0094] In the present application embodiment, Figure 6 As shown in FIG. 1 , assuming that the size of the current image frame is M*N pixels, S pixels at the edge of the image frame are removed, S is an integer greater than or equal to 1, and the remaining pixels are expressed as (MS)*(NS). The preset sampling number is Q, Q is an integer greater than or equal to 3, and the sampling distance is According to the sampling distance, the pixels that have not been removed in the image frame are uniformly sampled to obtain the following Figure 6 At least one 2D feature point shown in .

[0095] It can be seen that in the embodiment of the present application, a sparse plane is constructed by using a sparse point cloud obtained based on the currently input image frame, and the first association information between the target 3D mesh generated by the front end based on the image frame and the sparse plane is obtained; according to the first association information and the current first pose of the carrier, the first point-surface constraint is obtained; according to the first point-surface constraint, the reprojection constraint and the depth constraint obtained based on the image frame and the previous frame of the image frame, the optimized value of the first pose is obtained. Since the current input image frame is an RGBD image acquired by an RGBD image acquisition device, and the RGBD image usually contains observed 2D data, such as data provided by a depth sensor, and the observed 2D data is relatively accurate, then the sparse point cloud obtained based on the 2D data is more accurate than the estimated 3D data, thereby making the constructed sparse plane more accurate, which is beneficial to improving the accuracy of the point-to-surface association (i.e., the first association information). In addition, on the basis of the reprojection constraints and depth constraints given at the front end of the SLAM system, point-to-surface constraints based on point-to-surface association and the first pose (i.e., the first point-to-surface constraint condition) are added. By continuously optimizing the first pose so that the pose of the carrier satisfies the above three constraints, it is beneficial to reduce the deviation between the estimated and observed poses, thereby improving the positioning accuracy.

[0096] See also Figure 7 , Figure 7 A flow chart of another planar fusion positioning method provided in an embodiment of the present application is shown as follows: Figure 7 As shown, it includes steps 701-705:

[0097] 701: constructing a sparse plane using a sparse point cloud obtained based on a currently input image frame;

[0098] 702: Acquire first association information between a target 3D mesh and a sparse plane generated by a front end based on an image frame;

[0099] 703: Obtain second association information between a historical sparse plane constructed based on a historical sparse point cloud obtained in a previous frame of the image frame and a historical 3Dmesh generated by a front end based on the previous frame, and a second posture of the carrier in the previous frame;

[0100] 704: constructing a point-surface optimization model using the first association information, the first pose, the second association information, and the second pose to obtain a first point-surface constraint condition;

[0101] 705: Obtain an optimized value of the first pose according to the first point-plane constraint, the reprojection constraint obtained based on the image frame and a previous frame of the image frame, and the depth constraint.

[0102] The specific implementation of steps 701-705 is as follows: Figure 2 Relevant descriptions have been given in the illustrated embodiment and will not be repeated here.

[0103] It can be seen that the embodiment of the present application uses a sparse point cloud obtained based on the currently input image frame to construct a sparse plane; obtains first association information between the target 3D mesh generated by the front end based on the image frame and the sparse plane; obtains second association information obtained based on the previous frame of the image frame and the second posture of the carrier in the previous frame; uses the first association information, the first posture, the second association information and the second posture to construct a point-surface optimization model to obtain a first point-surface constraint; obtains the optimization value of the first posture based on the first point-surface constraint, the reprojection constraint and the depth constraint obtained based on the image frame and the previous frame of the image frame. Since the current input image frame is an RGBD image acquired by an RGBD image acquisition device, and the RGBD image usually contains observed 2D data, such as data provided by a depth sensor, and the observed 2D data is relatively accurate, then the sparse point cloud obtained based on the 2D data is more accurate than the estimated 3D data, thereby making the constructed sparse plane more accurate, which is beneficial to improving the accuracy of the point-surface association between the current input image frame and its adjacent frame (i.e., the previous frame), thereby making the accuracy of the constructed first point-surface constraint condition higher; in addition, on the basis of the reprojection constraint and depth constraint given at the front end of the SLAM system, the first point-surface constraint condition obtained based on the point-surface association and the first pose is also added. By continuously optimizing the first pose so that the pose of the carrier meets the above three constraints, it is beneficial to reduce the deviation between the estimated value and the observed value of the pose, thereby improving the positioning accuracy.

[0104] based on Figure 2 or Figure 7 The present application also provides a planar fusion positioning device, see Figure 8 , Figure 8A schematic diagram of the structure of a planar fusion positioning device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the device includes: an acquisition unit 801 and a processing unit 802;

[0105] An acquisition unit 801 is used to construct a sparse plane using a sparse point cloud obtained based on a currently input image frame, and acquire first association information between a target 3D mesh generated by a front end based on the image frame and the sparse plane;

[0106] The processing unit 802 is used to obtain a first point-surface constraint condition according to the first association information and the first current pose of the carrier;

[0107] The processing unit 802 is further configured to obtain an optimized value of the first pose according to the first point-surface constraint, the reprojection constraint obtained based on the image frame and the previous frame of the image frame, and the depth constraint.

[0108] In a possible implementation manner, in terms of obtaining the first point-surface constraint condition according to the first association information and the first current pose of the carrier, the processing unit 802 is specifically configured to:

[0109] Obtaining second association information between the historical sparse plane constructed based on the historical sparse point cloud obtained in the previous frame and the historical 3Dmesh generated by the front end based on the previous frame, as well as the second position and posture of the carrier in the previous frame;

[0110] The first association information, the first pose, the second association information and the second pose are used to construct a point-surface optimization model to obtain a first point-surface constraint condition.

[0111] In a possible implementation, in terms of obtaining an optimized value of the first pose according to the first point-surface constraint, the reprojection constraint and the depth constraint obtained based on the image frame and the previous frame of the image frame, the processing unit 802 is specifically used to:

[0112] Acquire at least one second point-surface constraint condition, the at least one second point-surface constraint condition includes the first point-surface constraint condition, the point-surface constraint condition other than the first point-surface constraint condition in the at least one second point-surface constraint condition is obtained based on target association information, and the target association information is association information obtained based on a first historical frame of a preset number of image frames;

[0113] The first pose is optimized, and a pose estimate satisfying at least one second point-surface constraint, a reprojection constraint, and a depth constraint is used as an optimized value.

[0114] In a possible implementation, the processing unit 802 is further configured to: marginalize the point-surface constraint condition obtained based on the second historical frame before the first historical frame.

[0115] In a possible implementation, the processing unit 802 is further configured to:

[0116] Uniformly sample the image frame to obtain at least one 2D feature point;

[0117] Determine a point in a depth map corresponding to at least one 2D feature point in the image frame;

[0118] The point of at least one 2D feature point in the depth map corresponding to the image frame is mapped to the 3D space to obtain a sparse point cloud.

[0119] In a possible implementation, the processing unit 802 is further configured to: construct a 2D mesh using at least one 2D feature point;

[0120] In terms of constructing a sparse plane using a sparse point cloud obtained based on a currently input image frame, the processing unit 802 is specifically used to:

[0121] Based on the correspondence between the 2D feature points in the 2D mesh and the 3D points in the sparse point cloud, the 3D points are used to construct the 3D mesh corresponding to the 2D mesh;

[0122] Cluster the 3D mesh based on the first pose to obtain a sparse plane.

[0123] In a possible implementation, in terms of uniformly sampling the image frame to obtain at least one 2D feature point, the processing unit 802 is specifically configured to:

[0124] Remove edge pixels of the image frame;

[0125] The sampling distance is calculated according to the size of the image frame and the preset sampling number;

[0126] The pixels that have not been removed in the image frame are sampled according to the sampling distance to obtain at least one 2D feature point.

[0127] In a possible implementation manner, in terms of obtaining first association information between a target 3D mesh and a sparse plane generated by a front end based on an image frame, the processing unit 802 is specifically configured to:

[0128] Get the difference between the direction of the target 3Dmesh and the sparse plane;

[0129] When the difference is less than or equal to the preset value and at least one 3D point constituting the target 3D mesh is on the sparse plane, the target 3D mesh is associated with the sparse plane to obtain first association information.

[0130] It can be seen that in Figure 8In the plane fusion positioning device shown, a sparse plane is constructed using a sparse point cloud obtained based on the currently input image frame, and first association information between the target 3D mesh generated by the front end based on the image frame and the sparse plane is obtained; based on the first association information and the current first pose of the carrier, a first point-surface constraint is obtained; based on the first point-surface constraint, the reprojection constraint and the depth constraint obtained based on the image frame and the previous frame of the image frame, an optimized value of the first pose is obtained. Since the current input image frame is an RGBD image acquired by an RGBD image acquisition device, and the RGBD image usually contains observed 2D data, such as data provided by a depth sensor, and the observed 2D data is relatively accurate, then the sparse point cloud obtained based on the 2D data is more accurate than the estimated 3D data, thereby making the constructed sparse plane more accurate, which is beneficial to improving the accuracy of the point-to-surface association (i.e., the first association information). In addition, on the basis of the reprojection constraints and depth constraints given at the front end of the SLAM system, point-to-surface constraints based on point-to-surface association and the first pose (i.e., the first point-to-surface constraint condition) are added. By continuously optimizing the first pose so that the pose of the carrier satisfies the above three constraints, it is beneficial to reduce the deviation between the estimated and observed poses, thereby improving the positioning accuracy.

[0131] According to one embodiment of the present application, Figure 8 The various units in the plane fusion positioning device shown can be respectively or completely merged into one or several other units to constitute, or one (some) of the units can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present application, the plane fusion positioning device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0132] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 or Figure 7 A computer program (including program code) for each step involved in the corresponding method shown in Figure 8 The plane fusion positioning device shown in the embodiment of the present application is implemented by the plane fusion positioning method. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0133] Based on the description of the above method embodiment and device embodiment, the present application embodiment also provides an electronic device. Fig. 9 The electronic device at least includes a processor 901, an input device 902, an output device 903, and a computer storage medium 904. The processor 901, the input device 902, the output device 903, and the computer storage medium 904 in the electronic device may be connected via a bus or other means.

[0134] The computer storage medium 904 may be stored in the memory of the electronic device, the computer storage medium 904 is used to store a computer program, the computer program includes program instructions, and the processor 901 is used to execute the program instructions stored in the computer storage medium 904. The processor 901 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0135] In one embodiment, the processor 901 of the electronic device provided in the embodiment of the present application can be used to perform a series of plane fusion positioning processes:

[0136] A sparse plane is constructed using a sparse point cloud obtained based on a currently input image frame, and first association information between a target 3D mesh generated by a front end based on the image frame and the sparse plane is obtained;

[0137] Obtaining a first point-surface constraint condition according to the first association information and the first current pose of the carrier;

[0138] According to the first point-surface constraint, the reprojection constraint obtained based on the image frame and the previous frame of the image frame, and the depth constraint, an optimized value of the first pose is obtained.

[0139] In another embodiment, the processor 901 obtains the first point-surface constraint condition according to the first association information and the first current pose of the carrier, including:

[0140] Obtaining the second association information of the historical sparse plane constructed based on the historical sparse point cloud obtained in the previous frame and the historical 3Dmesh generated by the front end based on the previous frame and the second posture of the carrier in the previous frame;

[0141] The first association information, the first pose, the second association information and the second pose are used to construct a point-surface optimization model to obtain a first point-surface constraint condition.

[0142] In another embodiment, the processor 901 executes the first point-plane constraint, the reprojection constraint and the depth constraint obtained based on the image frame and the previous frame of the image frame to obtain the optimization value of the first pose, including:

[0143] Acquire at least one second point-surface constraint condition, the at least one second point-surface constraint condition includes the first point-surface constraint condition, the point-surface constraint condition other than the first point-surface constraint condition in the at least one second point-surface constraint condition is obtained based on target association information, and the target association information is association information obtained based on a first historical frame of a preset number of image frames;

[0144] The first pose is optimized, and a pose estimate satisfying at least one second point-surface constraint, a reprojection constraint, and a depth constraint is used as an optimized value.

[0145] In yet another embodiment, the processor 901 is further configured to: marginalize the point-surface constraint condition obtained based on the second historical frame before the first historical frame.

[0146] In yet another embodiment, before constructing a sparse plane using a sparse point cloud obtained based on a currently input image frame, the processor 901 is further configured to:

[0147] Uniformly sample the image frame to obtain at least one 2D feature point;

[0148] Determine a point in a depth map corresponding to at least one 2D feature point in the image frame;

[0149] The point of at least one 2D feature point in the depth map corresponding to the image frame is mapped to the 3D space to obtain a sparse point cloud.

[0150] In yet another embodiment, after obtaining at least one 2D feature point, the processor 901 is further configured to:

[0151] Construct a 2D mesh using at least one 2D feature point;

[0152] The processor 901 executes the construction of a sparse plane using a sparse point cloud obtained based on a currently input image frame, including:

[0153] Based on the correspondence between the 2D feature points in the 2D mesh and the 3D points in the sparse point cloud, the 3D points are used to construct the 3D mesh corresponding to the 2D mesh;

[0154] Cluster the 3D mesh based on the first pose to obtain a sparse plane.

[0155] In another embodiment, the processor 901 performs uniform sampling on the image frame to obtain at least one 2D feature point, including:

[0156] Remove edge pixels of the image frame;

[0157] The sampling distance is calculated according to the size of the image frame and the preset sampling number;

[0158] The pixels that have not been removed in the image frame are sampled according to the sampling distance to obtain at least one 2D feature point.

[0159] In another embodiment, the processor 901 executes obtaining first association information between the target 3D mesh and the sparse plane generated by the front end based on the image frame, including:

[0160] Get the difference between the direction of the target 3Dmesh and the sparse plane;

[0161] When the difference is less than or equal to the preset value and at least one 3D point constituting the target 3D mesh is on the sparse plane, the target 3D mesh is associated with the sparse plane to obtain first association information.

[0162] It can be seen that in Fig. 9 In the electronic device shown, a sparse plane is constructed using a sparse point cloud obtained based on a currently input image frame, and first association information between a target 3D mesh generated by a front end based on the image frame and the sparse plane is obtained; a first point-surface constraint is obtained based on the first association information and the current first pose of the carrier; and an optimized value of the first pose is obtained based on the first point-surface constraint, the reprojection constraint obtained based on the image frame and the previous frame of the image frame, and the depth constraint. Since the current input image frame is an RGBD image acquired by an RGBD image acquisition device, and the RGBD image usually contains observed 2D data, such as data provided by a depth sensor, and the observed 2D data is relatively accurate, then the sparse point cloud obtained based on the 2D data is more accurate than the estimated 3D data, thereby making the constructed sparse plane more accurate, which is beneficial to improving the accuracy of the point-to-surface association (i.e., the first association information). In addition, on the basis of the reprojection constraints and depth constraints given at the front end of the SLAM system, point-to-surface constraints based on point-to-surface association and the first pose (i.e., the first point-to-surface constraint condition) are added. By continuously optimizing the first pose so that the pose of the carrier satisfies the above three constraints, it is beneficial to reduce the deviation between the estimated and observed poses, thereby improving the positioning accuracy.

[0163] Exemplarily, the electronic device may include but is not limited to a processor 901, an input device 902, an output device 903, and a computer storage medium 904, the input device 902 may be a keyboard, a touch screen, etc., and the output device 903 may be a speaker, a display, a radio frequency transmitter, etc. Those skilled in the art will appreciate that the schematic diagram is merely an example of an electronic device and does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0164] It should be noted that, since the processor 901 of the electronic device executes the computer program to implement the above Figure 2 or Figure 7 The steps in the plane fusion positioning method shown are as follows: Figure 2 or Figure 7 The embodiments of the planar fusion positioning method shown are all applicable to the electronic device and can achieve the same or similar beneficial effects.

[0165] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer storage medium here can include both the built-in storage medium in the terminal and the extended storage medium supported by the terminal. The computer storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor 901 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory; optionally, it can also be at least one computer storage medium located away from the aforementioned processor 901. In one embodiment, the processor 901 can load and execute one or more instructions stored in the computer storage medium to implement the corresponding steps of the above-mentioned planar fusion positioning method.

[0166] Exemplarily, the computer program of the computer storage medium includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0167] It should be noted that, since the computer program of the computer storage medium implements the steps in the above-mentioned plane fusion positioning method when executed by the processor, all embodiments of the above-mentioned plane fusion positioning method are applicable to the computer storage medium and can achieve the same or similar beneficial effects.

[0168] The embodiment of the present application further provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the steps in the above-mentioned planar fusion positioning method. The computer program product may be a software installation package.

[0169] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A planar fusion positioning method, characterized in that: The method comprises: A sparse plane is constructed using a sparse point cloud obtained based on a currently input image frame, and first association information between a target three-dimensional grid generated by a front end based on the image frame and the sparse plane is obtained; Obtaining a first point-surface constraint condition according to the first association information and the first current pose of the carrier; Obtaining an optimized value of the first posture according to the first point-plane constraint, a reprojection constraint obtained based on the image frame and a previous frame of the image frame, and a depth constraint; The image frame is an RGBD image; before constructing a sparse plane using a sparse point cloud obtained based on the currently input image frame, the method further includes: Uniformly sampling the image frame to obtain at least one two-dimensional feature point; Determine a point of the at least one two-dimensional feature point in a depth map corresponding to the image frame; Mapping the at least one two-dimensional feature point in the depth map corresponding to the image frame to a three-dimensional space to obtain the sparse point cloud; The acquiring first association information between the target three-dimensional grid and the sparse plane generated by the front end based on the image frame includes: Obtaining a difference between the direction of the target three-dimensional grid and the direction of the sparse plane; When the difference is less than or equal to a preset value and at least one three-dimensional point constituting the target three-dimensional grid is on the sparse plane, the target three-dimensional grid is associated with the sparse plane to obtain the first association information.

2. The method according to claim 1, characterized in that The obtaining of the first point-surface constraint condition according to the first association information and the first current pose of the carrier includes: Acquire second association information between a historical sparse plane constructed based on the historical sparse point cloud obtained in the previous frame and a historical three-dimensional grid generated by the front end based on the previous frame, and a second posture of the carrier in the previous frame; A point-plane optimization model is constructed using the first association information, the first posture, the second association information, and the second posture to obtain the first point-plane constraint condition.

3. The method according to claim 1 or 2, characterized in that: The step of obtaining the optimized value of the first posture according to the first point-plane constraint, the reprojection constraint and the depth constraint obtained based on the image frame and a previous frame of the image frame comprises: Acquire at least one second point-plane constraint condition, the at least one second point-plane constraint condition includes the first point-plane constraint condition, the point-plane constraint condition other than the first point-plane constraint condition in the at least one second point-plane constraint condition is obtained based on target association information, and the target association information is association information obtained based on a first historical frame of a preset number before the image frame; The first posture is optimized, and a posture estimation value satisfying the at least one second point-surface constraint, the reprojection constraint, and the depth constraint is used as the optimized value.

4. The method according to claim 3, characterized in that The method further comprises: The marginalization is based on point and surface constraints obtained from a second historical frame before the first historical frame.

5. The method according to claim 1, characterized in that After obtaining at least one two-dimensional feature point, the method further includes: constructing a two-dimensional grid using the at least one two-dimensional feature point; The method of constructing a sparse plane using a sparse point cloud obtained based on a currently input image frame includes: Based on the correspondence between the two-dimensional feature points in the two-dimensional grid and the three-dimensional points in the sparse point cloud, constructing a three-dimensional grid corresponding to the two-dimensional grid using the three-dimensional points; The three-dimensional grid is clustered based on the first posture to obtain the sparse plane.

6. The method according to claim 5, characterized in that The image frame is uniformly sampled to obtain at least one two-dimensional feature point, including: Removing edge pixels of the image frame; The sampling distance is calculated according to the size of the image frame and the preset sampling number; The pixels that have not been removed in the image frame are sampled according to the sampling distance to obtain the at least one two-dimensional feature point.

7. A planar fusion positioning device, characterized in that: The device comprises an acquisition unit and a processing unit; The acquisition unit is used to construct a sparse plane using a sparse point cloud obtained based on a currently input image frame, and acquire first association information between a target three-dimensional grid generated by a front end based on the image frame and the sparse plane; The processing unit is used to obtain a first point-surface constraint condition according to the first association information and the first current pose of the carrier; The processing unit is further used to obtain an optimized value of the first posture according to the first point-plane constraint, a reprojection constraint obtained based on the image frame and a previous frame of the image frame, and a depth constraint; The image frame is an RGBD image; the processing unit is further used to: uniformly sample the image frame to obtain at least one two-dimensional feature point; Determine a point of the at least one two-dimensional feature point in a depth map corresponding to the image frame; Mapping the at least one two-dimensional feature point in the depth map corresponding to the image frame to a three-dimensional space to obtain the sparse point cloud; In terms of acquiring first association information between a target three-dimensional grid generated by a front end based on the image frame and the sparse plane, the processing unit is specifically used to: Obtaining a difference between the direction of the target three-dimensional grid and the direction of the sparse plane; When the difference is less than or equal to a preset value and at least one three-dimensional point constituting the target three-dimensional grid is on the sparse plane, the target three-dimensional grid is associated with the sparse plane to obtain the first association information.

8. An electronic device, comprising an input device and an output device, characterized in that: Also includes: a processor adapted to implement one or more instructions; as well as, A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1-6.

Images