Plane construction method, device, electronic device and storage medium

Abstract

The present application provides a plane construction method, apparatus, electronic device and storage medium. Among them, the method includes: obtaining a first two-dimensional grid and a second two-dimensional grid of a currently input image frame; obtaining at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid and the pose information of the carrier; in the case where there is no structured plane currently, generating a structured plane according to the pose information and the at least one target three-dimensional grid. Among them, the first two-dimensional grid refers to a two-dimensional grid constructed based on two-dimensional feature points extracted by the front end, and the second two-dimensional grid refers to a two-dimensional grid constructed based on two-dimensional feature points extracted by the back end. In this way, using the feature points with higher quality in the back end and the feature points with faster tracking speed in the front end to construct a structured plane is beneficial to improving the construction efficiency of the structured plane.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular, to a method and apparatus for plane construction, an electronic device, and a storage medium. Background Art

[0002] With the popularization of artificial intelligence, related applications of computer vision technology have been continuously introduced, such as visualization of AR (Augmented Reality) or VR (Virtual Reality) models, positioning systems for drones and autonomous driving, and so on. Structured plane construction is a key technology in the above applications and is also an important issue in SLAM (simultaneous localization and mapping). Currently, structured plane construction mainly relies on dense data from depth sensors, but the construction efficiency of this solution is low in structured environments. Summary of the Invention

[0003] Embodiments of this application provide a method and apparatus for plane construction, an electronic device, and a storage medium.

[0004] In a first aspect of the embodiments of this application, a method for plane construction is provided, and the method includes:

[0005] Obtain a first two-dimensional grid and a second two-dimensional grid of a currently input image frame;

[0006] Obtain at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier;

[0007] In the case where there is no structured plane currently, generate a structured plane according to the pose information and the at least one target three-dimensional grid.

[0008] In combination with the first aspect, in a possible implementation manner, obtaining a first two-dimensional grid and a second two-dimensional grid of a currently input image frame includes:

[0009] Obtain a first two-dimensional feature point and a second two-dimensional feature point of the image frame, where the first two-dimensional feature point is a front-end feature point obtained by the front end of the simultaneous localization and mapping system through feature extraction of the image frame, and the second two-dimensional feature point is a back-end feature point obtained by the back end of the simultaneous localization and mapping system through feature extraction of the image frame;

[0010] Perform triangulation on the first two-dimensional feature points to obtain the first two-dimensional grid, and perform triangulation on the second two-dimensional feature points to obtain the second two-dimensional grid.

[0011] In combination with the first aspect, in a possible implementation manner, obtaining at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier includes:

[0012] Obtaining at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information;

[0013] Using the pose information to screen the at least one three-dimensional grid to obtain the at least one target three-dimensional grid.

[0014] In combination with the first aspect, in a possible implementation manner, obtaining at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information includes:

[0015] Obtaining a first three-dimensional point corresponding to a first two-dimensional feature point in the first two-dimensional grid;

[0016] Obtaining a second three-dimensional point corresponding to a second two-dimensional feature point in the second two-dimensional grid;

[0017] Using the pose information to screen the first three-dimensional point and the second three-dimensional point, and respectively projecting the remaining first three-dimensional points and the remaining second three-dimensional points after screening to obtain the at least one three-dimensional grid.

[0018] In combination with the first aspect, in a possible implementation manner, using the pose information to screen the at least one three-dimensional grid to obtain at least one target three-dimensional grid includes:

[0019] Determining a first distance between a three-dimensional grid in the at least one three-dimensional grid and the carrier according to the pose information;

[0020] Deleting the three-dimensional grids in the at least one three-dimensional grid whose first distance is greater than or equal to a distance threshold, and screening out the three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane from the remaining three-dimensional grids in the at least one three-dimensional grid;

[0021] Determining the three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane as the at least one target three-dimensional grid.

[0022] In combination with the first aspect, in a possible implementation manner, generating a structured plane according to the pose information and the at least one target three-dimensional grid includes:

[0023] Obtaining a second distance between any two target three-dimensional grids in the at least one target three-dimensional grid based on the pose information;

[0024] Clustering the at least one target three-dimensional grid based on the second distance to generate a structured plane.

[0025] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0026] In the case where a structured plane currently exists, associate the target 3D meshes in the at least one target 3D mesh with the existing structured plane.

[0027] In combination with the first aspect, in a possible implementation manner, associating the target 3D meshes in the at least one target 3D mesh with the existing structured plane includes:

[0028] Determine the difference in direction between the target 3D meshes in the at least one target 3D mesh and the existing structured plane;

[0029] In the case where the difference is less than or equal to a first preset value and at least one 3D point constituting the target 3D mesh is on the existing structured plane, associate the target 3D mesh with the existing structured plane.

[0030] In combination with the first aspect, in a possible implementation manner, after obtaining the at least one target 3D mesh, the method further includes:

[0031] Update the 3D mesh obtained before the image frame using the at least one target 3D mesh, where the number of the at least one target 3D mesh is less than or equal to a second preset value.

[0032] A second aspect of the embodiments of the present application provides a plane construction device, and the device includes:

[0033] An acquisition unit, configured to acquire a first 2D mesh and a second 2D mesh of a currently input image frame;

[0034] A processing unit, configured to obtain at least one target 3D mesh according to the first 2D mesh, the second 2D mesh, and the pose information of the carrier;

[0035] The processing unit is further configured to generate a structured plane according to the pose information and the at least one target 3D mesh in the case where no structured plane currently exists.

[0036] A third aspect of the embodiments of the present application provides an electronic device, and the electronic device includes an input device and an output device, and further includes a processor adapted to implement one or more instructions; and a computer storage medium storing one or more instructions, the one or more instructions being adapted to be loaded and executed by the processor to perform the steps in the method described in the first aspect above.

[0037] In a fourth aspect of the embodiments of the present application, a computer storage medium is provided. The computer storage medium stores one or more instructions, and the one or more instructions are adapted to be loaded and executed by a processor to perform the steps in the method described in the first aspect above.

[0038] In a fifth aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes a computer program, and the computer program is operable to cause a computer to perform the steps in the method described in the first aspect above. The computer program product can be a software installation package.

[0039] It can be seen that in the embodiments of the present application, by obtaining a first two-dimensional grid and a second two-dimensional grid of the currently input image frame; obtaining at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier; in the case where there is no structured plane currently, generating a structured plane according to the pose information and the at least one target three-dimensional grid. Among them, the first two-dimensional grid refers to a two-dimensional grid constructed based on two-dimensional feature points extracted by the front end, and the second two-dimensional grid refers to a two-dimensional grid constructed based on two-dimensional feature points extracted by the back end. In this way, using the feature points with higher quality in the back end and the feature points with faster tracking speed in the front end to construct the structured plane is beneficial to improving the construction efficiency of the structured plane. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is an architecture diagram of an application environment provided by the embodiments of the present application;

[0042] Figure 2 It is a schematic flowchart of a plane construction method provided by the embodiments of the present application;

[0043] Figure 3 It is a schematic diagram of obtaining a target three-dimensional grid provided by the embodiments of the present application;

[0044] Figure 4 It is a schematic diagram of constructing a structured plane provided by the embodiments of the present application;

[0045] Figure 5 It is a schematic flowchart of another plane construction method provided by the embodiments of the present application;

[0046] Figure 6Schematic structural diagram of a plane construction device provided by an embodiment of the present application;

[0047] Figure 7 Schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0048] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0049] The terms "including" and "having" and any variations thereof appearing in the specification, claims and drawings of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products or devices. In addition, the terms "first", "second", "third", etc. are used to distinguish different objects, rather than to describe a specific order.

[0050] Referring to "embodiment" in the present application means that a specific feature, structure or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in the present application may be combined with other embodiments.

[0051] An embodiment of the present application proposes a plane construction method, which can be implemented based on Figure 1 the application environment shown, such as Figure 1 shown, the application environment includes an electronic device 101 and at least one image acquisition device 102 and at least one terminal device 103 communicatively connected to the electronic device 101. Among them, the above-mentioned communicative connection manners may be serial port connection, wireless network connection, wired network connection, etc.

[0052] Among them, the electronic device 101 can be used to provide an operating environment and foundation for the SLAM system, and support the operation of the front-end algorithm and the back-end algorithm of the SLAM system. The electronic device 101 involved in the embodiments of the present application can include various devices with the ability to run program codes and communicate. For example, the electronic device 101 can be an independent physical server, or a server cluster or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0053] Among them, the image acquisition device 102 can be used to collect video image data and transmit the collected video stream to the electronic device 101, so that the electronic device 101 executes the plane construction method in the embodiments of the present application, and uses the feature points of the front end and the feature points of the back end of the SLAM system to construct a structured plane. The image acquisition device 102 involved in the embodiments of the present application can be a camera (or camera) without depth capture function, a camera (or camera) with depth capture function, a smart phone, etc. The image acquisition device 102 can exist independently, such as a camera fixed somewhere, or it can depend on certain objects, such as the shooting device of a drone, the driving recorder of a vehicle, the camera of a sweeping robot, etc.

[0054] Among them, the terminal device 103 can be used to display the image frames decoded by the electronic device 101, and provide a visualization interface for the SLAM system to visualize the generated structured plane. Further, the terminal device 103 can also be used to visualize the construction process of the structured plane. For example, it can display the constructed two-dimensional (2D, 2-Dimension) grid, three-dimensional (3D, 3-Dimension) grid, etc. The terminal device 103 can be a device with a display function, such as a computer, AR or VR glasses, etc., or an independent display.

[0055] The plane construction method in the embodiments of the present application can be used to realize the display of models and effects on AR or VR and the construction and display of a structured environment. Further, it can also be used to optimize the performance and tracking accuracy of the SLAM system.

[0056] The following elaborates in detail on the plane construction method provided by the embodiments of the present application in conjunction with the relevant drawings.

[0057] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a plane construction method provided by the embodiments of the present application. This method is applied to an electronic device, as Figure 2 shown, and includes steps 201-203:

[0058] 201: Obtain the first two-dimensional grid and the second two-dimensional grid of the currently input image frame.

[0059] In the embodiments of the present application, the currently input image frame can be an image frame in a real-time video stream collected by an image acquisition device, or an image frame in a video prepared in advance by a researcher during algorithm development. The first two-dimensional grid refers to a two-dimensional mesh (2D mesh) constructed based on 2D feature points extracted by the front end of the SLAM system, and the second two-dimensional grid refers to a two-dimensional mesh constructed based on 2D feature points extracted by the back end of the SLAM system. Among them, the front end of the SLAM system refers to visual-inertial odometry (VIO), and the selected 2D feature points can be tracked corner points, and such feature points have the characteristic of fast speed, which is beneficial to keeping the generation speed of the structured plane synchronized with the video stream.

[0060] Exemplarily, the above-mentioned obtaining the first two-dimensional grid and the second two-dimensional grid of the currently input image frame includes:

[0061] Obtain the first two-dimensional feature points and the second two-dimensional feature points of the image frame;

[0062] Perform triangulation on the first two-dimensional feature points to obtain the first two-dimensional grid, and perform triangulation on the second two-dimensional feature points to obtain the second two-dimensional grid.

[0063] In the embodiments of the present application, the first two-dimensional feature points are the front-end 2D feature points obtained by the front end of the SLAM system through feature extraction of the image frame; the second two-dimensional feature points are the back-end 2D feature points obtained by the back end of the SLAM system through feature extraction of the image frame. In some embodiments, they can also be the 2D feature points passed from the front end to the back end. Perform triangulation on the first two-dimensional feature points and the second two-dimensional feature points respectively to obtain the corresponding first two-dimensional grid and second two-dimensional grid. Exemplarily, the triangulation can adopt Delaunay triangulation.

[0064] 202: Obtain at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid and the pose information of the carrier.

[0065] In the embodiments of the present application, the carrier refers to an image acquisition device, such as a camera, and the pose information of the carrier can be calculated by the SLAM system. Exemplarily, the above-mentioned obtaining at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid and the pose information of the carrier includes:

[0066] Obtain at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid and the pose information;

[0067] Use the pose information to screen the at least one three-dimensional grid to obtain the at least one target three-dimensional grid.

[0068] In the embodiments of the present application, the first three-dimensional point corresponding to the first two-dimensional feature point in the first two-dimensional grid can be obtained through the SLAM system. The first three-dimensional point refers to the 3D feature point in the 3D point cloud corresponding to the first two-dimensional grid, and the first three-dimensional point has a mapping relationship with the first two-dimensional feature point. Similarly, the second three-dimensional point corresponding to the second two-dimensional feature point in the second two-dimensional grid can be obtained through the SLAM system. The second three-dimensional point refers to the 3D feature point in the 3D point cloud corresponding to the second two-dimensional grid, and the second three-dimensional point has a mapping relationship with the second two-dimensional feature point. Use the pose information to screen the first three-dimensional points. In the SLAM system, the distance between any two three-dimensional points in the first three-dimensional points can be estimated through the pose information of the carrier. Based on this distance, some three-dimensional points in the first three-dimensional points that are relatively far from other three-dimensional points can be deleted, and the remaining first three-dimensional points are the three-dimensional points with relatively close distances. Similar to the first three-dimensional points, the pose information of the carrier can be used to screen the second three-dimensional points, and the remaining second three-dimensional points are the three-dimensional points with relatively close distances. For the remaining first three-dimensional points and the remaining second three-dimensional points, the camera projection model is respectively used to project them to obtain at least one three-dimensional grid (3D mesh).

[0069] Exemplarily, as Figure 3 shown, the above-mentioned use of the pose information to screen the at least one three-dimensional grid to obtain at least one target three-dimensional grid includes steps 301-303:

[0070] 301: Determine the first distance between the three-dimensional grid in the at least one three-dimensional grid and the carrier according to the pose information;

[0071] 302: Delete the three-dimensional grids in the at least one three-dimensional grid whose first distance is greater than or equal to the distance threshold, and screen out the three-dimensional grids of the horizontal plane and the vertical plane from the remaining three-dimensional grids in the at least one three-dimensional grid;

[0072] It should be understood that the three-dimensional grids of the horizontal plane and the vertical plane are set based on experience. The embodiments of the present application should include but are not limited to the horizontal plane and the vertical plane. In some embodiments, three-dimensional grids in other directions can also be screened out.

[0073] 303: Determine the three-dimensional grids of the horizontal plane and the vertical plane as the at least one target three-dimensional grid.

[0074] In the embodiments of the present application, the distance (i.e., the first distance) between the carrier and the three-dimensional grids in at least one three-dimensional grid can be estimated according to the pose information of the carrier. The distance threshold can be set according to empirical values. For example, if the carrier is on the first floor, the three-dimensional grids on the tenth floor or above can be deleted, and the three-dimensional grids with a relatively close first distance to the carrier are retained. In addition, for the retained three-dimensional grids, their directions are further screened, and the three-dimensional grids other than the horizontal plane and the vertical plane are deleted, and only the three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane are retained. The three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane are the at least one target three-dimensional grid.

[0075] In this embodiment, the 3D point clouds after mapping the front-end 2D feature points and the back-end 2D feature points are both sparse point clouds. The sparse point clouds are more conducive to improving the generation speed of the three-dimensional grid compared with the dense data of the depth sensor, and thus are conducive to improving the construction efficiency of the sparse structured plane.

[0076] 203: In the case where there is currently no structured plane, a structured plane is generated according to the pose information and the at least one target three-dimensional grid.

[0077] In the embodiments of the present application, the absence of a structured plane indicates that no structured plane has been generated before the currently input image frame. Exemplarily, the generation of the structured plane according to the pose information and the at least one target three-dimensional grid includes:

[0078] Obtaining a second distance between any two target three-dimensional grids in the at least one target three-dimensional grid based on the pose information;

[0079] Clustering the at least one target three-dimensional grid based on the second distance to generate a structured plane.

[0080] It should be understood that the SLAM system can estimate the distance (i.e., the second distance) between any two target three-dimensional grids in the at least one target three-dimensional grid based on the pose information of the carrier, and the electronic device clusters the target three-dimensional grids with the second distance less than or equal to a certain value to generate a sparse structured plane.

[0081] Exemplarily, the method further includes:

[0082] In the case where there is currently a structured plane, associating the target three-dimensional grids in the at least one target three-dimensional grid with the existing structured plane.

[0083] Exemplarily, the above-mentioned associating the target three-dimensional grids in the at least one target three-dimensional grid with the existing structured plane includes:

[0084] Determining the difference in direction between the target three-dimensional grids in the at least one target three-dimensional grid and the existing structured plane;

[0085] When the difference is less than or equal to a first preset value and at least one three-dimensional point constituting the target three-dimensional grid is on an existing structured plane, the target three-dimensional grid is associated with the existing structured plane.

[0086] In the embodiments of the present application, if a structured plane has been generated before the currently input image frame, the target three-dimensional grid in at least one target three-dimensional grid is associated with the structured plane. Specifically, two conditions need to be met for the association with the plane: First, the directions are similar. The difference between the direction of the target three-dimensional grid and the direction of the existing structured plane can be calculated. If the difference is less than or equal to a first preset value, it can be considered that the directions are similar. Here, the first preset value can be set according to empirical values. Second, at least one of the three three-dimensional points constituting the target three-dimensional grid is on the existing structured plane. In other words, at least one of the three three-dimensional points satisfies the plane equation of the existing structured plane. When the above two conditions are met, it is determined that the target three-dimensional grid can be associated with the existing structured plane. Specifically, when one or two of the three three-dimensional points are on the existing structured plane, associating the target three-dimensional grid with the existing structured plane is for expanding the plane boundary; when all three three-dimensional points are on the existing structured plane, associating the target three-dimensional grid with the existing structured plane can be used to represent that the target three-dimensional grid and the existing structured plane are the same plane.

[0087] In this embodiment, since the 2D feature points selected in the previous stage have the characteristic of fast speed, they can be used to synchronize the video stream and improve the speed of plane expansion.

[0088] Exemplarily, after obtaining at least one target three-dimensional grid, the method further includes:

[0089] Updating the three-dimensional grid obtained before the image frame by using the at least one target three-dimensional grid, where the number of the at least one target three-dimensional grid is less than or equal to a second preset value.

[0090] In the embodiments of the present application, for at least one target three-dimensional grid obtained at the current moment, it can be used to update the target three-dimensional grid obtained at the previous moment for temporal propagation. In addition, the number of target three-dimensional grids obtained at each moment cannot be greater than a second preset value to ensure the performance and efficiency of the SLAM system. Specifically, the number of target three-dimensional grids at each moment can be controlled by restricting the number of adjacent frames. For example, if the adjacent image frames are set to 10 frames (frames 1 - 10 respectively), after the 10th frame, new three-dimensional points will enter, and the three-dimensional points obtained based on the first frame will be discarded. By setting the number of adjacent image frames, the number of three-dimensional points is controlled, and finally, the number of target three-dimensional grids obtained at each moment is controlled.

[0091] In short, as Figure 4 shown, in the embodiments of the present application, for the currently input image frame, the 2D feature points selected by the front end of the SLAM system and the 2D feature points selected by the back end are respectively obtained. Based on the 2D feature points, the corresponding 2D grids are respectively constructed. Then, the 2D feature points in the 2D grids are mapped into 3D point clouds, and operations such as screening, projection, and re-screening are performed on the 3D point clouds to obtain at least one target 3D grid. Then, in the case where there is no structured plane currently, clustering is performed on the at least one target 3D grid to generate a new structured plane, thereby improving the construction efficiency of the structured plane and reducing the dependence of the structured plane construction on dense data.

[0092] It can be seen that in the embodiments of the present application, the first two-dimensional grid and the second two-dimensional grid of the currently input image frame are obtained; at least one target three-dimensional grid is obtained according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier; in the case where there is no structured plane currently, a structured plane is generated according to the pose information and the at least one target three-dimensional grid. Among them, the first two-dimensional grid refers to the two-dimensional grid constructed based on the two-dimensional feature points extracted by the front end, and the second two-dimensional grid refers to the two-dimensional grid constructed based on the two-dimensional feature points extracted by the back end. In this way, using the feature points with higher quality at the back end and the feature points with faster tracking speed at the front end for the construction of the structured plane is beneficial to improving the construction efficiency of the structured plane, thereby reducing the construction time of the structured plane. At the same time, the feature points with higher quality at the back end can be mapped into 3D point clouds with higher quality, and the 3D point clouds with higher quality are beneficial to ensuring the accuracy of the generated structured plane.

[0093] Please refer to Figure 5 , Figure 5 which is a schematic flowchart of another plane construction method provided by the embodiments of the present application. As Figure 5 shown, it includes steps 501 - 505:

[0094] 501: Obtain the first two-dimensional feature points and the second two-dimensional feature points of the currently input image frame;

[0095] Among them, the first two-dimensional feature points are the front-end feature points obtained by the front end of the simultaneous localization and mapping system through feature extraction of the image frame, and the second two-dimensional feature points are the back-end feature points obtained by the back end of the simultaneous localization and mapping system through feature extraction of the image frame;

[0096] 502: Perform triangulation on the first two-dimensional feature points to obtain a first two-dimensional grid;

[0097] 503: Perform triangulation on the second two-dimensional feature points to obtain a second two-dimensional grid;

[0098] 504: Obtain at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier;

[0099] 505: In the case where there is currently no structured plane, generate a structured plane according to the pose information and the at least one target three-dimensional grid.

[0100] Among them, the specific implementation manners of steps 501-505 are already described in the Figure 2 shown embodiments, and can achieve the same or similar beneficial effects, and will not be elaborated here.

[0101] Based on Figure 2 or Figure 5 the description of the method embodiments shown, the embodiments of the present application also provide a plane construction device. Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a plane construction device provided by an embodiment of the present application. As Figure 6 shown, the device includes:

[0102] An acquisition unit 601, configured to acquire the first two-dimensional grid and the second two-dimensional grid of the currently input image frame;

[0103] A processing unit 602, configured to obtain at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier;

[0104] The processing unit 602 is further configured to generate a structured plane according to the pose information and the at least one target three-dimensional grid in the case where there is currently no structured plane.

[0105] In a possible implementation manner, in terms of acquiring the first two-dimensional grid and the second two-dimensional grid of the currently input image frame, the acquisition unit 601 is specifically configured to:

[0106] Obtain a first two-dimensional feature point and a second two-dimensional feature point of the image frame. The first two-dimensional feature point is a front-end feature point obtained by the front end of the simultaneous localization and mapping system through feature extraction of the image frame, and the second two-dimensional feature point is a back-end feature point obtained by the back end of the simultaneous localization and mapping system through feature extraction of the image frame;

[0107] Perform triangulation on the first two-dimensional feature point to obtain the first two-dimensional grid, and perform triangulation on the second two-dimensional feature point to obtain the second two-dimensional grid.

[0108] In a possible implementation manner, in terms of obtaining at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier, the processing unit 602 is specifically configured to:

[0109] Obtain at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information;

[0110] Use the pose information to screen the at least one three-dimensional grid to obtain the at least one target three-dimensional grid.

[0111] In a possible implementation manner, in terms of obtaining at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information, the processing unit 602 is specifically configured to:

[0112] Obtain a first three-dimensional point corresponding to the first two-dimensional feature point in the first two-dimensional grid;

[0113] Obtain a second three-dimensional point corresponding to the second two-dimensional feature point in the second two-dimensional grid;

[0114] Use the pose information to screen the first three-dimensional point and the second three-dimensional point, and project the remaining first three-dimensional point and the remaining second three-dimensional point after screening respectively to obtain the at least one three-dimensional grid.

[0115] In a possible implementation manner, in terms of using the pose information to screen the at least one three-dimensional grid to obtain at least one target three-dimensional grid, the processing unit 602 is specifically configured to:

[0116] Determine a first distance between the three-dimensional grid in the at least one three-dimensional grid and the carrier according to the pose information;

[0117] Delete the three-dimensional grids in the at least one three-dimensional grid whose first distance is greater than or equal to the distance threshold, and screen out the three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane from the remaining three-dimensional grids in the at least one three-dimensional grid;

[0118] Determine the three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane as the at least one target three-dimensional grid.

[0119] In a possible implementation, in terms of generating a structured plane based on the pose information and the at least one target three-dimensional grid, the processing unit 602 is specifically configured to:

[0120] Obtain a second distance between any two target three-dimensional grids in the at least one target three-dimensional grid based on the pose information;

[0121] Cluster the at least one target three-dimensional grid based on the second distance to generate a structured plane.

[0122] In a possible implementation, the processing unit 602 is further configured to: when there is a structured plane currently, associate the target three-dimensional grid in the at least one target three-dimensional grid with the existing structured plane.

[0123] In a possible implementation, in terms of associating the target three-dimensional grid in the at least one target three-dimensional grid with the existing structured plane, the processing unit 602 is specifically configured to:

[0124] Determine the difference in direction between the target three-dimensional grid in the at least one target three-dimensional grid and the existing structured plane;

[0125] When the difference is less than or equal to a first preset value and at least one three-dimensional point constituting the target three-dimensional grid is on the existing structured plane, associate the target three-dimensional grid with the existing structured plane.

[0126] In a possible implementation, after obtaining the at least one target three-dimensional grid, the processing unit 602 is further configured to:

[0127] Update the three-dimensional grid obtained before the image frame by using the at least one target three-dimensional grid, where the number of the at least one target three-dimensional grid is less than or equal to a second preset value.

[0128] The plane construction device provided by the embodiments of the present application obtains a first two-dimensional grid and a second two-dimensional grid of the currently input image frame by setting the acquisition unit 601; obtains at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier by setting the processing unit 602, and generates a structured plane according to the pose information and the at least one target three-dimensional grid when there is no structured plane currently, thereby facilitating the improvement of the construction efficiency of the structured plane.

[0129] According to an embodiment of the present application, Figure 6Each unit in the planar construction device shown can be separately or all combined into one or several other units to form, or a certain one (or some) of the units can be further split into multiple smaller units in terms of function to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above units are divided based on logical functions. In practical applications, the function of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the planar construction device can also include other units. In practical applications, these functions can also be assisted by other units and can be realized by the cooperation of multiple units.

[0130] According to another embodiment of the present application, it can be achieved by running a computer program (including program code) capable of executing the respective steps involved in the corresponding method shown in Figure 2 or Figure 5 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM), to construct the planar construction device equipment shown in Figure 6 and to implement the planar construction method of the embodiments of the present application. The computer program can be recorded on, for example, a computer-readable recording medium, and be loaded into the above computing device through the computer-readable recording medium and run therein.

[0131] Based on the descriptions of the above method embodiments and device embodiments, the embodiments of the present application also provide an electronic device, and this electronic device can be a server. Please refer to Figure 7 , this electronic device at least includes a processor 701, an input device 702, an output device 703, and a computer storage medium 704. Among them, the processor 701, the input device 702, the output device 703, and the computer storage medium 704 in the electronic device can be connected through a bus or other means.

[0132] The computer storage medium 704 can be stored in the memory of the electronic device. The computer storage medium 704 is used to store a computer program, and the computer program includes program instructions. The processor 701 is used to execute the program instructions stored in the computer storage medium 704. The processor 701 (or CPU (Central Processing Unit, central processor)) is the computing core and control core of the electronic device, and is adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions so as to implement the corresponding method process or the corresponding function.

[0133] In one embodiment, the processor 701 of the electronic device provided by the embodiments of the present application can be used to perform a series of planar construction processes:

[0134] Obtain a first two-dimensional grid and a second two-dimensional grid of the currently input image frame;

[0135] Obtain at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier;

[0136] In the case where there is currently no structured plane, generate a structured plane according to the pose information and the at least one target three-dimensional grid.

[0137] In another embodiment, the processor 701 executes the obtaining of the first two-dimensional grid and the second two-dimensional grid of the currently input image frame, including:

[0138] Obtain a first two-dimensional feature point and a second two-dimensional feature point of the image frame, where the first two-dimensional feature point is a front-end feature point obtained by the front end of the simultaneous localization and mapping system extracting features from the image frame, and the second two-dimensional feature point is a back-end feature point obtained by the back end of the simultaneous localization and mapping system extracting features from the image frame;

[0139] Perform triangulation on the first two-dimensional feature points to obtain the first two-dimensional grid, and perform triangulation on the second two-dimensional feature points to obtain the second two-dimensional grid.

[0140] In another embodiment, the processor 701 executes the obtaining of at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier, including:

[0141] Obtain at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information;

[0142] Use the pose information to screen the at least one three-dimensional grid to obtain the at least one target three-dimensional grid.

[0143] In another embodiment, the processor 701 executes the obtaining of at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information, including:

[0144] Obtain a first three-dimensional point corresponding to the first two-dimensional feature point in the first two-dimensional grid;

[0145] Obtain a second three-dimensional point corresponding to the second two-dimensional feature point in the second two-dimensional grid;

[0146] Use the pose information to screen the first three-dimensional point and the second three-dimensional point, and project the remaining first three-dimensional points and the remaining second three-dimensional points after screening respectively to obtain the at least one three-dimensional grid.

[0147] In another embodiment, the processor 701 executes screening the at least one three-dimensional grid by using the pose information to obtain at least one target three-dimensional grid, including:

[0148] Determine a first distance between the three-dimensional grid in the at least one three-dimensional grid and the carrier according to the pose information;

[0149] Delete the three-dimensional grids in the at least one three-dimensional grid whose first distance is greater than or equal to a distance threshold, and screen out the three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane from the remaining three-dimensional grids in the at least one three-dimensional grid;

[0150] Determine the three-dimensional grids of the horizontal plane and the three-dimensional grids of the vertical plane as the at least one target three-dimensional grid.

[0151] In another embodiment, the processor 701 executes generating a structured plane according to the pose information and the at least one target three-dimensional grid, including:

[0152] Obtain a second distance between any two target three-dimensional grids in the at least one target three-dimensional grid based on the pose information;

[0153] Cluster the at least one target three-dimensional grid based on the second distance to generate a structured plane.

[0154] In another embodiment, the processor 701 is further configured to execute: when there is an existing structured plane, associate the target three-dimensional grids in the at least one target three-dimensional grid with the existing structured plane.

[0155] In another embodiment, the processor 701 executes associating the target three-dimensional grids in the at least one target three-dimensional grid with the existing structured plane, including:

[0156] Determine a difference in direction between the target three-dimensional grids in the at least one target three-dimensional grid and the existing structured plane;

[0157] When the difference is less than or equal to a first preset value and at least one three-dimensional point constituting the target three-dimensional grid is on the existing structured plane, associate the target three-dimensional grid with the existing structured plane.

[0158] In another embodiment, after obtaining at least one target three-dimensional grid, the processor 701 is further configured to execute: updating the three-dimensional grid obtained before the image frame by using the at least one target three-dimensional grid, where the number of the at least one target three-dimensional grid is less than or equal to a second preset value.

[0159] Exemplarily, the electronic device may include but is not limited to a processor 701, an input device 702, an output device 703, and a computer storage medium 704. The input device 702 may be a keyboard, a touch screen, etc., and the output device 703 may be a speaker, a display, a radio frequency transmitter, etc. Those skilled in the art can understand that the schematic diagram is only an example of the electronic device, and does not constitute a limitation on the electronic device. It may include more or fewer components than shown in the figure, or combine some components, or different components.

[0160] It should be noted that since the processor 701 of the electronic device implements the steps in the above Figure 2 or Figure 5 plane construction method shown, therefore Figure 2 or Figure 5 the embodiments of the plane construction method shown are all applicable to this electronic device, and can achieve the same or similar beneficial effects.

[0161] The embodiment of the present application also provides a computer storage medium (Memory). The computer storage medium is a memory device in the electronic device, used to store programs and data. It can be understood that the computer storage medium here can include both the built-in storage medium in the terminal, and of course can also include the extended storage medium supported by the terminal. The computer storage medium provides a storage space, and this storage space stores the operating system of the terminal. And, one or more instructions suitable for being loaded and executed by the processor 701 are stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it can also be at least one computer storage medium located far from the aforementioned processor 701. In one embodiment, one or more instructions stored in the computer storage medium can be loaded and executed by the processor 701 to implement the corresponding steps of the above-mentioned plane construction method.

[0162] Exemplarily, the computer program of the computer storage medium includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0163] It should be noted that since the computer program in the computer storage medium implements the steps in the above-mentioned plane construction method when executed by the processor, all embodiments of the above-mentioned plane construction method are applicable to this computer storage medium and can achieve the same or similar beneficial effects.

[0164] An embodiment of the present application also provides a computer program product. Among them, the above computer program product includes a computer program, and the above computer program can operate to cause a computer to execute the steps in the above-mentioned plane construction method. This computer program product can be a software installation package.

[0165] The above has introduced the embodiments of the present application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for constructing a plane, characterized in that, the method includes: Obtaining a first two-dimensional grid and a second two-dimensional grid of the currently input image frame; the first two-dimensional grid refers to a two-dimensional grid constructed from two-dimensional feature points obtained by the front end of the simultaneous localization and mapping system for feature extraction of the image frame; the second two-dimensional grid refers to a two-dimensional grid constructed from two-dimensional feature points obtained by the back end of the simultaneous localization and mapping system for feature extraction of the image frame; Obtaining at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier; the carrier refers to an image acquisition device; In the case where there is no structured plane currently, generating a structured plane according to the pose information and the at least one target three-dimensional grid; The obtaining at least one target three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information of the carrier includes: Obtaining at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information; Using the pose information to screen the at least one three-dimensional grid to obtain the at least one target three-dimensional grid; The obtaining at least one three-dimensional grid according to the first two-dimensional grid, the second two-dimensional grid, and the pose information includes: Obtaining a first three-dimensional point corresponding to the first two-dimensional feature point in the first two-dimensional grid; Obtaining a second three-dimensional point corresponding to the second two-dimensional feature point in the second two-dimensional grid; Using the pose information to screen the first three-dimensional point and the second three-dimensional point, and respectively projecting the remaining first three-dimensional points and the remaining second three-dimensional points after screening to obtain the at least one three-dimensional grid; The generating a structured plane according to the pose information and the at least one target three-dimensional grid includes: Obtaining a second distance between any two target three-dimensional grids in the at least one target three-dimensional grid based on the pose information; Clustering the at least one target three-dimensional grid based on the second distance to generate a structured plane.

2. The method according to claim 1, characterized in that, the obtaining a first two-dimensional grid and a second two-dimensional grid of the currently input image frame includes: Obtaining a first two-dimensional feature point and a second two-dimensional feature point of the image frame, where the first two-dimensional feature point is a front-end feature point obtained by the front end of the simultaneous localization and mapping system for feature extraction of the image frame, and the second two-dimensional feature point is a back-end feature point obtained by the back end of the simultaneous localization and mapping system for feature extraction of the image frame; Triangulating the first two-dimensional feature points to obtain the first two-dimensional grid, and triangulating the second two-dimensional feature points to obtain the second two-dimensional grid.

3. The method according to claim 1, characterized in that, the using the pose information to screen the at least one three-dimensional grid to obtain the at least one target three-dimensional grid includes: Determining a first distance between the three-dimensional grid in the at least one three-dimensional grid and the carrier according to the pose information; Delete the 3D meshes in the at least one 3D mesh where the first distance is greater than or equal to the distance threshold, and screen out the 3D meshes of the horizontal plane and the 3D meshes of the vertical plane from the remaining 3D meshes in the at least one 3D mesh; Determine the 3D meshes of the horizontal plane and the 3D meshes of the vertical plane as the at least one target 3D mesh.

4. The method according to any one of claims 1-3, characterized in that, the method further comprises: When there is a structured plane currently, associate the target 3D meshes in the at least one target 3D mesh with the existing structured plane.

5. The method according to claim 4, characterized in that, the associating the target 3D meshes in the at least one target 3D mesh with the existing structured plane includes: Determine the difference in direction between the target 3D meshes in the at least one target 3D mesh and the existing structured plane; When the difference is less than or equal to the first preset value and at least one 3D point constituting the target 3D mesh is on the existing structured plane, associate the target 3D mesh with the existing structured plane.

6. The method according to any one of claims 1-5, characterized in that, After obtaining at least one target 3D mesh, the method further comprises: Update the 3D meshes obtained before the image frame by using the at least one target 3D mesh, wherein the number of the at least one target 3D mesh is less than or equal to the second preset value.

7. A plane construction device, characterized in that, the device comprises: An acquisition unit, configured to acquire a first two-dimensional mesh and a second two-dimensional mesh of a currently input image frame; the first two-dimensional mesh refers to a two-dimensional mesh constructed from two-dimensional feature points obtained by feature extraction of the image frame by the front end of the simultaneous localization and mapping system; the second two-dimensional mesh refers to a two-dimensional mesh constructed from two-dimensional feature points obtained by feature extraction of the image frame by the back end of the simultaneous localization and mapping system; A processing unit, configured to obtain at least one target 3D mesh according to the first two-dimensional mesh, the second two-dimensional mesh, and the pose information of the carrier; the carrier refers to an image acquisition device; The processing unit is further configured to generate a structured plane according to the pose information and the at least one target 3D mesh when there is no structured plane currently; In terms of obtaining at least one target 3D mesh according to the first two-dimensional mesh, the second two-dimensional mesh, and the pose information of the carrier, the processing unit specifically is configured to: Obtain at least one 3D mesh according to the first two-dimensional mesh, the second two-dimensional mesh, and the pose information; Use the pose information to screen the at least one 3D mesh to obtain the at least one target 3D mesh; In terms of obtaining at least one 3D mesh according to the first two-dimensional mesh, the second two-dimensional mesh, and the pose information, the processing unit specifically is configured to: Obtain the first 3D points corresponding to the first two-dimensional feature points in the first two-dimensional mesh; Obtain the second three-dimensional points corresponding to the second two-dimensional feature points in the second two-dimensional grid; Use the pose information to screen the first three-dimensional points and the second three-dimensional points, and project the remaining first three-dimensional points and the remaining second three-dimensional points after screening respectively to obtain the at least one three-dimensional grid; In terms of generating a structured plane according to the pose information and the at least one target three-dimensional grid, the processing unit is specifically configured to: Obtain a second distance between any two target three-dimensional grids in the at least one target three-dimensional grid based on the pose information; Cluster the at least one target three-dimensional grid based on the second distance to generate a structured plane.

8. An electronic device, including an input device and an output device, characterized in that, further comprising: a processor, adapted to implement one or more instructions; and, a computer storage medium, the computer storage medium stores one or more instructions, and the one or more instructions are adapted to be loaded and executed by the processor to perform the method according to any one of claims 1-6.

9. A computer storage medium, characterized in that, the computer storage medium stores one or more instructions, and the one or more instructions are adapted to be loaded and executed by a processor to perform the method according to any one of claims 1-6.

Images