A point-plane optimization method, a terminal device, and a computer storage medium

Abstract

The present application discloses a point-plane optimization method, a terminal device, and a computer storage medium. The point-plane optimization method includes: respectively obtaining the first coordinates of a plurality of observation points associated with a plane to be optimized in a first key frame, and the second coordinates of the observation points in a second key frame; based on the first coordinates, the second coordinates, and a preset projection equation, obtaining the relative position relationship between the observation points and the plane to be optimized; and optimizing the plane to be optimized based on the relative position relationship between the plurality of observation points and the plane to be optimized. The point-plane optimization method of the present application optimizes the data calculation time and data calculation amount in the plane fitting process through the relative position relationship between a plurality of observation points and the plane to be optimized.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision processing, and particularly to a point-plane optimization method, a terminal device, and a computer storage medium. Background Art

[0002] There are a large number of planar structures in the indoor environment, and the state optimization of integrating points and planes is a key issue in the planar SLAM (simultaneous localization and mapping) technology. Planar SLAM can play an important role in many fields, such as the visualization of AR (Augmented Reality) / VR (Virtual Reality) models, and the positioning systems of drones and autonomous driving.

[0003] However, the current optimization model based on homography point-plane has the problem of slow optimization speed, which seriously affects the real-time use of this algorithm on mobile devices, especially devices such as mobile phones and drones. Summary of the Invention

[0004] The present application provides a point-plane optimization method, a terminal device, and a computer storage medium.

[0005] One technical solution adopted by the present application is to provide a point-plane optimization method, and the point-plane optimization method includes:

[0006] Obtain the first coordinates of multiple observation points associated with the plane to be optimized in the first key frame and the second coordinates of the observation points in the second key frame respectively;

[0007] Based on the first coordinates, the second coordinates, and a preset projection equation, obtain the relative position relationship between the observation points and the plane to be optimized;

[0008] Optimize the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized.

[0009] Wherein, the obtaining the relative position relationship between the observation points and the plane to be optimized based on the first coordinates, the second coordinates, and a preset projection equation includes:

[0010] Based on the first coordinates and the preset projection equation, obtain the first reprojection relationship between the first key frame and the plane to be optimized;

[0011] Based on the second coordinates and the preset projection equation, obtain the second reprojection relationship between the second key frame and the plane to be optimized;

[0012] Using the first reprojection relationship and the second reprojection relationship, obtain the relative position relationship between the observation point and the plane to be optimized.

[0013] Among them, obtaining the first reprojection relationship between the first key frame and the plane to be optimized based on the first coordinate and the preset projection equation includes:

[0014] Using the camera parameters for collecting the first key frame, the plane parameters of the plane to be optimized, and the first coordinate, and combining with the preset projection equation, obtain the first reprojection relationship.

[0015] Among them, obtaining the second reprojection relationship between the second key frame and the plane to be optimized based on the second coordinate and the preset projection equation includes:

[0016] Using the camera parameters for collecting the second key frame, the camera parameters for collecting the second key frame, the first coordinate, and the second coordinate, and combining with the preset projection equation, obtain the second reprojection relationship.

[0017] Among them, using the first reprojection relationship and the second reprojection relationship to obtain the relative position relationship between the observation point and the plane to be optimized includes:

[0018] Combining the first reprojection relationship and the second reprojection relationship to obtain a third reprojection relationship for characterizing the relative position relationship between the observation point and the plane to be optimized;

[0019] Using the third reprojection relationship to obtain the homography factor of the observation point;

[0020] Before optimizing the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized, the point-plane optimization method further includes:

[0021] Using the homography factors of the multiple observation points to fuse and obtain the relative position relationship between the multiple observation points and the plane to be optimized.

[0022] Among them, using the homography factors of the multiple observation points to fuse and obtain the relative position relationship between the multiple observation points and the plane to be optimized, and optimizing the plane to be optimized includes:

[0023] Using the homography factor of the observation point to obtain the homography cost function of the observation point;

[0024] Fusing the homography cost functions of the multiple observation points to obtain a total homography cost function for characterizing the relative position relationship between the multiple observation points and the plane to be optimized;

[0025] Optimize the plane to be optimized by using the total homography cost function.

[0026] Among them, the optimizing the plane to be optimized by using the total homography cost function includes:

[0027] Construct the Jacobian matrix of the plane to be optimized by using the total homography cost function;

[0028] Solve the Jacobian matrix, and use the solution result as the optimization parameter of the plane to be optimized.

[0029] Among them, after optimizing the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized, the point-plane optimization method further includes:

[0030] Construct a map by using the optimized plane.

[0031] Another technical solution adopted by this application is to provide a terminal device, where the terminal device includes a memory and a processor coupled to the memory;

[0032] Among them, the memory is used to store program data, and the processor is used to execute the program data to implement the point-plane optimization method as described above.

[0033] Another technical solution adopted by this application is to provide a computer storage medium, where the computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the point-plane optimization method as described above.

[0034] The beneficial effects of this application are as follows: The terminal device respectively obtains the first coordinates of multiple observation points associated with the plane to be optimized in the first key frame, and the second coordinates of the observation points in the second key frame; based on the first coordinates, the second coordinates, and a preset projection equation, obtain the relative position relationship between the observation points and the plane to be optimized; optimize the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized. The point-plane optimization method of this application optimizes the data calculation time and data calculation amount in the plane fitting process through the relative position relationship between multiple observation points and the plane to be optimized. Description of the Drawings

[0035] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0036] Figure 1It is a schematic flowchart of an embodiment of the point-plane optimization method provided by this application;

[0037] Figure 2 is Figure 1 The specific flowchart of step 12 of the point-plane optimization method shown;

[0038] Figure 3 It is a schematic structural diagram of the homography factor provided by this application;

[0039] Figure 4 It is a schematic structural diagram of the compressed homography factor provided by this application;

[0040] Figure 5 It is a schematic structural diagram of an embodiment of the terminal device provided by this application;

[0041] Figure 6 It is a schematic structural diagram of another embodiment of the terminal device provided by this application;

[0042] Figure 7 It is a schematic structural diagram of an embodiment of the computer storage medium provided by this application. Detailed implementation manners

[0043] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0044] To solve the problem that the current optimization model of points and planes has a slow optimization speed, this application proposes a fast point-plane optimization model based on homography, which synthesizes a large amount of point-plane information into one optimization information and removes the reprojection information of redundant plane points to improve the optimization speed and achieve the user's usage efficiency.

[0045] Specifically, please refer to Figure 1 , Figure 1 It is a schematic flowchart of an embodiment of the point-plane optimization method provided by this application. The point-plane optimization method in the embodiments of this application can be applied to a terminal device. Among them, the terminal device of this application can be a server, an electronic device, or a system in which the server and the electronic device cooperate with each other. Correspondingly, each part included in the terminal device, such as each unit, sub-unit, module, and sub-module, can be all set in the server, all set in the electronic device, or respectively set in the server and the electronic device.

[0046] Further, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide a distributed server, or as a single software or software module, which is not specifically limited herein.

[0047] As Figure 1 shown, the point-plane optimization method of the embodiments of the present application may specifically include the following steps:

[0048] Step S11: Obtain the first coordinates of multiple observation points associated with the plane to be optimized in the first key frame and the second coordinates of the observation points in the second key frame respectively.

[0049] Step S12: Based on the first coordinates, the second coordinates, and a preset projection equation, obtain the relative position relationship between the observation points and the plane to be optimized.

[0050] In the embodiments of the present application, the terminal device needs to obtain the plane points in SLAM, that is, the two-dimensional coordinates of the observation points, the plane data, and the pose of the camera that captures the image. Further, the terminal device also needs to obtain the first coordinates of the observation points in the first key frame, that is, the coordinates of the normalized points of the observation points in the camera coordinate system of the first key frame; obtain the second coordinates of the observation points in the second key frame, that is, the coordinates of the normalized points of the observation points in the camera coordinate system of the second key frame.

[0051] In the embodiments of the present application, using the reprojection relationship of the same observation point on different images, the terminal device can construct a projection equation of the observation point based on the first coordinates and the second coordinates, that is, the relative position relationship between the observation point and the plane to be optimized. Among them, the projection equation of an observation point can be represented by the following formula:

[0052]

[0053] Among them, W represents the world coordinate system, C represents the camera coordinate system, i and j are two different cameras, p i =(x i ,y i 1) is the point of the normalized point in the camera coordinate system of the first camera, p j =(x j ,y j 1) is the coordinate of the normalized point in the camera coordinate system of the second camera, is the rotation corresponding to the first camera, is the rotation corresponding to the second camera, is the position corresponding to the first camera, is the position corresponding to the second camera; the plane to be optimized π=(nπ , d π are the direction and distance of the plane, respectively. In addition, H is the homography matrix, I is the identity matrix, and s is a scale parameter.

[0054] For the specific construction process of the projection equation, please continue to refer to Figure 2 , Figure 2 is Figure 1 the specific process schematic diagram of step 12 of the point-plane optimization method shown.

[0055] As Figure 2 shown, the point-plane optimization method of the embodiment of the present application may specifically include the following steps:

[0056] Step S121: Based on the first coordinate and the preset projection equation, obtain the first reprojection relationship between the first key frame and the plane to be optimized.

[0057] In the embodiment of the present application, the terminal device uses the camera parameters for collecting the first key frame, the plane parameters of the plane to be optimized, and the first coordinate, combined with the preset projection equation, to obtain the first reprojection relationship.

[0058] Step S122: Based on the second coordinate and the preset projection equation, obtain the second reprojection relationship between the second key frame and the plane to be optimized.

[0059] In the embodiment of the present application, using the camera parameters for collecting the first key frame, the camera parameters for collecting the second key frame, the first coordinate, and the second coordinate, combined with the preset projection equation, to obtain the second reprojection relationship.

[0060] In the embodiment of the present application, when the 3D map point is associated with the plane to be optimized, the terminal device can force the 3D map point to fall on the plane to be optimized, so that the 3D map point is transformed into a normalized point. Therefore, the terminal device can use the homography matrix to constrain two key frames and a plane to be optimized instead of using the distance constraint from the common point to the plane.

[0061] Specifically, assume that the observed point on the plane to be optimized is π w is observed by the first camera coordinate system i of the first camera th and the second camera coordinate system j of the second camera th From this, the following reprojection relationship from the point to the plane can be written:

[0062]

[0063]

[0064] Among them, in the embodiment of the present application, it is defined that p i =(xi , y i , 1) T and p j = (x j , y j , 1) T . Among them, (x i , y i , 1) T = K -1 (u i , v i , 1) T is the mapping from the coordinate point in the image coordinate system collected by the first camera to the coordinate point in the camera coordinate system i th . K is the intrinsic matrix, (u i , v i ) is the two-dimensional image feature in the key frame collected by the first camera, λ is the depth information of the image, and s is an unknown scale parameter.

[0065] Step S123: Use the first reprojection relationship and the second reprojection relationship to obtain the relative position relationship between the observation point and the plane to be optimized.

[0066] In the embodiment of the present application, combining the above first reprojection relationship (2) and the second reprojection relationship (3), the terminal device can obtain a projection equation of a single observation as shown in the above formula (1), that is, the relative position relationship between the observation point and the plane to be optimized:

[0067]

[0068] Among them, the homography matrix H includes the camera parameters of the first key frame and the camera parameters of the second key frame and and the plane parameters of the plane to be optimized

[0069] Specifically, when the observation point is on the plane to be optimized, the homography constraint and the reprojection constraint are equal. Among them, the homography constraint does not require the three-dimensional position of the point feature. In the BA (bundle adjustment) problem, the present application transforms the reprojection constraint into a homography constraint, which is equivalent to removing many state variables of the observation points on the plane to be optimized. Finally, using a smaller and sparser Hessian Matrix, the efficiency of the bundle adjustment is greatly improved.

[0070] In the embodiment of the present application, from the projection equation in the above formula (1), the homography constraint equation of the observation point can be obtained as follows:

[0071]

[0072] Furthermore, the present application can eliminate the unknown scale parameter s from the above formula (4), thereby obtaining the homography cost function as follows:

[0073]

[0074]

[0075] where C l is the coordinate matrix of the observation points.

[0076] Step S13: Optimize the plane to be optimized based on the relative position relationship between multiple observation points and the plane to be optimized.

[0077] In the embodiment of the present application, the terminal device uses the third projection relationship of the above formula (1) to obtain the homography factor of a single observation point in formulas (4) and (5); then, the terminal device fuses the homography factors of multiple observation points associated with the plane to be optimized to obtain the relative position relationship between multiple observation points and the plane to be optimized, that is, the total homography factor. For the derivation of the total homography factor, that is, the total homography cost function, reference can be continued to formula (6).

[0078] In the embodiment of the present application, homography relates the states of two key frames and a plane to be optimized. There may be many common observation results among these three states. Therefore, these observation results can be combined into one observation result to further improve the optimization speed.

[0079] Assume that there are N point features on the plane to be optimized. Then, the total homography cost function of the N point features on the plane to be optimized can be expressed as:

[0080]

[0081] where is matrix factorization. To ensure the stability of the solution, the present application uses eigenvalue decomposition, and G h is a constant 9×9 matrix during the optimization process and only depends on constant values. Therefore, this matrix can be calculated in advance.

[0082] In the embodiment of the present application, the present application can merge the observations of the N point features on the plane to be optimized into an observation matrix G h as shown in Figure 3 and Figure 4 . Figure 3 is the homography factor before merging, Figure 4 is the compressed homography factor after merging. By combining multiple cost functions into one cost function, the present application can effectively improve the efficiency of bundle adjustment.

[0083] Further, the compressed homography cost function, i.e., the Jacobian matrix of the total homography cost function, can be defined as:

[0084]

[0085] where L h is a lower triangular matrix, is an upper triangular matrix, and L h and are obtained by matrix decomposition of the above-mentioned constant 9×9 matrix G h through matrix decomposition.

[0086] where the residual function, i.e., the homography cost function, is:

[0087]

[0088] By solving the Jacobian matrix of the total homography cost function, the optimization parameters of the plane to be optimized can be obtained.

[0089] In addition, for some plane points with known point-plane constraints, such as internal observation points, the present application can also adopt the reprojection observation of deleting these plane points, so as to reduce the dimension of the entire optimization state quantity and the optimization matrix, and further reduce the calculation amount.

[0090] In the embodiment of the present application, the terminal device respectively obtains the first coordinates of a plurality of observation points associated with the plane to be optimized in the first key frame, and the second coordinates of the observation points in the second key frame; based on the first coordinates, the second coordinates and a preset projection equation, the relative position relationship between the observation points and the plane to be optimized is obtained; based on the relative position relationship between the plurality of observation points and the plane to be optimized, the plane to be optimized is optimized. The point-plane optimization method of the present application optimizes the data calculation time and data calculation amount in the plane fitting process through the relative position relationship between the plurality of observation points and the plane to be optimized; through the point-plane optimization method of the present application, the accuracy and performance of SLAM can be effectively optimized, and the tracking effect of SLAM can be improved.

[0091] The above embodiments are only one common case of the present application, and do not limit the technical scope of the present application in any way. Therefore, any minor modifications, equivalent changes or decorations made to the above content based on the essence of the present application solution still belong to the scope of the technical solution of the present application.

[0092] Please continue to refer to Figure 5 , Figure 5 which is a schematic structural diagram of an embodiment of the terminal device provided by the present application. Among them, the terminal device 30 includes an acquisition module 31, a projection module 32 and an optimization module 33.

[0093] Among them, the acquisition module 31 is configured to respectively acquire the first coordinates of multiple observation points associated with the plane to be optimized in the first key frame, and the second coordinates of the observation points in the second key frame.

[0094] The projection module 32 is configured to obtain the relative position relationship between the observation point and the plane to be optimized based on the first coordinate, the second coordinate, and a preset projection equation.

[0095] The optimization module 33 is configured to optimize the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized.

[0096] The projection module 32 is further configured to obtain the first reprojection relationship between the first key frame and the plane to be optimized based on the first coordinate and a preset projection equation; obtain the second reprojection relationship between the second key frame and the plane to be optimized based on the second coordinate and a preset projection equation; and use the first reprojection relationship and the second reprojection relationship to obtain the relative position relationship between the observation point and the plane to be optimized.

[0097] The projection module 32 is further configured to use the camera parameters for collecting the first key frame, the plane parameters of the plane to be optimized, and the first coordinate in combination with a preset projection equation to obtain the first reprojection relationship.

[0098] The projection module 32 is further configured to use the camera parameters for collecting the first key frame, the camera parameters for collecting the second key frame, the first coordinate, and the second coordinate in combination with a preset projection equation to obtain the second reprojection relationship.

[0099] The projection module 32 is further configured to combine the first reprojection relationship and the second reprojection relationship to obtain a third reprojection relationship for characterizing the relative position relationship between the observation point and the plane to be optimized; and use the third reprojection relationship to obtain the homography factor of the observation point.

[0100] The projection module 32 is further configured to fuse the homography factors of the multiple observation points to obtain the relative position relationship between the multiple observation points and the plane to be optimized.

[0101] The optimization module 33 is further configured to use the homography factor of the observation point to obtain the homography cost function of the observation point; fuse the homography cost functions of the multiple observation points to obtain a total homography cost function for characterizing the relative position relationship between the multiple observation points and the plane to be optimized; and use the total homography cost function to optimize the plane to be optimized.

[0102] The optimization module 33 is further configured to construct a Jacobian matrix of the plane to be optimized by using the total homography cost function, and solve the Jacobian matrix, and use the solution result as the optimization parameter of the plane to be optimized.

[0103] The optimization module 33 is further configured to construct a map by using the optimized plane.

[0104] Please continue to refer to Figure 6 , Figure 6 which is a schematic structural diagram of another embodiment of the terminal device provided by the present application. The terminal device 500 in the embodiment of the present application includes a processor 51, a memory 52, an input / output device 53, and a bus 54.

[0105] The processor 51, the memory 52, and the input / output device 53 are respectively connected to the bus 54. Program data is stored in the memory 52, and the processor 51 is configured to execute the program data to implement the point-plane optimization method described in any of the above embodiments.

[0106] In the embodiment of the present application, the processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP, Digital Signal Process), an application specific integrated circuit (ASIC, Application Specific Integrated Circuit), a field programmable gate array (FPGA, FieldProgrammable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor, or the processor 51 may also be any conventional processor, etc.

[0107] The present application also provides a computer storage medium. Please continue to refer to Figure 7 , Figure 7 which is a schematic structural diagram of an embodiment of the computer storage medium provided by the present application. Program data 61 is stored in the computer storage medium 600, and when the program data 61 is executed by a processor, it is used to implement the point-plane optimization method described in any of the above embodiments.

[0108] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0109] The above are only the embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A point-plane optimization method, characterized in that, The point-plane optimization method includes: Obtaining the first coordinates of multiple observation points associated with the plane to be optimized in the first key frame and the second coordinates of the observation points in the second key frame respectively; Based on the first coordinates, the second coordinates and a preset projection equation, obtaining the relative position relationship between the observation points and the plane to be optimized; Optimizing the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized.

2. The point-plane optimization method according to claim 1, wherein The obtaining the relative position relationship between the observation points and the plane to be optimized based on the first coordinates, the second coordinates and a preset projection equation includes: Based on the first coordinates and the preset projection equation, obtaining the first reprojection relationship between the first key frame and the plane to be optimized; Based on the second coordinates and the preset projection equation, obtaining the second reprojection relationship between the second key frame and the plane to be optimized; Using the first reprojection relationship and the second reprojection relationship to obtain the relative position relationship between the observation points and the plane to be optimized.

3. The point-plane optimization method according to claim 2, wherein The obtaining the first reprojection relationship between the first key frame and the plane to be optimized based on the first coordinates and a preset projection equation includes: Using the camera parameters for collecting the first key frame, the plane parameters of the plane to be optimized, the first coordinates and the preset projection equation to obtain the first reprojection relationship.

4. The point-plane optimization method according to claim 3, wherein The obtaining the second reprojection relationship between the second key frame and the plane to be optimized based on the second coordinates and the preset projection equation includes: Using the camera parameters for collecting the first key frame, the camera parameters for collecting the second key frame, the first coordinates and the second coordinates and the preset projection equation to obtain the second reprojection relationship.

5. The point-plane optimization method according to any one of claims 2 to 4, wherein The using the first reprojection relationship and the second reprojection relationship to obtain the relative position relationship between the observation points and the plane to be optimized includes: Combining the first reprojection relationship and the second reprojection relationship to obtain a third reprojection relationship for characterizing the relative position relationship between the observation points and the plane to be optimized; Using the third reprojection relationship to obtain the homography factor of the observation points; Before optimizing the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized, the point-plane optimization method further includes: Using the homography factors of the multiple observation points to fuse and obtain the relative position relationship between the multiple observation points and the plane to be optimized.

6. The point-plane optimization method according to claim 5, wherein The using the homography factors of the multiple observation points to fuse and obtain the relative position relationship between the multiple observation points and the plane to be optimized and optimizing the plane to be optimized includes: Obtain the homography cost function of the observation point by using the homography factor of the observation point; Fuse the homography cost functions of the multiple observation points to obtain a total homography cost function for characterizing the relative position relationship between the multiple observation points and the plane to be optimized; Optimize the plane to be optimized by using the total homography cost function.

7. The point-plane optimization method according to claim 6, wherein The optimizing the plane to be optimized by using the total homography cost function includes: Construct a Jacobian matrix of the plane to be optimized by using the total homography cost function; Solve the Jacobian matrix, and use the solution result as the optimization parameter of the plane to be optimized.

8. The point-plane optimization method according to any one of claims 1 to 7, wherein After optimizing the plane to be optimized based on the relative position relationship between the multiple observation points and the plane to be optimized, the point-plane optimization method further includes: Construct a map by using the optimized plane.

9. A terminal device, characterized in that, The terminal device includes a memory and a processor coupled to the memory; Wherein, the memory is used for storing program data, and the processor is used for executing the program data to implement the point-plane optimization method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium is used for storing program data, and when the program data is executed by a computer, it is used to implement the point-plane optimization method according to any one of claims 1 to 8.

Images