Arrow optimization modeling method and device

Abstract

The application relates to an arrow optimization modeling method and device, wherein the method reconstructs an arrow model according to constraint relationships among objects, carries out large-scale nonlinear optimization, adjusts a pose to eliminate accumulated errors in a system, and improves the modeling accuracy of the arrow under a SLAM system; the objects include feature lines and / or arrows; the constraint relationships include plane constraints between the feature lines and the arrows, parallel and vertical constraints between the feature lines, or coplanar constraints between the arrows; and the device is used for realizing the above method. Compared with the prior art, the application has the advantages of higher mapping accuracy and the like.

Description

Technical Field

[0001] This invention relates to the field of visual processing, and in particular to a method and apparatus for optimizing arrow modeling. Background Technology

[0002] Visual SLAM often neglects edge information when reconstructing planar objects. The generation of 3D edge feature lines is still achieved through triangulation of the left and right eyes during initialization or by matching two consecutive frames from the left eye during tracking. Compared to the overall map data, this approach uses insufficient information. Currently, conventional SLAM algorithms require pre-loading the CAD model of arrow objects or using point cloud segmentation from deep learning for reconstruction, a complex and cumbersome process. While arrows possess complete and clear edge features, and their 3D models can be successfully generated by extracting edge line features, the cumulative error of the SLAM system means that simply triangulating straight lines to construct an arrow model is inaccurate. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an arrow optimization modeling method and apparatus. By establishing constraint relationships between feature lines and arrows, between feature lines and arrow objects, large-scale nonlinear optimization is further performed to adjust the pose to eliminate accumulated errors in the system and improve the modeling accuracy of arrows in the SLAM system.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] According to a first aspect of the present invention, an arrow optimization modeling method is provided, which reconstructs an arrow model based on constraint relationships between objects; the objects include feature lines and / or arrows; the constraint relationships include planar constraints between feature lines and arrows, parallel and perpendicular constraints between feature lines, or coplanar constraints between arrows; wherein...

[0006] Reconstructing the arrow model based on the planar constraints between the feature lines and the arrowhead includes: extracting candidate arrow feature lines and assigning semantic information to them; 3Dizing the feature lines based on the semantic information to obtain 3D feature lines, and obtaining the Plück coordinates of the 3D feature lines in the arrow coordinate system; performing line-plane constraint optimization, transforming the Plück coordinates of the 3D feature lines from the arrow coordinate system to the world coordinate system, and then transforming the Plück coordinates in the world coordinate system to the Plück coordinates in the camera coordinate system, so that the 3D feature lines and the arrowhead are located in the same plane, and connecting the 3D feature lines to form a closed shape, which is the reconstructed arrowhead; the semantic information includes a first attribute and a second attribute; the first attribute is the arrow ID, and the second attribute is the feature line number;

[0007] Reconstructing the arrow model based on the parallel and perpendicular constraints between the feature lines includes: numbering all arrows in the region scene, extracting feature lines and using the arrow number as the third attribute; performing principal component analysis on feature lines with the same third attribute based on the PCA algorithm to obtain the corresponding primary and secondary directions of the arrow, and establishing the relationship between the direction vectors of all feature lines of the arrow and the primary and secondary directions; updating the Plück coordinates of the feature lines according to the relationship; and reconstructing the arrow model by number.

[0008] Reconstructing the arrow model based on the coplanar constraints between the arrows includes: constructing a pose rotation matrix, wherein the pose rotation matrix is ​​a 3×3 matrix and the elements in the third row are [0, 0, 1] in sequence; obtaining the 3D model of the first arrow in the region map as the initial model, and identifying the second arrow based on the initial model; recovering the pose of the second arrow using the pose rotation matrix to realize arrow modeling; the first arrow is all the arrows of different specifications that appear for the first time during the identification process, and the second arrow is the arrow with the same specifications as the first arrow that appears repeatedly during the identification process, and the first arrow and the second arrow are coplanar.

[0009] As a preferred technical solution, the method for obtaining the first attribute is as follows: when multiple candidate feature lines are coplanar with the same arrow and can form a closed shape, the ID of the arrow is used as the first attribute of the multiple candidate feature lines.

[0010] The method for obtaining the second attribute is as follows: the feature lines are numbered to obtain the second attribute of the feature lines, wherein the feature lines with the same first attribute are numbered differently.

[0011] As a preferred technical solution, obtaining the Plück coordinates of the feature line specifically includes:

[0012] Candidate feature lines are filtered based on the first attribute, and candidate feature lines with the same arrow ID are selected as feature lines.

[0013] Add arrows and binary edges to the feature lines whose first attribute is the corresponding arrow ID to obtain 3D feature lines;

[0014] The 3D feature lines are subjected to a relative pose transformation from the world coordinate system to the visual coordinate system to obtain 3D Plück coordinates in the visual coordinate system.

[0015] Based on the second attribute, obtain the Plück coordinates of multiple feature lines with the same first attribute.

[0016] As a preferred technical solution, the transformation of Plück coordinates specifically involves transforming the Plück coordinates from the arrow coordinate system to the camera coordinate system, the transformation being performed according to the formula... Proceed; where L M The coordinates are in the camera coordinate system. m in the Plück coordinate systemM Transpose of a vector on the axis In the Plück coordinate system, d M Transpose of a vector on the axis.

[0017] As a preferred technical solution, the arrow is reconstructed based on the parallel and perpendicular constraints between feature lines, specifically including:

[0018] Add binary edges to feature lines with the same arrow ID to obtain 3D feature lines, represent the 3D feature lines using Plück coordinates, and sample the 3D feature lines to obtain a set of three-dimensional space sample points;

[0019] Construct a sample matrix based on the sample point set, calculate the column mean, derive a standardized matrix based on the column mean, process the standardized matrix to obtain the covariance matrix, obtain the eigenvectors and eigenvalues ​​of the covariance matrix, and sort the eigenvalues.

[0020] The direction of the eigenvector corresponding to the largest eigenvalue is selected as the principal direction, and the direction of the eigenvector corresponding to the second largest eigenvalue is selected as the secondary direction; the principal direction and the secondary direction are perpendicular.

[0021] A separate set of feature lines is constructed for all arrows based on their arrow ID attributes. Determine the orientation of all feature lines in each set. If their angle direction matches the main direction, add them to the set of feature lines representing the main direction of that arrow. If the angle direction of the arrow is consistent with the secondary direction, then it is added to the set of secondary direction feature lines of that arrow. In the diagram, i represents the i-th arrow, and k represents the number of feature lines;

[0022] Update the set based on the parallelism between the principal and secondary directions and the feature lines, as well as the perpendicularity between the principal and secondary directions. and By taking the Plück coordinates of the characteristic lines in the set, we can obtain the new Plück coordinates of all characteristic lines in the two sets.

[0023] Specifically, the method for obtaining the new Plück coordinates is as follows:

[0024] For sets and The feature lines in the set are numbered, and the numbering of the feature lines in the same set is different;

[0025] Retained Set Z-axis coordinates C of all feature lines at both ends j,1 and C j,2 And execute according to the feature line number,

[0026] Projecting all feature lines onto the xoy plane of the camera coordinate system yields the feature line projections. The planar coordinates of the two endpoints of each feature line projection are then obtained. j,1,b j,1 ) and (a j,2 ,b j,2 ),

[0027] Based on the planar coordinates of the two endpoints, update the Plück coordinates of each corresponding feature line as (a j,1 -a j,2 b j,1 -b j,2 C j,1 -C i,2 (p1, p2, p3), where,

[0028]

[0029] a j,1 The x-axis coordinate of the starting point of characteristic line j is represented by b. j,1 The y-coordinate of the starting point of feature line j, C j,1 a represents the Z-axis coordinate of the starting point of feature line j. j,2 b represents the x-axis coordinate of the endpoint of characteristic line j. j,2 The y-coordinate of the endpoint of feature line j, C j,2 p1, p2, and p3 represent the Z-axis coordinates of the endpoint of characteristic line j, and p1, p2, and p3 represent the directional components of the Neo-Pluke coordinates of characteristic line j.

[0030] Obtain the set according to the feature line number. New Plück coordinates for the remaining feature lines;

[0031] By update collection A method for updating the set of Plück coordinates of all feature lines. The Plück coordinates of all feature lines are given.

[0032] As a preferred technical solution, the method for constructing the pose rotation matrix is ​​as follows:

[0033] Construct an initial pose rotation matrix based on the relative pose γ between any two chosen arrows. f,g =(γ f ) -1 γ g The pose rotation matrices of the two arrows are fitted multiple times to obtain candidate pose rotation matrices, and the first and second elements of each candidate pose rotation matrix are selected; where γ f,g Indicates the relative pose of arrows f and g, γ f Indicates the pose of arrow f, γ g Indicates the pose of arrow g;

[0034] The error value of the corresponding candidate pose rotation matrix is ​​calculated based on the first element and the second element, and the error values ​​are sorted to select the candidate pose rotation matrix with the smallest error value.

[0035] As a preferred technical solution, the method for recovering the second arrow based on the pose rotation matrix is ​​as follows:

[0036] Obtain information about the known first arrow, including the pose and primary and secondary directions of the first arrow;

[0037] The first arrow plane is taken as the xoy plane, the principal vector is set as the x-axis, the secondary vector is set as the y-axis, the cross product of the principal vector and the secondary vector is taken as the Z-axis, and the center point of the first arrow plane is taken as the origin.

[0038] Obtain the corresponding pose rotation matrix of the second arrow, and perform translation or rotation around the Z-axis in the xoy plane based on the pose rotation matrix.

[0039] According to a second aspect of the present invention, an arrow optimization modeling apparatus is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the aforementioned arrow optimization modeling method.

[0040] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0041] Compared with existing technologies, this invention optimizes the arrow objects and the edge feature lines of the arrows, and proposes three optimization schemes. Based on the constraint relationships between feature lines and arrows, between feature lines and arrow objects, large-scale nonlinear optimization is further performed to adjust the pose to eliminate the accumulated error in the system and improve the accuracy of arrow reconstruction in SLAM system. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a diagram showing the reconstruction effect of the arrow in Embodiment 1 of the present invention;

[0044] Figure 3 This is a schematic diagram of the main and secondary directions in Embodiment 2 of the present invention;

[0045] Figure 4 This is a diagram showing the reconstruction effect of the arrow without added constraints in Embodiment 2 of the present invention;

[0046] Figure 5 This is a diagram showing the reconstruction effect of the arrow in Embodiment 2 of the present invention;

[0047] Figure 6 This is a diagram showing the reconstruction effect of the arrow in Embodiment 3 of the present invention; Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0049] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0050] This invention reconstructs arrow models based on the constraint relationships between feature lines and / or arrows. These constraint relationships refer to planar constraints between feature lines and arrows, parallel and perpendicular constraints between feature lines, or coplanar constraints between arrows. The method categories and corresponding flowcharts are shown below. Figure 1 As shown.

[0051] Example 1:

[0052] This embodiment provides an arrow optimization modeling method, which reconstructs the arrow model based on the planar constraints between the feature line and the arrowhead. The detailed steps are as follows:

[0053] S11. Extract candidate arrow feature lines and assign semantic information to the candidate feature lines. The semantic information includes a first attribute and a second attribute. The first attribute is the arrow ID and the second attribute is the feature line number.

[0054] S111. Obtain the first attribute: When multiple candidate feature lines are coplanar with the same arrow and can form a closed figure, the ID of the arrow is used as the first attribute of the multiple candidate feature lines.

[0055] S112. Obtain the second attribute: Number the feature lines to obtain the second attribute of the feature lines, wherein the feature lines with the same first attribute are numbered differently.

[0056] S12. Based on semantic information, the feature lines are transformed into 3D to obtain 3D feature lines. The Plück coordinates of the 3D feature lines in the arrow coordinate system are obtained. Specifically, the Plück coordinates are obtained as follows:

[0057] S121. Filter the candidate feature lines according to the first attribute, and select the candidate feature lines with the same arrow ID as the feature lines.

[0058] S122. Add a binary edge to the feature line with an arrow and the first attribute being the corresponding arrow ID to obtain a 3D feature line;

[0059] S123. Perform a relative pose transformation on the 3D feature lines from the world coordinate system to the visual coordinate system to obtain the 3D Plück coordinates in the visual coordinate system.

[0060] S124. Based on the second attribute, repeat steps S121 to S123 to obtain the Plück coordinates of multiple feature lines with the same first attribute.

[0061] S13. Perform line and surface constraint optimization, transforming the Plück coordinates of the 3D feature lines from the arrow coordinate system to the world coordinate system, and then transforming the Plück coordinates in the world coordinate system according to the formula. Let L be the Plück coordinates of the camera coordinate system, such that the 3D feature line and the arrow lie in the same plane, where L M The coordinates are in the camera coordinate system. m in the Plück coordinate system M Transpose of a vector on the axis In the Plück coordinate system, d M The transpose of the vector on the axis, connected by 3D feature lines to form a closed shape, is the reconstructed arrow.

[0062] Using the above method, one of the arrows in the area map is reconstructed using a SLAM system. The reconstruction result is as follows: Figure 2 As shown.

[0063] Example 2:

[0064] This embodiment provides an arrow optimization modeling method, which reconstructs the arrow model based on the parallel and perpendicular constraints between feature lines, specifically including the following steps:

[0065] S21. Number all arrows in the area scene, extract feature lines, and use the arrow numbers as the third attribute.

[0066] S22. Based on the PCA algorithm, perform principal component analysis on feature lines with the same third attribute to obtain the corresponding principal and secondary directions of the arrow. Establish the relationship between the direction vectors of all feature lines of the arrow and the principal and secondary directions. The specific steps include:

[0067] S221. Add binary edges of feature lines with the same arrow ID to obtain 3D feature lines, and represent the 3D feature lines using Plück coordinates. Sample the 3D feature lines to obtain a set of three-dimensional space sample points.

[0068] S222. Construct a sample matrix based on the sample point set, calculate the column mean, derive the standardized matrix based on the column mean, process the standardized matrix to obtain the covariance matrix, find the eigenvectors and eigenvalues ​​of the covariance matrix, and sort the eigenvalues.

[0069] S223. Select the direction of the eigenvector corresponding to the largest eigenvalue as the principal direction, and the direction of the eigenvector corresponding to the second largest eigenvalue as the secondary direction. The principal direction and the secondary direction are perpendicular. The effect is as follows: Figure 3 As shown;

[0070] S224. Construct a separate set of feature lines for all arrows based on the arrow ID attribute. Determine the orientation of all feature lines in each set. If their angle direction matches the main direction, add them to the set of feature lines representing the main direction of that arrow. If the angle direction of the arrow is consistent with the secondary direction, then it is added to the set of secondary direction feature lines of that arrow. In the diagram, i represents the i-th arrow, and k represents the number of feature lines;

[0071] S225, Regarding sets and The feature lines in the set are numbered, and the numbering of the feature lines in the same set is different;

[0072] S226, Retained Set Z-axis coordinates C of all feature lines at both ends j,1 and C j,2 Projecting all feature lines onto the xoy plane of the camera coordinate system yields the feature line projections. The planar coordinates of the two endpoints of each feature line projection are then obtained. j,1 ,b j,1 ) and (a j,2 ,b j,2), and update the Plück coordinates of each corresponding feature line based on the planar coordinates of the two endpoints as (a j,1 -a j,2 b j,1 -b j,2 C j,1 -C i,2 (p1, p2, p3), where,

[0073]

[0074] a j,1 The x-axis coordinate of the starting point of characteristic line j is represented by b. j,1 The y-coordinate of the starting point of feature line j, C j,1 a represents the Z-axis coordinate of the starting point of feature line j. j,2 b represents the x-axis coordinate of the endpoint of characteristic line j. j,2 The y-coordinate of the endpoint of feature line j, C j,2 p1, p2, and p3 represent the Z-axis coordinates of the endpoint of characteristic line j, and p1, p2, and p3 represent the directional components of the Neo-Pluke coordinates of characteristic line j.

[0075] S227. Repeat step S226 to obtain the set according to the feature line number. New Plück coordinates for the remaining feature lines;

[0076] S228, By Updated Set A method for updating the set of Plück coordinates of all feature lines. The Plück coordinates of all feature lines are given.

[0077] S23. Update the Plück coordinates of the feature lines according to the relationship; update the Plück coordinates of the feature lines according to their numbers, and connect the feature lines after updating the Plück coordinates to form a closed figure, which is the reconstructed arrow model.

[0078] Two methods, the unconstrained modeling method and the method described above, were used to reconstruct one arrow in the region map using a SLAM system. The results of the unconstrained modeling method are shown below. Figure 4 As shown, Figure 5 The image shows the reconstruction of the arrow model based on the parallel and perpendicular constraints between feature lines; comparison. Figure 4 and Figure 5 It is evident that adding constraints between feature lines enhances the coplanarity of the same arrow feature line. Therefore, using this method in a SLAM system results in better accuracy in reconstructing the arrow.

[0079] Example 3:

[0080] This embodiment provides an arrow optimization modeling method, which reconstructs the arrow model based on the coplanar constraints between arrows. The specific steps include:

[0081] S31. Construct a pose rotation matrix, wherein the pose rotation matrix is ​​a 3×3 matrix and the elements in the third row are [0, 0, 1] in sequence, and the specific steps include:

[0082] S311. Construct the initial pose rotation matrix, based on the relative pose γ between any two selected arrows. f,g =(γ f ) -1 γ g The pose rotation matrices of the two arrows are fitted multiple times to obtain candidate pose rotation matrices, and the first and second elements of each candidate pose rotation matrix are selected; where γ f,g Indicates the relative pose of arrows f and g, γ f Indicates the pose of arrow f, γ g Indicates the pose of arrow g;

[0083] S312. Calculate the error value of the corresponding candidate pose rotation matrix based on the first element and the second element, sort the error values, and select the candidate pose rotation matrix with the smallest error value.

[0084] S32. Obtain the 3D model of the first arrow in the regional map as the initial model, and identify the second arrow based on the initial model. The first arrow is all the arrows of different specifications that appear for the first time in the identification process, and the second arrow is the arrow with the same specifications as the first arrow that appears repeatedly in the identification process. The first arrow and the second arrow are coplanar.

[0085] S33. Recover the pose of the second arrow using the pose rotation matrix to achieve arrow modeling, specifically as follows;

[0086] S331. Obtain information about the known first arrow, including the pose and primary and secondary directions of the first arrow;

[0087] S332. The first arrow plane is taken as the xoy plane, the principal vector is set as the x-axis, the secondary vector is set as the y-axis, the cross product of the principal vector and the secondary vector is taken as the Z-axis, and the center point of the first arrow plane is the origin.

[0088] S333. Obtain the corresponding pose rotation matrix of the second arrow, and perform translation or rotation around the Z-axis on the xoy plane based on the pose rotation matrix.

[0089] In this embodiment, the poses of the straight first arrow and the turning first arrow are obtained, and the second arrow in the area map is reconstructed using the method described above. The final result is as follows. Figure 6 As shown.

[0090] The above is an introduction to the method embodiments. The following describes the solution of the present invention further through device embodiments.

[0091] Example 4:

[0092] This embodiment also provides an arrow optimization modeling apparatus, including a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0093] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0094] The processing unit executes the various methods and processes described above, such as methods S11-S13, S21-S23, and / or S31-33. For example, in some embodiments, methods S11-S13, S21-S23, and / or S31-33 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S11-S13, S21-S23, and / or S31-33 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S11-S13, S21-S23, and / or S31-33 by any other suitable means (e.g., by means of firmware).

[0095] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0096] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An arrow optimization modeling method, characterized in that, This method reconstructs an arrow model based on the constraint relationships between objects; the objects include feature lines and / or arrows; the constraint relationships include planar constraints between feature lines and arrows, parallel and perpendicular constraints between feature lines, or coplanar constraints between arrows; wherein... Reconstructing the arrow model based on the planar constraints between the feature lines and the arrowhead includes: extracting candidate arrow feature lines and assigning semantic information to them; 3Dizing the feature lines based on the semantic information to obtain 3D feature lines, and obtaining the Plück coordinates of the 3D feature lines in the arrow coordinate system; performing line-plane constraint optimization, transforming the Plück coordinates of the 3D feature lines from the arrow coordinate system to the world coordinate system, and then transforming the Plück coordinates in the world coordinate system to the Plück coordinates in the camera coordinate system, so that the 3D feature lines and the arrowhead are located in the same plane, and connecting the 3D feature lines to form a closed shape, which is the reconstructed arrowhead; the semantic information includes a first attribute and a second attribute; the first attribute is the arrow ID, and the second attribute is the feature line number; The process of reconstructing an arrow model based on the parallel and perpendicular constraints between the feature lines includes: numbering all arrows in the scene, extracting feature lines and using the arrow number as the third attribute; performing principal component analysis on feature lines with the same third attribute using the PCA algorithm to obtain the corresponding primary and secondary directions of the arrow, and establishing the relationship between the direction vectors of all feature lines of the arrow and the primary and secondary directions; updating the Plück coordinates of the feature lines according to the relationship; updating the Plück coordinates of the feature lines according to their numbers, and connecting the feature lines after updating the Plück coordinates to form a closed figure, which is the reconstructed arrow model. Reconstructing the arrow model based on the coplanar constraints between the arrows includes: constructing a pose rotation matrix, wherein the pose rotation matrix is ​​a 3×3 matrix and the elements in the third row are [0, 0, 1] in sequence; obtaining the 3D model of the first arrow in the region map as the initial model, and identifying the second arrow based on the initial model; recovering the pose of the second arrow using the pose rotation matrix to realize arrow modeling; the first arrow is all the arrows of different specifications that appear for the first time during the identification process, and the second arrow is the arrow with the same specifications as the first arrow that appears repeatedly during the identification process, and the first arrow and the second arrow are coplanar.

2. The arrow optimization modeling method according to claim 1, characterized in that, The method for obtaining the first attribute is as follows: when multiple candidate feature lines are coplanar with the same arrow and can form a closed shape, the ID of the arrow is used as the first attribute of the multiple candidate feature lines. The method for obtaining the second attribute is as follows: the feature lines are numbered to obtain the second attribute of the feature lines, wherein the feature lines with the same first attribute are numbered differently.

3. The arrow optimization modeling method according to claim 1, characterized in that, Obtaining the Plück coordinates of a feature line specifically includes: Candidate feature lines are filtered based on the first attribute, and candidate feature lines with the same arrow ID are selected as feature lines. Add arrows and binary edges to the feature lines whose first attribute is the corresponding arrow ID to obtain 3D feature lines; The 3D feature lines are subjected to a relative pose transformation from the world coordinate system to the visual coordinate system to obtain 3D Plück coordinates in the visual coordinate system. Based on the second attribute, obtain the Plück coordinates of multiple feature lines with the same first attribute.

4. The arrow optimization modeling method according to claim 1, characterized in that, The specific transformation of Plück coordinates involves converting them from the arrow coordinate system to the camera coordinate system, according to the formula... Proceed; where L M The coordinates are in the camera coordinate system. m in the Plück coordinate system M Transpose of a vector on the axis In the Plück coordinate system, d M Transpose of a vector on the axis.

5. The arrow optimization modeling method according to claim 1, characterized in that, Reconstructing the arrow based on the parallel and perpendicular constraints between feature lines, specifically including: Add binary edges to feature lines with the same arrow ID to obtain 3D feature lines, represent the 3D feature lines using Plück coordinates, and sample the 3D feature lines to obtain a set of three-dimensional space sample points; Construct a sample matrix based on the sample point set, calculate the column mean, derive a standardized matrix based on the column mean, process the standardized matrix to obtain the covariance matrix, obtain the eigenvectors and eigenvalues ​​of the covariance matrix, and sort the eigenvalues. The direction of the eigenvector corresponding to the largest eigenvalue is selected as the principal direction, and the direction of the eigenvector corresponding to the second largest eigenvalue is selected as the secondary direction; the principal direction and the secondary direction are perpendicular. A separate set of feature lines is constructed for all arrows based on their arrow ID attributes. Determine the orientation of all feature lines in each set. If their angle direction matches the main direction, add them to the set of feature lines representing the main direction of that arrow. If the angle direction of the arrow is consistent with the secondary direction, then it is added to the set of secondary direction feature lines of that arrow. In the diagram, i represents the i-th arrow, and k represents the number of feature lines; Update the set based on the parallelism between the principal and secondary directions and the feature lines, as well as the perpendicularity between the principal and secondary directions. and By taking the Plück coordinates of the characteristic lines in the set, we can obtain the new Plück coordinates of all characteristic lines in the two sets.

6. The arrow optimization modeling method according to claim 5, characterized in that, The method for obtaining the new Plück coordinates is as follows: For sets and The feature lines in the set are numbered, and the numbering of the feature lines in the same set is different from that of the feature lines in the same set. Retained Set Z-axis coordinates c of all feature lines j,1 and c j,2 And execute according to the feature line number, Projecting all feature lines onto the xoy plane of the camera coordinate system yields the feature line projections. The planar coordinates of the two endpoints of each feature line projection are then obtained. j,1 ,b j,1 ) and (a j,2 ,b j,2 ), Based on the planar coordinates of the two endpoints, update the Plück coordinates of each corresponding feature line as (a j,1 -a j,2 b j,1 -b j,2 C j,1 -C i,2 (p1, p2, p3), where, a j,1 The x-axis coordinate of the starting point of characteristic line j is represented by b. j,1 The y-coordinate of the starting point of feature line j, C j,1 a represents the Z-axis coordinate of the starting point of feature line j. j,2 b represents the x-axis coordinate of the endpoint of characteristic line j. j,2 The y-coordinate of the endpoint of feature line j, C j,2 p1, p2, and p3 represent the Z-axis coordinates of the endpoint of characteristic line j, and p1, p2, and p3 represent the directional components of the Neo-Pluke coordinates of characteristic line j. Obtain the set according to the feature line number. New Plück coordinates for the remaining feature lines; By update collection A method for updating the set of Plück coordinates of all feature lines. The Plück coordinates of all feature lines are given.

7. The arrow optimization modeling method according to claim 1, characterized in that, The method for constructing the pose rotation matrix is ​​as follows: Construct an initial pose rotation matrix based on the relative pose γ between any two chosen arrows. f,g =(γ f ) -1 γ g The pose rotation matrices of the two arrows are fitted multiple times to obtain candidate pose rotation matrices, and the first and second elements of each candidate pose rotation matrix are selected; where γ f,g Indicates the relative pose of arrows f and g, γ f Indicates the pose of arrow f, γ g Indicates the pose of arrow g; The error value of the corresponding candidate pose rotation matrix is ​​calculated based on the first element and the second element, and the error values ​​are sorted to select the candidate pose rotation matrix with the smallest error value.

8. The arrow optimization modeling method according to claim 1, characterized in that, The method for recovering the second arrow based on the pose rotation matrix is ​​as follows: Obtain information about a known first arrow, including the pose, primary direction, and secondary direction of the first arrow; With the plane of the first arrow as the xoy plane, the main direction vector is set as the x-axis, the secondary direction vector is set as the y-axis, the cross product of the main direction vector and the secondary direction vector is taken as the Z-axis, and the center point of the first arrow plane is the origin. Obtain the corresponding pose rotation matrix of the second arrow, and perform translation or rotation around the Z-axis in the xoy plane based on the pose rotation matrix.

9. An arrow optimization modeling device, characterized in that, The method includes a memory, a processor, and a program stored in the memory, characterized in that the processor executes the program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images