Optimized positioning method and device based on visual label

Abstract

The application relates to an optimized positioning method and device based on a visual label, wherein the optimized positioning method comprises the following steps: recognizing a set visual label in a target area, extracting a candidate feature line in a preset range of the visual label, and giving semantic information to the candidate feature line, wherein the semantic information comprises a first attribute, a second attribute and a third attribute; screening the candidate feature line with the first attribute being a visual label feature line, screening all feature lines with the same visual label ID according to the second attribute, performing graph optimization processing on all feature lines according to the third attribute, and obtaining 3D Prouk coordinates of all feature lines in a visual label coordinate system; converting the 3D Prouk coordinates of all feature lines from a visual coordinate system into a camera coordinate system, performing line-to-surface constraint optimization processing on all feature lines, and realizing positioning of the visual label; and the optimization device is used for realizing the above method. Compared with the prior art, the application has the advantages of more accurate positioning and the like.

Description

Technical Field

[0001] The present invention relates to the field of visual processing, and in particular to an optimized positioning method and device based on visual tags. Background Art

[0002] In a visual SLAM system, adding the ability to identify and locate visual tags (markers) can improve the overall SLAM system's ability to locate the camera, thereby increasing the accuracy of mapping. Because visual tags have their own scale, they can avoid the scale uncertainty problem of monocular cameras. Moreover, visual tags are stored as objects in the SLAM map, but they have a 6-degree-of-freedom pose, which can achieve more accurate mapping.

[0003] It can be seen that the accuracy of positioning the visual tags is closely related to the accuracy of SLAM mapping. Chinese patent CN111739088B discloses a positioning method based on visual tags. Although it solves the problems of slow camera positioning or positioning failure during the mapping process through positioning circles and coding circles, it only realizes the positioning of visual tags from a two-dimensional level and cannot accurately reflect the position of visual tags in the environment at the three-dimensional spatial level. Problems such as inaccurate positioning and low mapping accuracy may occur.

[0004] Therefore, avoiding the problem of low mapping accuracy caused by inaccurate positioning of visual labels is a problem that needs to be solved. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a visual label edge segment plane optimization method for optimizing visual label positioning.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided a method for optimizing positioning based on visual tags, the method comprising:

[0008] Preprocessing: Identify the visual labels set in the target area, extract candidate feature lines within the preset range of the visual labels, and assign semantic information to the candidate feature lines. The semantic information includes a first attribute, a second attribute, and a third attribute. The first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number.

[0009] Obtaining 3D Plücker coordinates: Screening candidate feature lines whose first attribute is that they are visual label feature lines, screening all feature lines with the same visual label ID based on the second attribute, performing graph optimization processing on all the feature lines based on the third attribute, and obtaining 3D Plücker coordinates of all the feature lines in the visual label coordinate system;

[0010] Line-to-surface constraint optimization processing: converting the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, performing line-to-surface constraint optimization processing on all the feature lines, and realizing the positioning of the visual labels.

[0011] As a preferred technical solution, the method for obtaining the first attribute is to use the LSD algorithm to judge the candidate feature lines.

[0012] As a preferred technical solution, the judgment method is:

[0013] If the distances between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints are both less than a set threshold, the first attribute of the candidate feature line is set as a visual label feature line;

[0014] If the distance between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints is greater than or equal to the set threshold, the first attribute is not set for the candidate feature line.

[0015] As a preferred technical solution, the method for obtaining the second attribute is: based on the first attribute, the visual label ID is used as the second attribute of the candidate feature line. When multiple candidate feature lines with the first attribute are coplanar with the same visual label and can form a closed figure, the ID of the visual label is used as the second attribute of the multiple candidate feature lines.

[0016] As a preferred technical solution, the characteristic lines are numbered to obtain the third attribute of the characteristic lines, wherein characteristic lines with the same second attribute have different numbers.

[0017] As a preferred technical solution, the step of obtaining 3D Plücker coordinates further includes:

[0018] Add the binary edge of the feature line with the visual label and the second attribute corresponding to the visual label ID to obtain the 3D feature line;

[0019] Performing relative pose transformation from the world coordinate system to the visual coordinate system on the 3D feature line to obtain the 3D Plücker coordinates in the visual coordinate system;

[0020] According to the third attribute, 3D Plücker coordinates of a plurality of feature lines having the same second attribute are obtained.

[0021] As a preferred technical solution, in the line-to-surface constraint optimization processing step, the transformation of the 3D Plücker coordinates from the visual coordinate system to the camera coordinate system is performed according to the formula Carry out; among them L M is the coordinate in the camera coordinate system, is m in the Plücker coordinate system Mtranspose of the vector on the axis, the d M transpose of the vector on the axis.

[0022] According to a second aspect of the present application, there is provided an optimized positioning device, the device comprising:

[0023] A first module, preprocessing: identifying a set visual tag in a target area, extracting a candidate feature line within a preset range of the visual tag, and assigning semantic information to the candidate feature line, the semantic information including a first attribute, a second attribute, and a third attribute; the first attribute being whether it is a visual tag feature line, the second attribute being a visual tag ID, and the third attribute being a feature line serial number;

[0024] A second module, configured to obtain 3D Prouk coordinates, screen the candidate feature line with the first attribute being a visual tag feature line, screen all feature lines with the same visual tag ID according to the second attribute, and perform graph optimization processing on the all feature lines according to the third attribute, to obtain 3D Prouk coordinates of the all feature lines in a visual tag coordinate system;

[0025] A third module, line-to-plane constraint optimization processing, converting the 3D Prouk coordinates of the all feature lines from a visual coordinate system to a camera coordinate system, performing line-to-plane constraint optimization processing on the all feature lines, and realizing positioning of the visual tag.

[0026] According to a third aspect of the present application, there is provided an electronic device comprising a memory and a processor, the memory having a computer program stored thereon, and the processor realizing the method when executing the program.

[0027] According to a fourth aspect of the present application, there is provided a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to realize the method.

[0028] Compared with the prior art, the present application has the following advantages:

[0029] The visual tag object is added with a coplanar constraint of its edge line segment to itself, the feature line of the visual tag is characterized by Prouk coordinates, the position relationship of each feature line in a three-dimensional coordinate system is accurately described, the coplanar constraint of the feature line and the visual tag is realized, and the positioning is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 A flowchart of the optimized positioning method based on a visual tag of the present application;

[0031] Figure 2 A schematic diagram of a visual tag in the present application;

[0032] Figure 3 This is a flow chart of the detection of visual tags in the present invention;

[0033] Figure 4 Schematic diagram of the Plücker coordinate expression of features in 3D space in the present invention;

[0034] Figure 5 Schematic diagram of semantic information of visual label feature lines in the present invention;

[0035] Figure 6 Plücker coordinates of the feature line seq3 in Example 2 of the present invention in the world coordinate system and the visual label coordinate system;

[0036] Figure 7 The actual scene graph collected in Example 2 of the present invention;

[0037] Figure 8 This is a front view showing the optimization effect in Example 2 of the present invention;

[0038] Figure 9 This is a side view showing the optimization effect in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0040] It should be understood that the steps described in the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0041] The term "determining" broadly encompasses a variety of actions, and may include obtaining, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and the like, and may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like, as well as parsing, selecting, choosing, establishing, and the like. Other terms are defined in the following description.

[0042] It should be noted that the concepts of "first" and "second" mentioned in this embodiment are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "multiple" mentioned in this embodiment are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0044] Example 1:

[0045] This embodiment provides an optimized positioning method based on visual labels, which is applicable to the field of visual processing. By adding coplanar constraints of the edge segments of the visual label object to itself, the line segment information in the scene can be fully reconstructed.

[0046] In one embodiment, the flowchart of the optimization positioning method based on visual tags is as follows: Figure 1 As shown, the specific steps include:

[0047] S1. Preprocessing: Identify the visual labels set in the target area, extract candidate feature lines within the preset range of the visual labels, and assign semantic information to the candidate feature lines. The semantic information includes a first attribute, a second attribute, and a third attribute. The first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number.

[0048] S2. Obtaining 3D Plücker coordinates: Screening candidate feature lines whose first attribute is that they are visual label feature lines, screening all feature lines with the same visual label ID based on the second attribute, performing graph optimization processing on all the feature lines based on the third attribute, and obtaining 3D Plücker coordinates of all the feature lines in the visual label coordinate system;

[0049] S3. Line-to-surface constraint optimization processing: converting the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, performing line-to-surface constraint optimization processing on all the feature lines, and realizing the positioning of the visual labels.

[0050] As a preferred solution, step S1 further includes:

[0051] S11, Feature detection: Figure 2 As shown in the figure, the visual tag is similar to a QR code; its detection and recognition process mainly utilizes the binary concept within the computer. By detecting its four edge corners and extracting the outline for binary processing, the "0" and "1" bit streams are used, and several geometric shapes corresponding to the binary are used to represent the numerical value and other information to complete the decoding of the visual tag; the detection process is as follows Figure 3 As shown, it includes image preprocessing, contour extraction, ID decoding and corner refinement.

[0052] S12. Visual Label Pose Estimation: In the experiment, we used the infinitesimal plane-based pose estimation algorithm (IPPE), which is robust to noise and abnormal data and has high computational efficiency.

[0053] Construct a visual label plane coordinate system, define the center point of the visual label as the origin of its own coordinate system, define its own plane as the xoy plane, the x-axis is horizontal to the right, the y-axis is vertical downward, and the z-axis is perpendicular to its own plane;

[0054] Assuming the size of the visual label is s, the coordinates of the four corner points in the coordinate system of the visual label to which they belong can be obtained:

[0055]

[0056] Assuming that the coordinates of A, B, C, and D in the camera coordinate system are known, and the correspondence between the corner 3D and 2D points is known, the relative pose transformation between the visual label coordinate system and the camera coordinate system can be calculated.

[0057] S13. Line segments are given semantic information:

[0058] All candidate feature lines in the space are detected and represented according to Plücker coordinates as follows Figure 4 As shown, "O" is the origin of the coordinate system, "M" and "N" are the two endpoints of the characteristic line, Indicates the normal vector of the plane formed by the two endpoints of the feature line and the origin, The vector representing the characteristic line;

[0059] according to Figure 5 As shown, semantic information about the three attributes of the visual label is added to PLM-SLAM. The semantic information includes: the first attribute, the second attribute and the third attribute. The method for obtaining the three attributes is:

[0060] S131. Obtain the first attribute: Use the LSD algorithm to judge the candidate feature line. If the distance between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints is less than the set threshold, set the first attribute of the candidate feature line as a visual label feature line; if the distance between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints is greater than or equal to the set threshold, do not set the first attribute for the candidate feature line. The reason why the corner points of the visual label are not directly used to connect adjacent points to generate a new feature line is that: the corner point extraction of the visual label is not completely accurate and there will be certain errors, while the LSD algorithm can more accurately extract the edge feature information of the visual label; and constructing a new feature line will cause data redundancy to a certain extent;

[0061] S132. Obtain the second attribute: Based on the first attribute, use the visual label ID as the second attribute of the candidate feature line. When multiple candidate feature lines with the first attribute are coplanar with the same visual label and can form a closed figure, the ID of the visual label is used as the second attribute of the multiple candidate feature lines.

[0062] S133. Obtain the third attribute: number the feature lines to obtain the third attribute of the feature lines, wherein the feature lines with the same second attribute are numbered differently. In this embodiment, the upper and lower parallel edges in the visual label are specified as seq1 and seq3, and the left and right parallel edges are specified as seq2 and seq4. In this way, the feature lines belonging to the same visual label have different attributes, that is, the vertical and parallel relationships between different edges can be known through different sequences, so that straight lines with different attributes can be optimized more reasonably when adding graph optimization edges. However, there is no specific numbering rule for the numbering order, and it is only necessary to ensure that the numbers of all feature lines with the same visual label are different.

[0063] As a preferred technical solution, step S2 further includes:

[0064] In this embodiment, a visual tag in a regional map is used as an example to explain how to obtain the 3D Plücker coordinates of the visual tag.

[0065] Under the conditions of three attributes, find the set of all local visual labels in the local map by the first attribute Traverse and set and find the feature line corresponding to the visual label through the second attribute, add the binary edge of the feature line of the visual label ID in the graph optimization to obtain the 3D feature line; obtain the 3D Plücker coordinates in the visual label coordinate system through the relative pose change of the world coordinate system and the visual label coordinate system; obtain the 3D Plücker coordinates of all the feature lines of the visual label according to the third attribute. The conversion process is as follows: Figure 6 As shown in the figure, the coordinate system on the left is the world coordinate system, and the coordinate system on the right is the visual label coordinate system.

[0066] As a preferred technical solution, in step S3:

[0067] The transformation of the 3D Plücker coordinates from the visual coordinate system to the camera coordinate system is as follows: Carry out; among them L M is the coordinate in the camera coordinate system, is m in the Plücker coordinate system M The transpose of the vector on the axis, is d in the Plücker coordinate system M Transpose of a vector on an axis;

[0068] Taking the characteristic line seq3 as an example, Should be x M Axis direction parallel, Should be with z M The axes are parallel, and the following relationship holds:

[0069]

[0070] is a unit vector parallel to the x-axis, is a unit vector parallel to the z-axis, so The first two terms of should be constrained to be 0, The last two terms of should be constrained to 0, so as to realize the constraint of line to visual coordinate plane. For feature lines seq1 and seq3, the above formula is used for optimization, while for feature lines seq2 and seq4, only As long as the feature lines are parallel to the unit vector of the y-axis, they can be parallel to each other. By setting parallel constraints between the feature lines and the coordinate axes in the visual coordinate system, the parallel and perpendicular constraints between different feature lines can be optimized indirectly, making the semantic information clearer and the map construction more accurate.

[0071] Example 2:

[0072] In one embodiment, a visual label-based optimization positioning device is provided, which includes a first module, a second module and a third module, wherein the first module is used to implement the preprocessing function in the optimization positioning method of Example 1, the second module is used to implement the function of obtaining the 3D Plücker coordinates of the characteristic line in the above method, and the third module is used to implement the line-to-surface constraint optimization processing function in the above method.

[0073] In this embodiment, the device is first used to collect images of the area map, such as Figure 7 As shown, the image is processed in the device to obtain the processed image, and the result is as follows Figure 8 and Figure 9 As shown in the figure. The red circle in the figure is the visual label; the blue line circled without other blue lines inside is the candidate feature line in the environment; the blue line circled with other blue lines in the shape of a cross inside represents the past keyframes in the environment, that is, the historical position of the odometry; the green line circled with other green lines in the shape of a cross inside represents the current keyframe, that is, the current position of the odometry.

[0074] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0075] In one embodiment, an electronic device is provided, which includes a central processing unit (CPU) that can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). Various programs and data required for device operation can also be stored in the RAM. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0076] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0077] The processing unit performs the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S3 by any other appropriate means (e.g., by means of firmware).

[0078] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0079] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage medium would include one or more lines of electrical wire, portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0081] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, and the present disclosure is not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

[0082] The above description is merely illustrative of the application and not restrictive. The scope of the application should be determined by reasonable interpretation of the appended claims and thus variations of the embodiments disclosed herein that come within the scope of the claims and equivalence thereof are intended to fall within the scope of the application.

Claims

1. An optimization positioning method based on visual labels, characterized in that: The method includes: Preprocessing: Identify the visual labels set in the target area, extract candidate feature lines within the preset range of the visual labels, and assign semantic information to the candidate feature lines, wherein the semantic information includes a first attribute, a second attribute, and a third attribute; the first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number; the method for obtaining the first attribute is to use the LSD algorithm to judge the candidate feature line, and the judgment method is: if the point distance between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints is less than a set threshold, the first attribute of the candidate feature line is set to a visual label feature line; if the point distance between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints is greater than or equal to the set threshold, the first attribute is not set for the candidate feature line; Obtaining 3D Plücker coordinates: Screening candidate feature lines whose first attribute is a visual label feature line, screening all feature lines with the same visual label ID based on the second attribute, performing graph optimization processing on all the feature lines based on the third attribute, and obtaining 3D Plücker coordinates of all the feature lines in the visual label coordinate system; Line-to-surface constraint optimization processing: converting the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, performing line-to-surface constraint optimization processing on all the feature lines, and realizing the positioning of the visual labels.

2. The method for optimizing positioning based on visual tags according to claim 1, characterized in that: The method for obtaining the second attribute is: based on the first attribute, the visual label ID is used as the second attribute of the candidate feature line. When multiple candidate feature lines with the first attribute are coplanar with the same visual label and can form a closed figure, the ID of the visual label is used as the second attribute of the multiple candidate feature lines.

3. The method for optimizing positioning based on visual tags according to claim 1, characterized in that: The characteristic lines are numbered to obtain a third attribute of the characteristic lines, wherein characteristic lines with the same second attribute have different numbers.

4. The method for optimizing positioning based on visual tags according to claim 1, characterized in that: The steps to obtain 3D Plücker coordinates also include: Add the binary edge of the feature line with the visual label and the second attribute corresponding to the visual label ID to obtain the 3D feature line; Performing relative pose transformation from the world coordinate system to the visual coordinate system on the 3D feature line to obtain the 3D Plücker coordinates in the visual coordinate system; According to the third attribute, 3D Plücker coordinates of a plurality of feature lines having the same second attribute are obtained.

5. The method for optimizing positioning based on visual tags according to claim 1, characterized in that: In the line-to-surface constraint optimization process, the 3D Plücker coordinates are transformed from the visual coordinate system to the camera coordinate system according to the formula conduct; is the coordinate in the camera coordinate system, In the Plücker coordinate system The transpose of the vector on the axis, In the Plücker coordinate system The transpose of a vector on an axis.

6. An optimized positioning device based on visual tags, characterized in that: The device includes: The first module, preprocessing: identifies the visual labels set in the target area, extracts candidate feature lines within the preset range of the visual labels, and assigns semantic information to the candidate feature lines. The semantic information includes a first attribute, a second attribute, and a third attribute. The first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number. The second module is used to obtain 3D Plücker coordinates, screen candidate feature lines whose first attribute is that they are visual label feature lines, screen all feature lines with the same visual label ID based on the second attribute, perform graph optimization processing on all the feature lines according to the third attribute, and obtain 3D Plücker coordinates of all the feature lines in the visual label coordinate system; The third module is line-to-surface constraint optimization processing, which converts the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, and performs line-to-surface constraint optimization processing on all the feature lines to achieve the positioning of the visual labels.

7. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images