Optimized positioning method and device based on visual label

Abstract

The invention relates to an optimized positioning method and device based on a visual tag, and the method comprises the steps: recognizing the visual tag set in a target region, extracting a candidate feature line in a preset range of the visual tag, and endowing the candidate feature line with semantic information which comprises a first attribute, a second attribute and a third attribute; screening candidate feature lines of which the first attribute is visual label feature lines, screening out all feature lines with the same visual label ID according to the second attribute, and performing graph optimization processing on all the feature lines according to the third attribute to obtain 3D Pluecker coordinates of all the feature lines in a visual label coordinate system; the 3D Pluecker coordinates of all the feature lines are converted into a camera coordinate system from a visual coordinate system, line-to-plane constraint optimization processing is carried out on all the feature lines, and positioning of the visual label is achieved; the optimization device is used for implementing the method. Compared with the prior art, the method has the advantages of more accurate positioning and the like.

Description

Technical Field

[0001] The present invention relates to the field of visual processing, and in particular to an optimized positioning method and device based on visual tags. Background Art

[0002] In the visual SLAM system, adding the recognition and positioning of visual tags (Markers) can improve the overall SLAM system's ability to locate the camera and improve the accuracy of mapping. Because the visual tag has its own scale, it can avoid the scale uncertainty problem of the monocular camera, and the visual tag is stored as an object in the SLAM map but has a 6-degree-of-freedom posture, which can achieve more accurate mapping.

[0003] It can be seen that the accuracy of positioning the visual tags is closely related to the accuracy of SLAM mapping. Chinese patent CN111739088B discloses a positioning method based on visual tags. Although it solves the problems of slow camera positioning or positioning failure during mapping through positioning circles and coding circles, it only realizes the positioning of visual tags on a two-dimensional level and cannot accurately reflect the position of visual tags in the environment on a three-dimensional spatial level. Problems such as inaccurate positioning and low mapping accuracy may occur.

[0004] Therefore, avoiding the problem of low mapping accuracy caused by inaccurate positioning of visual labels is a problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a visual label edge segment plane optimization method for optimizing the positioning of visual labels.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided an optimization positioning method based on visual tags, the method comprising:

[0008] Preprocessing: Identify the visual labels set in the target area, extract candidate feature lines within the preset range of the visual labels, and assign semantic information to the candidate feature lines, where the semantic information includes a first attribute, a second attribute, and a third attribute; the first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number;

[0009] Obtaining 3D Plücker coordinates: screening candidate feature lines whose first attribute is that they are visual label feature lines, screening all feature lines with the same visual label ID according to the second attribute, performing graph optimization processing on all the feature lines according to the third attribute, and obtaining 3D Plücker coordinates of all the feature lines in the visual label coordinate system;

[0010] Line-to-surface constraint optimization processing: convert the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, perform line-to-surface constraint optimization processing on all the feature lines, and realize the positioning of the visual label.

[0011] As a preferred technical solution, the method for acquiring the first attribute is to use an LSD algorithm to judge the candidate feature lines.

[0012] As a preferred technical solution, the judgment method is:

[0013] If the distances between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints are both less than a set threshold, the first attribute of the candidate feature line is set as a visual label feature line;

[0014] If the distance between the two endpoints of the candidate feature line and the two most adjacent corner points corresponding to the endpoints is greater than or equal to the set threshold, the first attribute is not set for the candidate feature line.

[0015] As a preferred technical solution, the method for obtaining the second attribute is: based on the first attribute, the visual label ID is used as the second attribute of the candidate feature line. When multiple candidate feature lines with the first attribute are coplanar with the same visual label and can form a closed figure, the ID of the visual label is used as the second attribute of the multiple candidate feature lines.

[0016] As a preferred technical solution, the characteristic lines are numbered to obtain the third attribute of the characteristic lines, wherein the characteristic lines with the same second attribute have different numbers.

[0017] As a preferred technical solution, the step of obtaining 3D Plücker coordinates also includes:

[0018] Add the binary edge of the feature line whose visual label and second attribute are the corresponding visual label ID to obtain the 3D feature line;

[0019] The 3D feature line is transformed from the world coordinate system to the visual coordinate system to obtain the 3D Plücker coordinates in the visual coordinate system;

[0020] According to the third attribute, 3D Plücker coordinates of a plurality of feature lines having the same second attribute are obtained.

[0021] As a preferred technical solution, in the line-to-surface constraint optimization processing step, the transformation of the 3D Plücker coordinates from the visual coordinate system to the camera coordinate system is performed according to the formula Carry out; among them L M is the coordinate in the camera coordinate system, is m in the Plücker coordinate system MThe transpose of a vector on an axis, is d in the Plücker coordinate system M The transpose of a vector on an axis.

[0022] According to a second aspect of the present invention, there is provided an optimized positioning device, the device comprising:

[0023] The first module, preprocessing: identifying the visual label set in the target area, extracting the candidate feature lines within the preset range of the visual label, and assigning semantic information to the candidate feature lines, the semantic information including a first attribute, a second attribute and a third attribute; the first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number;

[0024] The second module is used to obtain 3D Plücker coordinates, screen candidate feature lines whose first attribute is that they are visual label feature lines, screen all feature lines with the same visual label ID according to the second attribute, perform graph optimization processing on all the feature lines according to the third attribute, and obtain 3D Plücker coordinates of all the feature lines in the visual label coordinate system;

[0025] The third module is line-to-surface constraint optimization processing, which converts the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, performs line-to-surface constraint optimization processing on all the feature lines, and realizes the positioning of the visual labels.

[0026] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0027] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] The coplanar constraint of the edge line segment of the visual label object is added to itself, and the feature line of the visual label is represented by Plücker coordinates, which accurately describes the positional relationship of each feature line in the three-dimensional coordinate system, realizes the coplanar constraint of the feature line and the visual label, and makes its positioning more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the optimization positioning method based on visual tags of the present invention;

[0031] Figure 2 is a schematic diagram of a visual label in the present invention;

[0032] Figure 3 This is a flow chart of the detection of visual tags in the present invention;

[0033] Figure 4 It is a schematic diagram of the Plücker coordinate expression method of the feature in the 3D space in the present invention;

[0034] Figure 5 A schematic diagram of semantic information of a visual label feature line in the present invention;

[0035] Figure 6 are the Plücker coordinates of the feature line seq3 in Example 2 of the present invention in the world coordinate system and the visual label coordinate system;

[0036] Figure 7 is the actual scene graph collected in Example 2 of the present invention;

[0037] Figure 8 It is a front view showing the optimization effect in Example 2 of the present invention;

[0038] Fig. 9 It is a side view showing the optimization effect in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be understood that the steps described in the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0041] The term "determine" broadly covers a variety of actions, which may include obtaining, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database or other data structure), ascertaining, and similar actions, and may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and similar actions, as well as parsing, selecting, choosing, establishing and similar actions, etc. The relevant definitions of other terms will be given in the following description.

[0042] It should be noted that the concepts such as "first" and "second" mentioned in this embodiment are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "plurality" mentioned in this embodiment are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0044] Embodiment 1:

[0045] This embodiment provides an optimization positioning method based on visual labels, which is applicable to the field of visual processing. By adding coplanar constraints of edge segments to the visual label object itself, the line segment information in the scene can be fully reconstructed.

[0046] In one embodiment, the flowchart of the optimization positioning method based on visual tags is as follows: Figure 1 As shown, the specific steps include:

[0047] S1. Preprocessing: Identify the visual label set in the target area, extract the candidate feature lines within the preset range of the visual label, and assign semantic information to the candidate feature lines, wherein the semantic information includes a first attribute, a second attribute, and a third attribute; the first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number;

[0048] S2. Obtaining 3D Plücker coordinates: Screening candidate feature lines whose first attribute is that they are visual label feature lines, screening all feature lines with the same visual label ID according to the second attribute, performing graph optimization processing on all the feature lines according to the third attribute, and obtaining 3D Plücker coordinates of all the feature lines in the visual label coordinate system;

[0049] S3, line-to-surface constraint optimization processing: convert the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, perform line-to-surface constraint optimization processing on all the feature lines, and realize the positioning of the visual label.

[0050] As a preferred solution, step S1 further includes:

[0051] S11, feature detection: Figure 2 As shown in the figure, the visual tag is similar to a QR code; its detection and recognition process mainly uses the binary concept inside the computer. By detecting its four edge corners and extracting the outline for binary processing, using the "0" and "1" bit streams, and using several geometric shapes corresponding to the binary to represent the value and other information to complete the decoding of the visual tag; the detection process is as follows Figure 3 As shown, it includes image preprocessing, contour extraction, ID decoding and corner refinement.

[0052] S12. Visual label pose estimation: In the experiment, the infinitesimal plane-based pose estimation algorithm (IPPE) was used, which has strong robustness to noise and abnormal data and high computational efficiency.

[0053] Construct the visual label plane coordinate system, define the center point of the visual label as the origin of its own coordinate system, define its own plane as the xoy plane, the x-axis is horizontal to the right, the y-axis is vertical downward, and the z-axis is perpendicular to its own plane;

[0054] Assuming the size of the visual label is s, the coordinates of the four corner points in the coordinate system of the visual label to which they belong can be obtained:

[0055]

[0056] Assuming that the coordinates of A, B, C, and D in the camera coordinate system are known, and the correspondence between the corner 3D and 2D points is known, the relative pose transformation between the visual label coordinate system and the camera coordinate system can be calculated.

[0057] S13, line segments are given semantic information:

[0058] All candidate feature lines in the space are detected and characterized according to Plücker coordinates as follows Figure 4 As shown, "O" is the origin of the coordinate system, "M" and "N" are the two endpoints of the characteristic line. Represents the normal vector of the plane formed by the two endpoints of the feature line and the origin. The vector representing the characteristic line;

[0059] according to Figure 5 As shown, semantic information about three attributes of visual tags is added to PLM-SLAM, and the semantic information includes: a first attribute, a second attribute, and a third attribute; the method for obtaining the three attributes is:

[0060] S131. Get the first attribute: Use the LSD algorithm to judge the candidate feature line. If the distance between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints is less than the set threshold, set the first attribute of the candidate feature line as a visual label feature line; if the distance between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints is greater than or equal to the set threshold, do not set the first attribute for the candidate feature line. The reason why the corner points of the visual label are not directly used to connect adjacent points to generate a new feature line is that: the extraction of the corner points of the visual label is not completely accurate and there will be certain errors, while the LSD algorithm can extract the edge feature information of the visual label more accurately; and constructing a new feature line will cause data redundancy to a certain extent;

[0061] S132, obtaining a second attribute: based on the first attribute, using the visual label ID as the second attribute of the candidate feature line; when multiple candidate feature lines with the first attribute are coplanar with the same visual label and can form a closed figure, the ID of the visual label is used as the second attribute of the multiple candidate feature lines.

[0062] S133. Obtain the third attribute: number the feature lines to obtain the third attribute of the feature lines, wherein the feature lines with the same second attribute are numbered differently. In the present embodiment, the upper and lower parallel edges in the visual label are designated as seq1 and seq3, and the left and right parallel edges are designated as seq2 and seq4. This allows the feature lines belonging to the same visual label to have different attributes, that is, the vertical and parallel relationships between different edges can be known through different sequences, so that straight lines with different attributes can be optimized more reasonably when adding graph optimization edges. However, there is no specific numbering rule for the order of numbering, and it is only necessary to ensure that the numbers of all feature lines with the same visual label are different.

[0063] As a preferred technical solution, step S2 also includes:

[0064] In this embodiment, a visual tag in a regional map is used as an example to explain how to obtain the 3D Plücker coordinates of the visual tag.

[0065] Under the conditions of three attributes, find the set of all local visual labels in the local map through the first attribute Traverse and set and find the feature line corresponding to the visual label through the second attribute, add the binary edge of the feature line of the visual label ID in the graph optimization, and obtain the 3D feature line; obtain the 3D Plücker coordinates in the visual label coordinate system through the relative pose change of the world coordinate system and the visual label coordinate system; obtain the 3D Plücker coordinates of all the feature lines of the visual label according to the third attribute, and the conversion process is as follows: Figure 6 As shown, the coordinate system on the left side of the figure is the world coordinate system, and the coordinate system on the right side is the visual label coordinate system.

[0066] As a preferred technical solution, in step S3:

[0067] The transformation of the 3D Plücker coordinates from the visual coordinate system to the camera coordinate system is based on the formula Carry out; among them L M is the coordinate in the camera coordinate system, is m in the Plücker coordinate system M The transpose of a vector on an axis, is d in the Plücker coordinate system M Transpose of a vector on an axis;

[0068] Taking feature line seq3 as an example, Should be x M Axis direction parallel, Should be with z M The axes are parallel, and the following relationship holds:

[0069]

[0070] is a unit vector parallel to the x-axis, is a unit vector parallel to the z-axis, so The first two terms of should be constrained to be 0, The last two terms of should be constrained to 0, so as to realize the constraint of line to visual coordinate plane. For feature lines seq1 and seq3, the above formula is used for optimization, while for feature lines seq2 and seq4, only It only needs to be parallel to the unit vector of the y-axis. By constraining each feature line to be parallel to the coordinate axis in the visual coordinate system, the parallel and vertical constraints between different feature lines are also indirectly optimized, making the semantic information clearer and the map construction more accurate.

[0071] Embodiment 2:

[0072] In one embodiment, a visual label-based optimization positioning device is provided, which includes a first module, a second module and a third module, wherein the first module is used to implement the preprocessing function in the optimization positioning method of Example 1, the second module is used to implement the function of obtaining the 3D Plücker coordinates of the feature line in the above method, and the third module is used to implement the line-to-surface constraint optimization processing function in the above method.

[0073] In this embodiment, the device is first used to collect images of the area map, such as Figure 7 As shown, the image is processed in the device to obtain the processed image, and the result is as follows Figure 8 and Fig. 9 As shown in the figure. The red circle is the visual label; the blue line is circled by a blue circle, and there are no other blue lines inside the blue circle, which indicates the candidate feature line in the environment; the blue line is circled by a blue circle, and there are other blue lines in the shape of a cross symbol inside, which indicates the past key frames in the environment, that is, the historical position of the odometer; the green line is circled by a green circle, and there are other green lines in the shape of a cross symbol inside the green circle, which indicates the current key frame, that is, the current position of the odometer.

[0074] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0075] An electronic device is provided in one embodiment, which includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0076] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0077] The processing unit performs the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S3 in any other appropriate manner (e.g., by means of firmware).

[0078] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0079] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0081] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0082] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An optimized positioning method based on visual labels, It is characterized in that The method includes: Preprocessing: Identify the visual labels set in the target area, extract candidate feature lines within the preset range of the visual labels, and assign semantic information to the candidate feature lines, where the semantic information includes a first attribute, a second attribute, and a third attribute; the first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number; Obtaining 3D Plücker coordinates: screening candidate feature lines whose first attribute is that they are visual label feature lines, screening all feature lines with the same visual label ID according to the second attribute, performing graph optimization processing on all the feature lines according to the third attribute, and obtaining 3D Plücker coordinates of all the feature lines in the visual label coordinate system; Line-to-surface constraint optimization processing: convert the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, perform line-to-surface constraint optimization processing on all the feature lines, and realize the positioning of the visual label.

2. According to the optimization positioning method based on visual tags according to claim 1, It is characterized in that The method for acquiring the first attribute is to use the LSD algorithm to judge the candidate feature line.

3. According to claim 2, an optimization positioning method based on visual tags, It is characterized in that The judgment method is: If the distances between the two endpoints of the candidate feature line and the two nearest corner points corresponding to the endpoints are both less than a set threshold, the first attribute of the candidate feature line is set as a visual label feature line; If the distance between the two endpoints of the candidate feature line and the two most adjacent corner points corresponding to the endpoints is greater than or equal to the set threshold, the first attribute is not set for the candidate feature line.

4. According to claim 1, an optimization positioning method based on visual tags, It is characterized in that The method for obtaining the second attribute is: based on the first attribute, the visual label ID is used as the second attribute of the candidate feature line. When multiple candidate feature lines with the first attribute are coplanar with the same visual label and can form a closed figure, the ID of the visual label is used as the second attribute of the multiple candidate feature lines.

5. According to claim 1, an optimization positioning method based on visual tags, It is characterized in that The characteristic lines are numbered to obtain a third attribute of the characteristic lines, wherein characteristic lines with the same second attribute have different numbers.

6. The method for optimizing positioning based on visual tags according to claim 1, It is characterized in that The steps of obtaining 3D Plücker coordinates also include: Add the binary edge of the feature line whose visual label and second attribute are the corresponding visual label ID to obtain the 3D feature line; The 3D feature line is transformed from the world coordinate system to the visual coordinate system to obtain the 3D Plücker coordinates in the visual coordinate system; According to the third attribute, 3D Plücker coordinates of a plurality of feature lines having the same second attribute are obtained.

7. The method for optimizing positioning based on visual tags according to claim 1, It is characterized in that In the line-to-surface constraint optimization process, the 3D Plücker coordinates are transformed from the visual coordinate system to the camera coordinate system according to the formula Carry out; among them L M is the coordinate in the camera coordinate system, is m in the Plücker coordinate system M The transpose of a vector on an axis, is d in the Plücker coordinate system M The transpose of a vector on an axis.

8. An optimized positioning device based on visual tags, It is characterized in that The equipment includes: The first module, preprocessing: identifying the visual label set in the target area, extracting the candidate feature lines within the preset range of the visual label, and assigning semantic information to the candidate feature lines, the semantic information including a first attribute, a second attribute and a third attribute; the first attribute is whether it is a visual label feature line, the second attribute is the visual label ID, and the third attribute is the feature line sequence number; The second module is used to obtain 3D Plücker coordinates, screen candidate feature lines whose first attribute is that they are visual label feature lines, screen all feature lines with the same visual label ID according to the second attribute, perform graph optimization processing on all the feature lines according to the third attribute, and obtain 3D Plücker coordinates of all the feature lines in the visual label coordinate system; The third module is line-to-surface constraint optimization processing, which converts the 3D Plücker coordinates of all the feature lines from the visual coordinate system to the camera coordinate system, performs line-to-surface constraint optimization processing on all the feature lines, and realizes the positioning of the visual labels.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory. It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images