Loopback detection method, computer readable storage medium and intelligent equipment

Abstract

The invention relates to the technical field of intelligent driving, in particular to a loopback detection method, a computer readable storage medium and intelligent equipment, and aims to solve the technical problem of how to realize more universal and reliable loopback detection so as to correct positioning deviation. In order to achieve the purpose, the method comprises the following steps: performing vector optimization according to continuous multi-frame environment perception results to obtain semantic vector information, constructing a local descriptor of the current frame environment perception result based on the semantic vector information, and obtaining a plurality of candidate loopback frames of the current frame according to the local descriptor of the current frame and a word bag library; and acquiring a loopback frame of the current frame from the plurality of candidate loopback frames according to the semantic vector information of the current frame. According to the method, the local descriptor has more environmental information, the features are more stable, the discrimination degree is higher, the candidate loopback frame can have higher accuracy, the more reliable loopback frame can be obtained, and therefore effective correction of the current frame is achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a loop detection method, a computer-readable storage medium, and an intelligent device. Background Art

[0002] With the vigorous development of autonomous driving technology, more and more intelligent driving functions have been developed, including automatic navigation requirements for various scenarios, such as parking lots, highway service areas, etc. High-precision maps usually only cover the main urban areas, and cannot cover the scenes in these parks, which puts higher requirements on the positioning function. For the positioning function, the odometer can guarantee relative accuracy, but as the error accumulation increases, the odometer must introduce loop detection, GNSS (Global Navigation Satellite System) signals and other external information to achieve correction. GNSS signals are usually not available in urban canyons, especially indoor scenes, so the accuracy of loop detection directly determines the accuracy of positioning.

[0003] Traditional loop detection methods are simple and easy to use. However, in some scenes with highly similar geometric structures and lack of texture, such as indoor parking lots, it is difficult to recall the correct loop based on traditional geometric feature / visual feature descriptor methods. Although deep learning-based methods can well extract scene features and achieve high-performance loop detection, a single model is often difficult to adapt to the loop detection requirements of different scenes and lacks generalization. In addition, most of the above methods perform loop detection based on single-frame observations, which are easily affected by sensor noise and environmental changes.

[0004] Accordingly, the art needs a new loop detection solution to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of how to achieve more universal and reliable loop detection to correct positioning.

[0006] In a first aspect, a loop detection method is provided, the method comprising:

[0007] Obtaining a local descriptor of the current frame environment perception result; wherein the local descriptor is a descriptor constructed based on semantic vector information obtained by vector optimization of multiple consecutive frames of environment perception results;

[0008] According to the local descriptor of the current frame and a preset word bag library, a plurality of candidate loop frames of the current frame are obtained; wherein the word bag library includes the local descriptors of the historical frames;

[0009] A loop frame of the current frame is obtained according to the candidate loop frame and the semantic vector information of the current frame.

[0010] In a technical solution of the above loop detection method, the method further includes obtaining the local descriptor according to the following steps:

[0011] Performing vector optimization according to the continuous multi-frame environment perception results to obtain a vector optimization result;

[0012] Acquire semantic vector information of the current frame according to multiple vector optimization results within a first preset range of the current frame;

[0013] Taking the center point of the semantic vector corresponding to each semantic vector information of the current frame as a key point, and associating each key point with the semantic vector information of a second preset range of the key point, to construct a two-dimensional horizontal descriptor with the key point as the center point;

[0014] Traverse all key points of the current frame and construct a vector triangle with every three key points to obtain the local descriptor; wherein the side length and semantic information of the vertices of the vector triangle are unique, the side length contains the relative position relationship between the semantic vectors within the second preset range, and the vertex contains the semantic vector information and the two-dimensional horizontal descriptor.

[0015] In a technical solution of the above loop detection method, performing vector optimization according to the continuous multi-frame environment perception results to obtain the vector optimization result includes:

[0016] Obtaining a semantic vector map according to the continuous multi-frame environment perception results and odometer results;

[0017] constructing a vector residual according to the semantic vector map;

[0018] Vector optimization is performed according to the vector residual to obtain the vector optimization result.

[0019] In a technical solution of the above loop detection method, the step of obtaining semantic vector information of the current frame according to multiple vector optimization results within a first preset range of the current frame includes:

[0020] Obtaining multiple vector optimization results within a first preset range of the current frame;

[0021] Filter out the vector optimization results whose cumulative driving distance is greater than a third preset range from the obtained multiple vector optimization results;

[0022] The filtered vector optimization result is used as the semantic vector information of the current frame.

[0023] In a technical solution of the above loop detection method, the obtaining of the local descriptor includes:

[0024] A vector triangle is constructed with every three key points, and after sorting the vertices of the vector triangle according to the side length of the vector triangle, the vector triangle with the sorted vertices is used as the local descriptor.

[0025] In a technical solution of the above loop detection method, the step of obtaining multiple candidate loop frames of the current frame according to the local descriptor of the current frame and a preset bag-of-words library includes:

[0026] Traversing all local descriptors of the current frame, and searching the local descriptors of the historical frames that are the same as the local descriptors of each current frame in the bag-of-words library according to the side length and vertex information of the local descriptors;

[0027] Verify the local descriptor of each retrieved historical frame to obtain a verified local descriptor;

[0028] According to the number of local descriptors of each historical frame that has been retrieved and verified, a plurality of candidate loop frames are obtained.

[0029] In a technical solution of the loop detection method, the local descriptor of each retrieved historical frame is verified to obtain a verified local descriptor, including:

[0030] Performing a timing check on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and obtaining the local descriptor of the historical frame with a timing difference as the local descriptor that passes the timing check;

[0031] Performing a direction check on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and obtaining the local descriptor of the historical frame whose difference in the vector direction of the vertex is within a preset first difference range as the local descriptor that passes the direction check;

[0032] A horizontal descriptor check is performed on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and the local descriptor of the historical frame with which the horizontal descriptor matches is obtained as the local descriptor that passes the horizontal descriptor check.

[0033] In a technical solution of the above loop detection method, acquiring the loop frame of the current frame according to the semantic vector information of the candidate loop frame and the current frame includes:

[0034] For each candidate loop frame, obtain the relative pose between each local descriptor of the candidate loop frame and the corresponding local descriptor of the current frame;

[0035] According to the relative position, obtaining the matching degree between the semantic vector information corresponding to the two local descriptors;

[0036] The loop frame of the current frame is obtained according to the candidate loop frame with the highest matching degree among all candidate loop frames.

[0037] In a technical solution of the above loop detection method, obtaining the loop frame of the current frame according to the candidate loop frame with the highest matching degree among all candidate loop frames includes:

[0038] Taking the relative pose of the candidate loop frame with the highest matching degree as an initial value, performing global vector matching on the candidate loop frame with the highest matching degree and the current frame to obtain a matched relative pose;

[0039] According to the matched relative posture, obtaining a matching score between the candidate loop frame with the highest matching degree and the current frame;

[0040] According to the matching score and a preset score threshold, a loop frame of the current frame is obtained.

[0041] In a technical solution of the above loop detection method, the method further includes:

[0042] According to the relative position of the loop frame, loop optimization is performed on the current frame.

[0043] In a technical solution of the above loop detection method, the method further includes:

[0044] For historical frames that need to be put into the word bag library, the historical frames are filtered, and the historical frames whose number of semantic vector information is less than a first preset number or whose feature distribution of semantic vector information conforms to a preset distribution rule are removed, and then the historical frames are put into the word bag library.

[0045] In a second aspect, a smart device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions of the above-mentioned loop detection method is implemented.

[0046] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned loop detection method.

[0047] Solution 1. A loop detection method, characterized in that the method comprises:

[0048] Obtaining a local descriptor of the current frame environment perception result; wherein the local descriptor is a descriptor constructed based on semantic vector information obtained by vector optimization of multiple consecutive frames of environment perception results;

[0049] According to the local descriptor of the current frame and a preset word bag library, a plurality of candidate loop frames of the current frame are obtained; wherein the word bag library includes the local descriptors of the historical frames;

[0050] A loop frame of the current frame is obtained according to the candidate loop frame and the semantic vector information of the current frame.

[0051] Solution 2. The loop detection method according to Solution 1 is characterized in that the method further comprises obtaining the local descriptor according to the following steps:

[0052] Performing vector optimization according to the continuous multi-frame environment perception results to obtain a vector optimization result;

[0053] Acquire semantic vector information of the current frame according to multiple vector optimization results within a first preset range of the current frame;

[0054] Taking the center point of the semantic vector corresponding to each semantic vector information of the current frame as a key point, and associating each key point with the semantic vector information of a second preset range of the key point, to construct a two-dimensional horizontal descriptor with the key point as the center point;

[0055] Traverse all key points of the current frame and construct a vector triangle with every three key points to obtain the local descriptor; wherein the side length and semantic information of the vertices of the vector triangle are unique, the side length contains the relative position relationship between the semantic vectors within the second preset range, and the vertex contains the semantic vector information and the two-dimensional horizontal descriptor.

[0056] Solution 3. The loop detection method according to Solution 2 is characterized in that:

[0057] The performing vector optimization according to the continuous multi-frame environment perception results to obtain the vector optimization result includes:

[0058] Obtaining a semantic vector map according to the continuous multi-frame environment perception results and odometer results;

[0059] constructing a vector residual according to the semantic vector map;

[0060] Vector optimization is performed according to the vector residual to obtain the vector optimization result.

[0061] Solution 4. The loop detection method according to Solution 2 is characterized in that:

[0062] The obtaining semantic vector information of the current frame according to the multiple vector optimization results within the first preset range of the current frame includes:

[0063] Obtaining multiple vector optimization results within a first preset range of the current frame;

[0064] Filter out the vector optimization results whose cumulative driving distance is greater than a third preset range from the obtained multiple vector optimization results;

[0065] The filtered vector optimization result is used as the semantic vector information of the current frame.

[0066] Solution 5. The loop detection method according to Solution 2 is characterized in that:

[0067] The obtaining of the local descriptor comprises:

[0068] A vector triangle is constructed with every three key points, and after sorting the vertices of the vector triangle according to the side length of the vector triangle, the vector triangle with the sorted vertices is used as the local descriptor.

[0069] Solution 6. The loop detection method according to Solution 2 is characterized in that:

[0070] The step of obtaining a plurality of candidate loop frames of the current frame according to the local descriptor of the current frame and a preset bag-of-words library includes:

[0071] Traversing all local descriptors of the current frame, and searching the local descriptors of the historical frames that are the same as the local descriptors of each current frame in the bag-of-words library according to the side length and vertex information of the local descriptors;

[0072] Verify the local descriptor of each retrieved historical frame to obtain a verified local descriptor;

[0073] According to the number of local descriptors of each historical frame that has been retrieved and verified, a plurality of candidate loop frames are obtained.

[0074] Solution 7. The loop detection method according to Solution 6 is characterized in that:

[0075] The step of verifying the local descriptor of each retrieved historical frame to obtain a verified local descriptor includes:

[0076] Performing a timing check on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and obtaining the local descriptor of the historical frame with a timing difference as the local descriptor that passes the timing check;

[0077] Performing a direction check on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and obtaining the local descriptor of the historical frame whose difference in the vector direction of the vertex is within a preset first difference range as the local descriptor that passes the direction check;

[0078] A horizontal descriptor check is performed on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and the local descriptor of the historical frame with which the horizontal descriptor matches is obtained as the local descriptor that passes the horizontal descriptor check.

[0079] Solution 8. The loop detection method according to Solution 1 is characterized in that:

[0080] The step of acquiring the loop frame of the current frame according to the semantic vector information of the candidate loop frame and the current frame includes:

[0081] For each candidate loop frame, obtain the relative pose between each local descriptor of the candidate loop frame and the corresponding local descriptor of the current frame;

[0082] According to the relative position, obtaining the matching degree between the semantic vector information corresponding to the two local descriptors;

[0083] The loop frame of the current frame is obtained according to the candidate loop frame with the highest matching degree among all candidate loop frames.

[0084] Solution 9. The loop detection method according to Solution 8 is characterized in that:

[0085] The step of obtaining a loop frame of the current frame according to the candidate loop frame with the highest matching degree among all the candidate loop frames includes:

[0086] Taking the relative pose of the candidate loop frame with the highest matching degree as an initial value, performing global vector matching on the candidate loop frame with the highest matching degree and the current frame to obtain a matched relative pose;

[0087] According to the matched relative posture, obtaining a matching score between the candidate loop frame with the highest matching degree and the current frame;

[0088] According to the matching score and a preset score threshold, a loop frame of the current frame is obtained.

[0089] Solution 10. The loop detection method according to Solution 8 or 9, characterized in that:

[0090] The method further comprises:

[0091] According to the relative position of the loop frame, loop optimization is performed on the current frame.

[0092] Solution 11. The loop detection method according to any one of Solutions 1 to 9, characterized in that the method further comprises:

[0093] For historical frames that need to be put into the word bag library, the historical frames are filtered, and the historical frames whose number of semantic vector information is less than a first preset number or whose feature distribution of semantic vector information conforms to a preset distribution rule are removed, and then the historical frames are put into the word bag library.

[0094] Solution 12. A smart device, comprising:

[0095] at least one processor;

[0096] and, a memory communicatively coupled to the at least one processor;

[0097] The memory stores a computer program, and when the computer program is executed by the at least one processor, the loop detection method described in any one of Schemes 1 to 11 is implemented.

[0098] Solution 13. A computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the loop detection method described in any one of Solutions 1 to 11.

[0099] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0100] In the implementation of the loop detection method technical solution provided by the present application, the present application obtains semantic vector information after vector optimization based on the continuous multi-frame environmental perception results, constructs a local descriptor of the current frame environmental perception results based on the semantic vector information, and obtains multiple candidate loop frames of the current frame based on the local descriptor and word bag library of the current frame, and obtains the loop frame of the current frame from multiple candidate loop frames based on the semantic vector information of the current frame. Through the above configuration, the present application can fuse the multi-frame environmental perception results to obtain the local descriptor, so that the local descriptor has more environmental information, the features are more stable and the discrimination is higher, and it can be more suitable for loop detection in scenes with highly similar geometric structures or lack of texture, and the detection process is not easily affected by sensor noise and environmental changes. At the same time, based on the local descriptor of the current frame and the local descriptor of the historical frame in the word bag library, the candidate loop frame is obtained, which can make the candidate loop frame have higher accuracy. Further, according to the semantic vector information of the current frame, the loop frame of the current frame is obtained from the candidate loop frame, and a more reliable loop frame can be obtained, thereby achieving effective deviation correction of the current frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0102] Figure 1 is a schematic flow chart of the main steps of a loop detection method according to an embodiment of the present application;

[0103] Figure 2 It is a schematic flow chart of the main steps of a loop detection method according to an implementation method of an embodiment of the present application;

[0104] Figure 3 is a schematic diagram of a local descriptor of a key frame according to an example of an embodiment of the present application;

[0105] Figure 4 is a result schematic diagram of a candidate loop frame according to an example of an embodiment of the present application;

[0106] Figure 5 is a schematic diagram of a result of loop optimization of a current frame according to a loop frame according to an example of an embodiment of the present application;

[0107] Figure 6 It is a schematic diagram of the main structure of a smart device according to an embodiment of the present application.

[0108] Reference numerals:

[0109] 11: memory; 12: processor. DETAILED DESCRIPTION

[0110] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0111] In the description of the present application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B or A and B. The singular terms "one" and "the" may also include plural forms.

[0112] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0113] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0114] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0115] Here we first explain some terms involved in this application.

[0116] Descriptors are quantitative data used to describe local structural features, which can fully reflect the shape, texture structure and other information near the feature points.

[0117] A semantic vector is a quantity that contains semantic information and has both magnitude and direction.

[0118] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of a loop detection method according to an embodiment of the present application. Figure 1 As shown, the loop detection method in the embodiment of the present application mainly includes the following steps S101 to S103.

[0119] Step S101: Obtain a local descriptor of the current frame environment perception result; wherein the local descriptor is a descriptor constructed based on semantic vector information obtained by vector optimization based on multiple consecutive frames of environment perception results.

[0120] In this embodiment, the local descriptor of the current frame environment perception result can be obtained based on the continuous multi-frame environment perception results. The environment perception result is the result obtained by the environment perception device to perceive the environment. The environment perception device can be a laser radar, a millimeter wave radar, a camera, etc., and the corresponding environment perception result can be a point cloud, an image, etc. The environment perception result can be obtained by one environment perception device or by multiple different environment perception devices.

[0121] In one implementation, the semantic vector information may include semantic information, coordinates and direction information of the vector, and the like.

[0122] Step S102: obtaining a plurality of candidate loop frames of the current frame according to the local descriptor of the current frame and a preset word bag library; wherein the word bag library includes the local descriptors of the historical frames.

[0123] In this embodiment, a plurality of candidate loop frames may be obtained from a preset bag-of-words library according to a local descriptor of the current frame.

[0124] Step S103: Obtain a loop frame of the current frame according to the semantic vector information of the candidate loop frames and the current frame.

[0125] In this embodiment, the loop frame of the current frame can be screened from the candidate loop frames according to the semantic vector information of the current frame.

[0126] Based on the method described in the above steps S101 to S103, the embodiment of the present application obtains semantic vector information after vector optimization based on the continuous multi-frame environmental perception results, constructs a local descriptor of the current frame environmental perception results based on the semantic vector information, and obtains multiple candidate loop frames of the current frame based on the local descriptor and word bag library of the current frame, and obtains the loop frame of the current frame from multiple candidate loop frames based on the semantic vector information of the current frame. Through the above configuration, the embodiment of the present application can fuse the multi-frame environmental perception results to obtain the local descriptor, so that the local descriptor has more environmental information, the features are more stable and the discrimination is higher, and it can be more adapted to the loop detection in the scene with highly similar geometric structure or lack of texture, and the detection process is not easily affected by sensor noise and environmental changes. At the same time, based on the local descriptor of the current frame and the local descriptor of the historical frame in the word bag library, the candidate loop frame is obtained, which can make the candidate loop frame have higher accuracy. Further, according to the semantic vector information of the current frame, the loop frame of the current frame is obtained from the candidate loop frame, and a more reliable loop frame can be obtained, thereby realizing effective deviation correction of the current frame.

[0127] The process of obtaining the local descriptor and step S102 and step S103 are further described below.

[0128] In one implementation of the embodiment of the present application, the local descriptor may be obtained according to the following steps S201 to S204.

[0129] Step S201: performing vector optimization according to multiple consecutive frames of environmental perception results to obtain vector optimization results.

[0130] In this embodiment, step S201 may further include the following steps S2011 to S2013:

[0131] Step S2011: Obtain a semantic vector map based on multiple consecutive frames of environment perception results and odometer results.

[0132] Step S2012: construct a vector residual according to the semantic vector map.

[0133] Step S2013: Perform vector optimization according to the vector residual to obtain a vector optimization result.

[0134] In this embodiment, based on continuous multi-frame environmental perception results and odometer results, the environmental perception results in the local coordinate system can be mapped to the odometer coordinate system for semantic vector mapping to obtain a semantic vector map; and the vector residual of the semantic vector map is constructed, and vector optimization is performed based on the vector residual to obtain a vector optimization result.

[0135] In one implementation, a point-to-point residual in a semantic vector map can be constructed, and a heuristic residual can be further constructed based on a vector geometry model to obtain a vector residual. The vector residual is nonlinearly optimized to minimize the residual to achieve vector optimization and obtain a vector optimization result. In one example, the vector converges when it is observed 5 times by a key frame and optimized more than 5 times by the vector. Among them, 5 times is an empirical value in practice, and those skilled in the art can also adjust it according to the needs of actual applications.

[0136] Step S202: acquiring semantic vector information of the current frame according to a plurality of vector optimization results within a first preset range of the current frame.

[0137] In this embodiment, step S202 may further include the following steps S2021 to S2023:

[0138] Step S2021: Obtain multiple vector optimization results within a first preset range of the current frame.

[0139] Step S2022: filtering out the vector optimization results whose cumulative driving distance is greater than a third preset range from the obtained multiple vector optimization results.

[0140] Step S2023: The filtered vector optimization result is used as the semantic vector information of the current frame.

[0141] In this embodiment, multiple vector optimization results within the first preset range of the current frame can be obtained. That is, after the vector optimization converges, the semantic vector information is extracted and assigned to the current frame. The vector optimization results with a cumulative driving distance greater than the third preset range are filtered out to avoid returning to the position corresponding to the vector optimization result. In the case where the vector optimization fails due to positioning error, the filtered vector optimization results are used as the semantic vector information of the current frame. Among them, the current frame can be a frame that needs to be looped, or it can be a historical frame that needs to be put into the bag of words library. For the semantic vector map, the current frame is the key frame of the semantic vector map. The key frame of the semantic vector map can be extracted along the odometer trajectory.

[0142] In a specific example, the first preset range is within 20 meters around the center of the current frame pose. The third preset range is 80 meters. Those skilled in the art can also set the values ​​of the first preset range and the third preset range according to the needs of actual applications.

[0143] In one embodiment, for key frames (i.e., historical frames) that need to be put into the word bag library, the discrimination of the semantic vector information of the key frames can be checked, and the historical frames whose number of semantic vector information is less than the first preset number or whose feature distribution of semantic vector information conforms to the preset distribution rule are removed, and then the historical frames are put into the word bag library. That is, the key frames with small number of semantic vector information and highly regular feature distribution are filtered out.

[0144] Step S203: taking the center point of the semantic vector corresponding to each semantic vector information of the current frame as the key point, and associating each key point with the semantic vector information of a second preset range of the key point, to construct a two-dimensional horizontal descriptor with the key point as the center point.

[0145] In this embodiment, the center point of the semantic vector corresponding to the semantic vector information can be used as a key point, and each key point is associated with the semantic information, vector direction information, etc. in the semantic vector information. Each key point is associated with the semantic vector information within the surrounding second preset range to obtain a two-dimensional horizontal descriptor with the key point as the center point, which is used to express the features near the semantic vector information.

[0146] In a specific example, a circle may be drawn with the key point as the center and 5 meters as the radius to determine the second preset range. Those skilled in the art may also set the second preset range according to the needs of actual applications.

[0147] Step S204: traverse all key points of the current frame, and construct a vector triangle with every three key points to obtain a local descriptor; wherein the side length and semantic information of the vertex of the vector triangle are unique, the side length contains the relative position relationship between the semantic vectors within a second preset range, and the vertex contains the semantic vector information and a two-dimensional horizontal descriptor.

[0148] In this embodiment, all key points of the current frame can be traversed, and for each key point, the key points nearby are polled, and a vector triangle is attempted to be constructed with every three key points. The vector triangle sorts the vertices according to the size of the side length to prepare for the subsequent semantic vector information registration and posture solution. Each vector triangle needs to ensure the uniqueness of the semantic information of the side length and vertex. The side length stores the relative position relationship between the semantic vectors, and the vertex stores the semantic information, vector direction information, and two-dimensional horizontal descriptor of the semantic vector information.

[0149] Each vector triangle is used as a local descriptor of the current frame. Each frame will contain multiple different local descriptors. For each current frame, its local descriptor can be added to the bag-of-words library after loop closure detection is completed. Figure 3 , Figure 3 is a schematic diagram of a local descriptor of a key frame according to an example of an embodiment of the present application. Figure 3As shown in the figure, each key frame can include multiple local descriptors, each local descriptor is a vector triangle, and each vertex of the local descriptor contains semantic information, vector direction, and high-order horizontal descriptor information that integrates the surrounding environment, making the local descriptor have higher discrimination. At the same time, because the local descriptor is a vector triangle, it has translation invariance and rotation invariance when indexing semantic information through side length and vertex.

[0150] In one implementation of the embodiment of the present application, step S102 may further include the following steps S1021 to S1023:

[0151] Step S1021: traverse all local descriptors of the current frame, and retrieve local descriptors of historical frames that are the same as the local descriptors of each current frame in the word bag library according to the side length and vertex information of the local descriptors.

[0152] In this implementation, all local descriptors of the current frame may be traversed, and the same local descriptor in the bag-of-words database may be retrieved according to the semantic information of the edge length and vertex of the local descriptor.

[0153] Step S1022: verify the local descriptors of each retrieved historical frame to obtain local descriptors that pass the verification.

[0154] In this embodiment, step S1022 may further include the following steps S10221 to S10223:

[0155] Step S10221: Perform a timing check on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and obtain the local descriptor of the historical frame with a timing difference as the local descriptor that passes the timing check.

[0156] In this embodiment, it is necessary to perform a timing check on the local descriptors of the retrieved historical frames to ensure that the local descriptors of the current frame and the local descriptors of the historical frames are different in timing, so as to avoid the loop detection from obtaining frames that are close in timing as loop frames.

[0157] Step S10222: perform direction check on the local descriptor of the current frame and the corresponding local descriptor of the retrieved historical frame, and obtain the local descriptor of the historical frame whose vertex vector direction difference is within a preset first difference range as the local descriptor that passes the direction check.

[0158] In this embodiment, the local descriptor of the current frame and the local descriptor of the retrieved historical frame can be directionally checked to ensure that the difference in the vector direction of the vertices between the two is within a preset first difference range. Specifically, since the vertices of the vector triangle corresponding to the local descriptor are ordered, the vertices corresponding to the vector triangles corresponding to the two local descriptors can be connected to obtain three edges, and the cosine values ​​are calculated two by two. If the cosine values ​​are within the preset first difference range, it is considered that the vector directions of the vertices of the two local descriptors are consistent, and the direction check is passed.

[0159] In one embodiment, the preset first difference range may be 30 degrees. Those skilled in the art may also set the preset first difference range according to the needs of actual applications.

[0160] Step S10223: Perform horizontal descriptor verification on the local descriptor of the current frame and the corresponding local descriptor of the retrieved historical frame, and obtain the local descriptor of the historical frame that matches the horizontal descriptor as the local descriptor that passes the horizontal descriptor verification.

[0161] In this embodiment, the local descriptor of the current frame and the local description of the historical frame can be checked for horizontal descriptors to ensure that the horizontal descriptors associated with the vertices of the two local descriptors match each other. Specifically, the similarity between the corresponding horizontal descriptors can be calculated to determine whether the horizontal descriptors match each other.

[0162] In one implementation, the Hamming distance of the feature matrix of the horizontal descriptor can be calculated, and the calculated Hamming distance can be compared with the distance threshold. If the Hamming distance is less than the distance threshold, it is considered that the horizontal descriptors match each other. Those skilled in the art can set the distance threshold according to the needs of the actual application. The smaller the distance threshold, the stricter the verification.

[0163] Step S1023: Acquire multiple candidate loop frames according to the number of local descriptors of each historical frame that has been retrieved and verified.

[0164] In this embodiment, the historical frames in the word bag library can be voted according to the number of local descriptors of each historical frame that has been retrieved and verified. That is, the number of local descriptors of the historical frames in the word bag library that match the current frame is counted, and if there is a matching local descriptor, it gets one vote. The first n frames with the highest votes in the word bag library are used as candidate loop frames. Figure 4 , Figure 4 FIG. 1 is a result diagram of a candidate loop frame according to an example of an embodiment of the present application. Figure 4 As shown, the horizontal and vertical axes are both frame indexes. Figure 4It shows the similarity between the frames on the horizontal and vertical axes. The greater the heat, the more loop frames. It can be seen that the heat on the diagonal is very high, because the key frames are very close to the semantic vector features of the frames before and after them. Figure 4 It can also be seen that the thermal distribution is concentrated and not dispersed, which also shows that the local descriptor has a high degree of discrimination.

[0165] In one implementation of the embodiment of the present application, step S103 may further include the following steps S1031 to S1033:

[0166] Step S1031: For each candidate loop frame, obtain the relative pose between each local descriptor of the candidate loop frame and the corresponding local descriptor of the current frame.

[0167] In this embodiment, the local descriptors retrieved and verified in the bag-of-words library correspond one-to-one to the local descriptors of the current frame, and can be aligned based on the vector triangles of the local descriptors. During the alignment process, the vector triangles are decentralized, and singular value decomposition is performed to calculate the relative rotation of the two vector triangles. The relative translation is inferred based on the relative rotation, thereby obtaining the relative pose between the local descriptor of the candidate loop frame and the local descriptor corresponding to the current frame.

[0168] Step S1032: According to the relative posture, obtain the matching degree between the semantic vector information corresponding to the two local descriptors.

[0169] In this embodiment, the matching degree between two local descriptors can be obtained according to the relative posture.

[0170] Step S1033: Obtain a loop frame of the current frame according to the candidate loop frame with the highest matching degree among all the candidate loop frames.

[0171] In this implementation, step S1033 may further include the following steps S10331 to S10333:

[0172] Step S10331: Taking the relative pose of the candidate loop frame with the highest matching degree as the initial value, global vector matching is performed on the candidate loop frame with the highest matching degree and the current frame to obtain the matched relative pose.

[0173] In this embodiment, the relative pose of the candidate loop frame with the highest matching degree is used as the initial value, and global vector matching is performed on the candidate loop frame with the highest matching degree and the current frame to obtain the matched relative pose.

[0174] In one implementation, an iterative closest point registration algorithm may be used to perform global vector matching.

[0175] Step S10332: According to the matched relative posture, obtain the matching score between the candidate loop frame with the highest matching degree and the current frame.

[0176] In this embodiment, the degree of vector feature matching between the candidate loop frame with the highest matching degree and the current frame can be verified based on the matched relative posture, that is, the relative posture is transformed through the matched relative posture, and the intersection-and-union ratio of the transformed vector features is calculated as the matching score between the candidate loop frame with the highest matching degree and the current frame.

[0177] Step S10333: Obtain a loop frame of the current frame according to the matching score and a preset score threshold.

[0178] In this embodiment, the matching score can be compared with a preset score threshold. If the matching score is greater than the score threshold, the candidate loop frame with the highest matching degree is determined to be the loop frame of the current frame. The current frame can be loop optimized based on the loop frame and the corresponding relative pose. Figure 5 , Figure 5 FIG. 1 is a schematic diagram of the result of loop optimization of the current frame according to a loop frame according to an example of an embodiment of the present application. Figure 5 As shown in the figure, different shapes represent vector features of different functions. Label 1 represents the vector features of the current frame, label 2 represents the vector features of the loop frame, label 3 represents the position of the current frame, and label 4 represents the position of the loop frame. Figure 5 It can be seen that the current frame and the loop closure frame are basically aligned, which can provide the initial pose for loop closure optimization.

[0179] In one embodiment, please refer to the attached Figure 2 , Figure 2 FIG. 1 is a flow chart of the main steps of a loop detection method according to an implementation of an embodiment of the present application. Figure 2 As shown, the loop detection method may include the following steps S301 to S308:

[0180] Step S301: semantic mapping.

[0181] In this embodiment, the method described in step S301 is similar to the aforementioned step S2011, and for the sake of simplicity, it will not be repeated here.

[0182] Step S302: extracting key frames.

[0183] In this implementation, after the semantic vector map is acquired, key frames in the semantic vector map may be extracted.

[0184] Step S303: extract key points.

[0185] In this embodiment, the method described in step S303 is similar to the aforementioned step S203, and for the sake of simplicity, it will not be repeated here.

[0186] Step S304: constructing local descriptors.

[0187] In this embodiment, the method described in step S304 is similar to the aforementioned step S204, and for the sake of simplicity, it is not repeated here.

[0188] Step S305: Extract candidate loop frames.

[0189] In this embodiment, the method described in step S305 is similar to the aforementioned step S102, and for the sake of simplicity, it will not be repeated here.

[0190] Step S306: Coarse alignment of vector features.

[0191] In this embodiment, the method described in step S306 is similar to the aforementioned step S1031, and for the sake of simplicity, it will not be repeated here.

[0192] Step S307: Refine the relative pose.

[0193] In this embodiment, the method described in step S307 is similar to the aforementioned step S1032, and for the sake of simplicity, it will not be repeated here.

[0194] Step S308: loop check.

[0195] In this embodiment, the method described in step S308 is similar to the aforementioned step S1033, and for the sake of simplicity, it will not be repeated here.

[0196] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

[0197] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0198] Another aspect of the present application also provides a computer-readable storage medium.

[0199] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium may be configured to store a program for executing the loop detection method of the above method embodiment, and the program may be loaded and run by a processor to implement the above loop detection method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium may be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0200] Another aspect of the present application also provides a smart device.

[0201] In an embodiment of an intelligent device according to the present application, the intelligent device may include at least one processor; and a memory connected to the at least one processor in communication; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. The intelligent device described in the present application may include a driving device, a smart car, a robot, and the like. Figure 6 , Figure 6 FIG. 4 exemplarily shows that the memory 11 and the processor 12 are communicatively connected via a bus.

[0202] In some embodiments of the present application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in the present application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive itself or assist in driving. The processor communicates with the sensor and / or the autonomous driving system to complete the method described in any of the above embodiments.

[0203] So far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A loop detection method, characterized in that: The method comprises: Obtaining a local descriptor of the current frame environment perception result; wherein the local descriptor is a descriptor constructed based on semantic vector information obtained by vector optimization of multiple consecutive frames of environment perception results; According to the local descriptor of the current frame and a preset word bag library, a plurality of candidate loop frames of the current frame are obtained; wherein the word bag library includes the local descriptors of the historical frames; A loop frame of the current frame is obtained according to the candidate loop frame and the semantic vector information of the current frame.

2. The loop detection method according to claim 1, characterized in that: The method further comprises obtaining the local descriptor according to the following steps: Performing vector optimization according to the continuous multi-frame environment perception results to obtain a vector optimization result; Acquire semantic vector information of the current frame according to multiple vector optimization results within a first preset range of the current frame; Taking the center point of the semantic vector corresponding to each semantic vector information of the current frame as a key point, and associating each key point with the semantic vector information of a second preset range of the key point, to construct a two-dimensional horizontal descriptor with the key point as the center point; Traverse all key points of the current frame and construct a vector triangle with every three key points to obtain the local descriptor; wherein the side length and semantic information of the vertices of the vector triangle are unique, the side length contains the relative position relationship between the semantic vectors within the second preset range, and the vertex contains the semantic vector information and the two-dimensional horizontal descriptor.

3. The loop detection method according to claim 2, characterized in that: The performing vector optimization according to the continuous multi-frame environment perception results to obtain the vector optimization result includes: Obtaining a semantic vector map according to the continuous multi-frame environment perception results and odometer results; constructing a vector residual according to the semantic vector map; Vector optimization is performed according to the vector residual to obtain the vector optimization result.

4. The loop detection method according to claim 2, characterized in that: The obtaining semantic vector information of the current frame according to the multiple vector optimization results within the first preset range of the current frame includes: Obtaining multiple vector optimization results within a first preset range of the current frame; Filter out the vector optimization results whose cumulative driving distance is greater than a third preset range from the obtained multiple vector optimization results; The filtered vector optimization result is used as the semantic vector information of the current frame.

5. The loop detection method according to claim 2, characterized in that: The obtaining of the local descriptor comprises: A vector triangle is constructed with every three key points, and after sorting the vertices of the vector triangle according to the side length of the vector triangle, the vector triangle with the sorted vertices is used as the local descriptor.

6. The loop detection method according to claim 2, characterized in that: The step of obtaining a plurality of candidate loop frames of the current frame according to the local descriptor of the current frame and a preset bag-of-words library includes: Traversing all local descriptors of the current frame, and searching the local descriptors of the historical frames that are the same as the local descriptors of each current frame in the bag-of-words library according to the side length and vertex information of the local descriptors; Verify the local descriptor of each retrieved historical frame to obtain a verified local descriptor; According to the number of local descriptors of each historical frame that has been retrieved and verified, a plurality of candidate loop frames are obtained.

7. The loop detection method according to claim 6, characterized in that: The step of verifying the local descriptor of each retrieved historical frame to obtain a verified local descriptor includes: Performing a timing check on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and obtaining the local descriptor of the historical frame with a timing difference as the local descriptor that passes the timing check; Performing a direction check on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and obtaining the local descriptor of the historical frame whose difference in the vector direction of the vertex is within a preset first difference range as the local descriptor that passes the direction check; A horizontal descriptor check is performed on the local descriptor of the current frame and the local descriptor of the corresponding retrieved historical frame, and the local descriptor of the historical frame with which the horizontal descriptor matches is obtained as the local descriptor that passes the horizontal descriptor check.

8. The loop detection method according to claim 1, characterized in that: The step of acquiring the loop frame of the current frame according to the semantic vector information of the candidate loop frame and the current frame includes: For each candidate loop frame, obtain the relative pose between each local descriptor of the candidate loop frame and the corresponding local descriptor of the current frame; According to the relative position, obtaining the matching degree between the semantic vector information corresponding to the two local descriptors; The loop frame of the current frame is obtained according to the candidate loop frame with the highest matching degree among all candidate loop frames.

9. The loop detection method according to claim 8, characterized in that: The step of obtaining a loop frame of the current frame according to the candidate loop frame with the highest matching degree among all the candidate loop frames includes: Taking the relative pose of the candidate loop frame with the highest matching degree as an initial value, performing global vector matching on the candidate loop frame with the highest matching degree and the current frame to obtain a matched relative pose; According to the matched relative posture, obtaining a matching score between the candidate loop frame with the highest matching degree and the current frame; According to the matching score and a preset score threshold, a loop frame of the current frame is obtained.

10. The loop detection method according to claim 8 or 9, characterized in that: The method further comprises: According to the relative position of the loop frame, loop optimization is performed on the current frame.

Images