A line segment matching method, device, computer equipment and storage medium

Abstract

The present disclosure provides a line segment matching method and device, computer equipment and a storage medium, wherein the method comprises: obtaining a first image and a second image; the first image and the second image are images in the same environment; extracting a plurality of first point features in the first image, and determining a first line segment to which each first point feature belongs, and extracting a plurality of second point features in the second image, and determining a second line segment to which each second point feature belongs; matching the first point features in the first image with the second point features in the second image to obtain a matching result; based on the matching result, determining the matching degree of any first line segment and any second line segment, and taking the first line segment and the second line segment whose matching degree meets a preset condition as mutually matched line segments. The present disclosure uses point features to represent line segments, which improves the accuracy of line segment matching results when the length and end points of the line segments change.

Description

Technical Field

[0001] This disclosure relates to the fields of vision and computer technology, and more specifically, to a line segment matching method, apparatus, computer device, and storage medium. Background Technology

[0002] Line segment features play an important role in visual simultaneous localization and mapping (SLAM), providing accurate constraints during pose calculation.

[0003] Current line segment feature matching methods are mostly based on feature description information extracted from the image region or the endpoints of the line segment. However, due to the uncertainty of the endpoints and length of the line segment detection, such as occlusion at the endpoints or in the middle of the line segment, or the line segment being too long and spanning two frames of the image, the accuracy of the matching results will drop significantly. In addition, when integrating line segment features into visual SLAM, the matching of the extracted line segment feature description information and the matching of the extracted point features are independent of each other, which increases the amount of computation and the processing time, and reduces the efficiency of line segment matching. Summary of the Invention

[0004] This disclosure provides at least one line segment matching method, apparatus, computer device, and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide a line segment matching method, including:

[0006] Acquire a first image and a second image; the first image and the second image are images from the same environment;

[0007] Extract multiple first point features from the first image and determine the first line segment to which each first point feature belongs; and extract multiple second point features from the second image and determine the second line segment to which each second point feature belongs.

[0008] The first feature in the first image is matched with the second feature in the second image to obtain the matching result;

[0009] Based on the matching results, the degree of matching between any first line segment and any second line segment is determined, and the first line segment and the second line segment whose degree of matching meets the preset conditions are regarded as mutually matched line segments.

[0010] In one optional implementation, determining the degree of matching between any first line segment and any second line segment based on the matching result includes:

[0011] Based on the matching results, target point features that match the second point features included in any second line segment are selected from the first point features included in any first line segment.

[0012] The degree of matching between the arbitrary first line segment and the arbitrary second line segment is determined based on the first number of target point features, the second number of first point features included in the arbitrary first line segment, and the third number of second point features included in the arbitrary second line segment.

[0013] In one optional implementation, determining the matching degree between the arbitrary first line segment and the arbitrary second line segment based on a first number of features of the target point, a second number of first point features included in the arbitrary first line segment, and a third number of second point features included in the arbitrary second line segment includes:

[0014] Determine a first ratio between the first quantity and the second quantity, and determine a second ratio between the first quantity and the third quantity;

[0015] Based on the first ratio and the second ratio, the degree of matching between the arbitrary first line segment and the arbitrary second line segment is determined.

[0016] In one optional implementation, determining the matching degree between the arbitrary first line segment and the arbitrary second line segment based on the first ratio and the second ratio includes:

[0017] Based on the first ratio and the second ratio, a target ratio is determined, and the target ratio is used as the degree of matching.

[0018] The step of designating the first line segment and the second line segment whose matching degree meets the preset conditions as mutually matched line segments includes:

[0019] If the target ratio is greater than or equal to a first preset threshold, the first line segment and the second line segment are determined to be mutually matched line segments.

[0020] In an optional implementation, after extracting multiple first point features from the first image and determining the first line segment to which each first point feature belongs, and extracting multiple second point features from the second image and determining the second line segment to which each first point feature belongs, the method further includes:

[0021] Determine a first set of first point features included in any first line segment, and determine a second set of second point features included in any second line segment;

[0022] The step of filtering target point features from the first point features included in any first line segment that match the second point features included in any second line segment, based on the matching result, includes:

[0023] Based on the matching results, the first set, and the second set, target point features that match the second point features included in any second line segment are selected from the first point features included in any first line segment.

[0024] In an optional implementation, before matching the first point feature in the first image with the second point feature in the second image to obtain a matching result, the method further includes:

[0025] Determine the number of first point features in the first line segment; if the number of first point features is less than or equal to a second preset threshold, remove the first line segment and the first point features within the first line segment; and...

[0026] Determine the number of second point features in the second line segment. If the number of second point features is less than or equal to a third preset threshold, remove the second line segment and the second point features in the second line segment.

[0027] In an optional implementation, after extracting multiple first point features from the first image and multiple second point features from the second image, the method further includes:

[0028] Determine the first feature description information corresponding to the multiple first point features and the second feature description information corresponding to the multiple second point features;

[0029] The step of matching the first feature point in the first image with the second feature point in the second image to obtain a matching result includes:

[0030] Based on the first feature description information and the second feature description information, the first point feature and the second point feature are matched to obtain the matching result.

[0031] In one optional implementation, determining the first line segment to which each first point feature belongs includes:

[0032] For each of the plurality of first point features, determine the distance from the first target point corresponding to the first point feature to any first line segment; and take the first point feature corresponding to the first target point whose distance is less than or equal to a fourth preset threshold as the point feature belonging to the arbitrary first line segment;

[0033] Determining the second line segment to which each second point feature belongs includes:

[0034] For each of the plurality of second point features, determine the distance from the second target point corresponding to the second point feature to any second line segment; and take the second point feature corresponding to the second target point whose distance is less than or equal to a fifth preset threshold as the point feature belonging to the arbitrary second line segment.

[0035] Secondly, embodiments of this disclosure also provide a line segment matching device, comprising:

[0036] The acquisition module is used to acquire a first image and a second image; the first image and the second image are images from the same environment;

[0037] The determination module is used to extract multiple first point features in the first image and determine the first line segment to which each first point feature belongs, and to extract multiple second point features in the second image and determine the second line segment to which each second point feature belongs;

[0038] The first matching module is used to match the first point feature in the first image with the second point feature in the second image to obtain a matching result;

[0039] The second matching module is used to determine the degree of matching between any first line segment and any second line segment based on the matching result, and to take the first line segment and the second line segment whose degree of matching meets the preset conditions as mutually matched line segments.

[0040] In one optional implementation, the second matching module is configured to, based on the matching result, filter target point features from the first point features included in the arbitrary first line segment that match the second point features included in the arbitrary second line segment; and determine the degree of matching between the arbitrary first line segment and the arbitrary second line segment based on the first number of target point features, the second number of first point features included in the arbitrary first line segment, and the third number of second point features included in the arbitrary second line segment.

[0041] In one optional implementation, the second matching module is configured to determine a first ratio between the first quantity and the second quantity, and to determine a second ratio between the first quantity and the third quantity; and to determine the degree of matching between the arbitrary first line segment and the arbitrary second line segment based on the first ratio and the second ratio.

[0042] In one optional implementation, the second matching module is configured to determine a target ratio based on the first ratio and the second ratio, and use the target ratio as the degree of matching; if the target ratio is greater than or equal to a first preset threshold, the first line segment and the second line segment are determined to be mutually matched line segments.

[0043] In an optional implementation, the determining module is further configured to, after extracting multiple first point features in the first image and determining the first line segment to which each first point feature belongs, and after extracting multiple second point features in the second image and determining the second line segment to which each first point feature belongs, determine a first set of first point features included in any first line segment, and determine a second set of second point features included in any second line segment.

[0044] The second matching module is used to filter target point features that match the second point features included in any second line segment from the first point features included in any first line segment, based on the matching result, the first set, and the second set.

[0045] In one optional embodiment, the line segment matching device further includes a filtering module;

[0046] The filtering module is configured to, before matching the first point feature in the first image with the second point feature in the second image to obtain a matching result, determine the number of first point features in the first line segment, and if the number of first point features is less than or equal to a second preset threshold, remove the first line segment and the first point features in the first line segment; and determine the number of second point features in the second line segment, and if the number of second point features is less than or equal to a third preset threshold, remove the second line segment and the second point features in the second line segment.

[0047] In an optional implementation, the determining module is further configured to, after extracting multiple first point features in the first image and multiple second point features in the second image, determine first feature description information corresponding to the multiple first point features and second feature description information corresponding to the multiple second point features;

[0048] The first matching module is used to match the first point feature and the second point feature based on the first feature description information and the second feature description information to obtain the matching result.

[0049] In one optional implementation, the determining module is configured to: determine the distance from a first target point corresponding to a first point feature to any first line segment for each of the plurality of first point features; designate the first point feature corresponding to the first target point whose distance is less than or equal to a fourth preset threshold as a point feature belonging to the arbitrary first line segment; and determine the distance from a second target point corresponding to a second point feature to any second line segment for each of the plurality of second point features; designate the second point feature corresponding to the second target point whose distance is less than or equal to a fifth preset threshold as a point feature belonging to the arbitrary second line segment.

[0050] Thirdly, embodiments of this disclosure also provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible line segment matching method in the first aspect, are performed.

[0051] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible line segment matching method described above.

[0052] For a description of the effects of the aforementioned line segment matching device, computer equipment, and storage medium, please refer to the description of the aforementioned line segment matching method; it will not be repeated here.

[0053] This disclosure provides a line segment matching method, apparatus, computer device, and storage medium. The method involves acquiring a first image and a second image, both located in the same environment. Multiple first point features are extracted from the first image, and a first line segment to which each first point feature belongs is determined. Similarly, multiple second point features are extracted from the second image, and a second line segment to which each second point feature belongs is determined. The first point features in the first image are matched with the second point features in the second image to obtain a matching result. Based on the matching result, the matching degree between any first line segment and any second line segment is determined. First and second line segments whose matching degree meets preset conditions are considered mutually matched line segments. This method utilizes point features to represent line segments, i.e., the first line segment to which each first point feature belongs and the second line segment to which each second point feature belongs. This improves the accuracy of line segment matching results when the length and endpoints of the line segment change. For example, it can match an entire line segment using only a portion of its features. In addition, the matching of the above line segments is based on the matching results of the first point feature and the second point feature. Therefore, there is no need to extract the feature description information of the line segments, which simplifies the extraction process of the feature description information of the line segments, saves the amount of computation and the computation time, and further improves the efficiency of line segment matching.

[0054] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0056] Figure 1 A flowchart of a line segment matching method provided by an embodiment of this disclosure is shown;

[0057] Figure 2 A schematic diagram of an algorithm flow provided by an embodiment of this disclosure is shown;

[0058] Figure 3 A schematic diagram of a line segment matching device provided in an embodiment of this disclosure is shown;

[0059] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0061] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0062] In this article, "multiple or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0063] Research has shown that line segment features play a crucial role in SLAM, providing accurate constraints during pose calculation. Current line segment feature matching methods are mostly based on feature descriptions extracted from image regions or line segment endpoints. However, uncertainties in line segment detection endpoints and lengths—such as occlusion at endpoints or in the middle, or excessively long lines spanning two frames—significantly reduce matching accuracy. Furthermore, when integrating line segment features into visual SLAM, matching the extracted line segment feature descriptions and matching the extracted point features are independent, increasing computational load and time, and reducing line segment matching efficiency.

[0064] Based on the above research, this disclosure provides a line segment matching method, apparatus, computer device, and storage medium. It utilizes point features to represent line segments, specifically a first line segment to which each first point feature belongs and a second line segment to which each second point feature belongs. This improves the accuracy of line segment matching results when the length and endpoints of the line segment change. For example, it can match an entire line segment using only a portion of its features. Furthermore, since the line segment matching is based on the matching results of the first and second point features, there is no need to extract the feature description information of the line segment, simplifying the extraction process, saving computational load and time, and further improving the efficiency of line segment matching.

[0065] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0066] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0067] To facilitate understanding of the embodiments of this disclosure, the terms used in the embodiments of this disclosure will be described in detail below:

[0068] 1. Superpoint is a point feature detection and descriptor (feature description information) extraction algorithm based on self-supervised training.

[0069] 2. Line Segment Detector (LSD) algorithm: First, calculate the gradient magnitude and direction of all points in the image. Then, treat adjacent points with small gradient direction changes as a connected region. Next, determine whether each region needs to be broken according to rules to form multiple regions with larger rectangles based on the rectangularity of each region. Finally, improve and filter all generated regions, and retain the regions that meet the conditions, which is the final line detection result.

[0070] 3. OpenCV is a cross-platform computer vision and machine learning software library released under the BSD license (open source), which can run on Linux, Windows, Android and Mac OS operating systems.

[0071] 4. SuperGlue, a feature matching algorithm based on graph convolutional neural networks, is used for point feature matching in this embodiment of the disclosure.

[0072] 5. The Bag-of-Words model is a simplified representation model used in Natural Language Processing and Information Retrieval (IR).

[0073] To facilitate understanding of this embodiment, the application scenario of the line segment matching method disclosed in this embodiment will be introduced first. The line segment matching method provided by this embodiment can be applied to visual SLAM. In some weak texture areas, line segments can be used as a front-end feature to supplement the shortcomings of insufficient target points, improve the robustness of visual SLAM front-end tracking. At the same time, the detection of line segments is more accurate than the detection of target points, and can provide more accurate constraints when performing pose calculation.

[0074] To facilitate understanding of this embodiment, a line segment matching method disclosed in this disclosure will first be described in detail. The execution subject of the line segment matching method provided in this disclosure is generally a computer device with certain computing capabilities. In some possible implementations, the line segment matching method can be implemented by a processor calling computer-readable instructions stored in memory.

[0075] The line segment matching method provided in this disclosure is illustrated below using a computer device as an example.

[0076] Based on the above application scenarios, this disclosure provides a line segment matching method. See also... Figure 1 The diagram shows a flowchart of a line segment matching method provided in an embodiment of this disclosure. The method includes steps S101 to S104, wherein:

[0077] S101: Acquire the first image and the second image; the first image and the second image are images from the same environment.

[0078] In this step, the first image and the second image can be images of the same environment acquired using a photographic device, such as a camera. Here, the environment can be the robot's operating environment, such as the operating environment when the robot is creating a map.

[0079] For example, in a map creation application scenario, the first image can be an image of the current frame or the current location acquired by the robot; the second image can be an image of the second frame (excluding the current frame) or other frames (excluding the current frame and the second frame) acquired by the robot, or an image of other locations (excluding the current location) in the robot's operating environment acquired by the robot. Two consecutive frames include target points with the same point features.

[0080] Continuing the previous example, in the application scenario of relocalization after map creation, the first image can be the image obtained during the map creation process in the above embodiments (including the first image and the second image). The second image can be the image obtained by the robot during the relocalization process in the same operating environment. The first image of location A collected during the map creation process and the second image of location A collected during the relocalization process include target points with the same point features.

[0081] S102: Extract multiple first point features from the first image and determine the first line segment to which each first point feature belongs; and extract multiple second point features from the second image and determine the second line segment to which each second point feature belongs.

[0082] In this step, the first feature is the feature corresponding to the first target point in the first image. The second feature is the feature corresponding to the second target point in the second image. The first target point is a target point with obvious features in the first image, and the second target point is a target point with obvious features in the second image. The first image includes multiple first target points, and the second image includes multiple second target points. Therefore, the first image includes multiple first feature points, and the second image includes multiple second feature points.

[0083] For target points with distinct features, such as the first image captured of a robot in a warehouse, including shelves, corners, and workstations, the first target point can include points on the shelves, corners, or workstations. The first point feature can include features of points on the shelves, corners, or workstations. The second point feature can also include features of prominent warehouse features such as shelves, corners, or workstations.

[0084] Here, after extracting multiple first-point features from the first image, first feature description information corresponding to the extracted first-point features can be further determined based on these features. Similarly, after extracting multiple second-point features from the second image, second feature description information corresponding to the extracted second-point features can be further determined based on these features. Here, the first feature description information describes the first-point features; the second feature description information describes the second-point features.

[0085] Here, the Superpoint network can be used to detect point features and extract the corresponding feature description information. Specifically, the Superpoint network can be used to extract multiple first point features from the first image and determine the position information of each first point feature in the first image; multiple second point features can be extracted from the second image and the position information of each second point feature in the second image can be determined. Furthermore, the first feature description information corresponding to each of the above first point features and the second feature description information corresponding to each second point feature are detected.

[0086] Here, the first feature description information and the second feature description information can be used to detect two matching point features. For example, it can be used to detect whether the first point feature and the second point feature are the same point feature.

[0087] Alternatively, the LSD algorithm in OpenCV can be used to detect line segments in an image. For example, multiple first line segments can be extracted from a first image, and the position information of the first line segments in the first image can be determined; and multiple second line segments can be extracted from a second image, and the position information of the second line segments in the second image can be determined.

[0088] In some embodiments, after extracting multiple first line segments and multiple second line segments, a first set of first point features included in any first line segment can be determined, and a second set of second point features included in any second line segment can be determined.

[0089] For example, firstly, the extracted first-point features are combined into a first-point feature set, denoted as {P1, P2, ..., P...}. n}, where P1, P2, ..., P n Each of the first features is defined as a first feature. The first feature description information corresponding to each first feature is then determined. Finally, the first feature description information corresponding to multiple first features is combined into a first feature description information set, denoted as {D1, D2, ..., D...}. n′}, where D1, D2, ..., D n′ These are the first feature descriptions corresponding to the first feature point. And, the extracted multiple second feature points are combined into a second feature set, denoted as {P′1, P′2, ..., P′}. n′}, where P′1, P′2, ..., P′ n′ For each second feature, a second feature description information is determined. Then, the second feature description information corresponding to multiple second features is combined into a second feature description information set, denoted as {D′1, D′2, ..., D′}. n′ Where, D′1, D′2, ..., D′ n′These are the descriptions of the second features corresponding to the second feature points. Then, the extracted first line segments from the first image are grouped into a first line segment set, namely {L1, L2, ..., L...}. m}, where L1, L2, ..., L m The first line segment is defined as {L′1, L′2, ..., L′}; and the second line segment set is formed by extracting multiple second line segments from the second image, denoted as {L′1, L′2, ..., L′}. m′}, where L′1, L′2, ..., L′ m′ These are the second line segments. In the above, n represents the number of first-point features in the first-point feature set, n′ represents the number of second-point features in the second-point feature set, m represents the number of first-line segments in the first-line segment set, and m′ represents the number of second-line segments in the second-line segment set. Then, we can iterate through the distance from each first target point corresponding to each first-point feature to each first line segment, determine whether the first-point feature belongs to the first line segment, and use the first-point features corresponding to the first target points belonging to the first line segment as the point feature set of the first line segment, i.e., the first set, denoted as L1: {P 11 P 12 ..., P 1i}, L2: [P 21 P 22 ..., P 1j}, ..., L m :{P m1 P m2 ..., P mk}, where i represents the number of first-point features in L1, j represents the number of first-point features in L2, and k represents L m The number of first-point features; and, traversing the distance from the second target point corresponding to each second-point feature to each second line segment, determining whether the second-point feature belongs to the second line segment, and using the second-point features corresponding to the second target points belonging to the first line segment as the point feature set of the second line segment, i.e., the second set, denoted as L′1: {P′ 11 , P′ 12 , ..., P′ 1i′}, L′2:{P′ 21 , P′ 22 , ..., P′ 2j′},……,L′ m′ :{P′ m′1 , P′ m′2 , ..., P′ m′k′}, where i′ represents the number of second-point features in L′1, j′ represents the number of second-point features in L′2, and k′ represents the number of second-point features in L′1. m′ The number of the second feature.

[0090] S103: Match the first point feature in the first image with the second point feature in the second image to obtain the matching result.

[0091] In this step, point features in the first and second point feature sets determined in S102 can be matched to determine the matching result.

[0092] For different application scenarios, the matching method for point features can be different, such as the following two methods:

[0093] Method 1: For map creation applications, SuperGlue can be used to match first-point features and second-point features. Specifically, based on the location information of the first-point feature, its corresponding first-feature description information, the location information of the second-point feature, and its corresponding second-feature description information, the first-point feature and the second-point feature can be matched to determine the matching result. For example, firstly, the location information of the first-point feature and the second-point feature can be used to match a preset location range within the same environment. Then, the first-feature description information of the first-point feature and the second-feature description information of the second-point feature within the same preset location range can be matched to achieve feature matching. This determines the matching result for each first-point feature in the first-point feature set and each second-point feature in the second-point feature set, denoted as M1, M2, ..., M... f Here, the matching result can be a matching pair where the first feature and the second feature match each other, for example, in P1 and P′. 10 Matching, P2 and P′3 match, P n With P′ n′ In the case of a match, the matching results can be denoted as M1: {P1, P′} 10}, M2: {P2, P′3},…,M f :{P n , P′ n′}

[0094] Method 2: For relocation application scenarios, the bag-of-words model can be used to match the first and second features. Specifically, based on the first feature description information corresponding to the first feature and the second feature description information corresponding to the second feature, the first and second features are matched to determine the matching result. For example, for each second feature, the second feature description information corresponding to that second feature is iterated through, and then compared sequentially with the second feature description information corresponding to the unmatched first feature to determine the matching result between that second feature and the matched first feature.

[0095] S104: Based on the matching results, determine the degree of matching between any first line segment and any second line segment, and take the first line segment and the second line segment whose degree of matching meets the preset conditions as mutually matched line segments.

[0096] In this step, any first line segment can be any line segment extracted from the first image. Any second line segment can be any line segment extracted from the second image. Since a line segment is composed of multiple target points, the matching result of the point features corresponding to the target points, i.e., the matching result of the first point feature and the second point feature, can be used, denoted as {M1, M2, ..., M...}. f}, further determining the degree of matching between any first line segment and any second line segment.

[0097] The degree of matching between any first line segment and any second line segment can be determined according to S1041 to S1042:

[0098] S1041: Based on the matching results, select target point features from the first point features included in any first line segment that match the second point features included in any second line segment.

[0099] In this step, the matching results include {M1, M2, ..., M} f}

[0100] For example, from the first line segment L1, the first set {P} of the first point features. 11 P 12 ..., P 1i} Filtering the second line segment L′ m′ :{P′ m′1 , P′ m′2 , ..., P′ m′k′ If the second feature in} matches the target point feature, and the first feature P 11 With the second feature P′ m′1 If a match is found, the feature of the target point is P. 11 or P′ m′1 It should be noted that the first and second line segments that are determined to match each other include multiple target point features.

[0101] S1042: Determine the degree of matching between any first line segment and any second line segment based on the first number of target point features, the second number of first point features included in any first line segment, and the third number of second point features included in any second line segment.

[0102] Here, the first quantity of target point features is the number of target points corresponding to those features. The second quantity of the first point features included in any first line segment is the number of first target points included in that first line segment. The third quantity of the second point features included in any second line segment is the number of second target points included in that second line segment.

[0103] In some embodiments, after determining the matching degree between any first line segment and any second line segment, a matrix representing the first number of target point features can be constructed in real time. Specifically, taking a relocation application scenario as an example, a second image is acquired, and any second line segment in the second image is traversed and matched with each first line segment to determine the first number of target point features.

[0104] For example, the first number of target point features that determine whether the second line segment L′1 matches the first line segment L1 is S. 11 The first number of target point features that match the second line segment L′1 and the first line segment L2 is S. 12 ..., determine the relationship between the second line segment L′1 and the first line segment L... m The first number of matching target point features is S 1m The first number of target point features that match the second line segment L′2 with the first line segment L1 is S. 21 The first number of target point features that match the second line segment L′2 with the first line segment L2 is S. 22 ..., determine the relationship between the second line segment L′2 and the first line segment L. m The first number of matching target point features is S 2m Determine the second line segment L′. m′ The first number of target point features matching the first line segment L1 is S. m′1 Determine the second line segment L′ m′ The first number of target point features that match the first line segment L2 is S. m′2 ..., determine the second line segment L′ m′ With the first line segment L m The first number of matching target point features is S m′m For details, please refer to the matrix shown in Table 1.

[0105] Table 1

[0106]

[0107]

[0108] It should be noted that the first number of target point features that match the first line segment and the second line segment shown in Table 1 above can be 0, that is, the first line segment and the second line segment do not match.

[0109] In other embodiments, Table 1 matrix is ​​constructed in real time to represent the first number of target point features. Only the first and second line segments that contain target point features after matching can be constructed, saving computation and improving matching efficiency.

[0110] Based on the first number of target point features calculated above, the second number of first point features included in any first line segment, and the third number of second point features included in any second line segment, the ratio of the first number to the second number and the ratio of the first number to the third number are determined. Based on the first ratio and the second ratio, the degree of matching between any first line segment and any second line segment can be determined.

[0111] Continuing the previous example, if L m The second number of the first feature is N. m , L′ m′ The third number of the included second feature is N′ m′ Then the first ratio is The second ratio is Determine the maximum value between the first ratio and the second ratio, and take the maximum value as the first line segment L. m Second line segment L′ m′ Matching degree v m′m See Formula 1:

[0112]

[0113] The above includes, but is not limited to, taking the maximum of the first ratio and the second ratio as the matching degree. It also allows for optimization of the first ratio and / or the second ratio, determining the weight values ​​of the first ratio and the second ratio respectively, and comprehensively determining the matching degree based on the weight values ​​of the first ratio and the second ratio. This disclosure does not limit the scope of the embodiments. Those skilled in the art can make various substitutions and modifications to the process of determining the matching degree using the first ratio and the second ratio without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

[0114] The matching degree determines whether the first line segment and the second line segment match. Specifically, a target ratio can be determined based on the first ratio and the second ratio, and the target ratio can be used as the matching degree. If the target ratio is greater than or equal to the first preset threshold, the first line segment and the second line segment are determined to be matching line segments.

[0115] For example, in v m′mIf the value is greater than or equal to the first preset threshold, the first line segment L can be determined. m With the second line segment L′ m′ These are line segments that match each other.

[0116] Here, the target ratio can be the maximum value of the target ratios selected above, or it can be an optimized ratio, etc., which is not limited in this embodiment. The first preset threshold value ranges from 0 to 1, and can be the result of parameter adjustment by those skilled in the art. The specific data is not limited in this embodiment.

[0117] Regarding S102, if the number of point features included in the detected line segment is small, it can be determined that there is an error in the detection result or that the line segment does not exist. It is necessary to filter out these line segments to improve the accuracy of subsequent line segment matching.

[0118] In some embodiments, after determining the first line segment to which the first point feature belongs, it can be further determined whether the first line segment is the first line segment to be matched. Here, the first line segment to be matched can be a pre-set line segment, including line segments with a number of first point features greater than a second preset threshold. In specific implementation, the number of first point features in the first line segment is determined, and if the number of first point features is less than or equal to the second preset threshold, the first line segment and the first point features in the first line segment are removed.

[0119] Furthermore, after determining the second line segment to which the second feature belongs, it is possible to further determine whether the second line segment is the second line segment to be matched. Here, the second line segment to be matched can be a pre-set line segment, including line segments with the number of second features greater than a third preset threshold. In specific implementation, the number of second features in the second line segment is determined, and if the number of second features is less than or equal to the third preset threshold, the second line segment and the second features in the second line segment are removed.

[0120] For S102, the first line segment to which each first point feature belongs can be determined by following these steps S1021 to S1022:

[0121] S1021: For each of the multiple first point features, determine the distance from the first target point corresponding to the first point feature to any first line segment.

[0122] S1022: Take the first point feature corresponding to the first target point whose distance is less than or equal to the fourth preset threshold as the point feature belonging to any first line segment.

[0123] For example, given the equation of each first line segment, the distance d1 from each first target point to any first line segment can be determined using the determined position information of each first target point, such as coordinates. Then, a pre-set fourth threshold is used. Will The first point feature corresponding to the first target point is taken as the point feature belonging to any first line segment, that is, the first point feature is the point feature in the first set corresponding to the arbitrary first line segment.

[0124] Furthermore, determining the second line segment to which each second point feature belongs can be done according to the following steps S1023~S1024:

[0125] S1023: For each of the multiple second point features, determine the distance from the second target point corresponding to the second point feature to any second line segment.

[0126] S1024: Take the second point feature corresponding to the second target point whose distance is less than or equal to the fifth preset threshold as the point feature belonging to any second line segment.

[0127] For example, given the equation of each second line segment, the distance d2 from each second target point to any second line segment can be determined using the determined position information of each second target point, such as coordinates. Then, a pre-set fourth threshold is used. Will The second point feature corresponding to the second target point is taken as the point feature belonging to any second line segment, that is, the second point feature is the point feature in the second set corresponding to the arbitrary second line segment.

[0128] The second, third, fourth, and fifth preset thresholds mentioned above can be determined by those skilled in the art based on experience, and are not specifically limited in the embodiments disclosed herein.

[0129] Through steps S101 to S104, line segments are represented using point features, specifically the first line segment to which each first feature point belongs and the second line segment to which each second feature point belongs. This improves the accuracy of line segment matching results when the length and endpoints of the line segment change. For example, it can match an entire line segment using only a portion of its features. Furthermore, since the line segment matching is based on the matching results of the first and second feature points, there is no need to extract the feature description information of the line segment. This simplifies the extraction process, saves computational load and time, and further improves the efficiency of line segment matching.

[0130] For S101 to S104 above, please refer to Figure 2As shown in the figure, this disclosure also provides an algorithm flowchart, wherein the algorithm involved in this disclosure includes the following modules, wherein 211 represents the Superpoint and LSD module corresponding to the first image, 212 represents the Superpoint and LSD module corresponding to the second image, 221 represents the point-line association module corresponding to the first point feature and the first line segment, 222 represents the point-line association module corresponding to the second point feature and the second line segment, 23 represents the point feature matching module, and 24 represents the line segment matching module.

[0131] For example, the input of module 211 is the first image, and the output of module 211 is the first set of point features, i.e., {P1, P2, ..., P...} n The first feature description information is {D1, D2, ..., D}. n′} and the first set of line segments, i.e. {L1, L2, ..., L m The input to module 212 is the second image, and the output of module 212 is the second set of point features, i.e., {P′1, P′2, ..., P′}. n′ The second feature description information is {D′1, D′2, ..., D′}. n′} and the second set of line segments, namely {L′1, L′2, ..., L′ m′}

[0132] The input to module 221 can be {P1, P2, ..., P...} n} and {L1, L2, ..., L m The first feature point is associated with the first line segment. Using the distance d1 from the first target point corresponding to the first feature point to the first line segment, it is determined whether the first feature point belongs to the first line segment. The output of module 221 can be L1: {P 11 P 12 ..., P 1i}, L2: {P 21 P 22 ..., P 1j}, ..., L m : {P m1 P m2 ..., P mk The input to module 222 can be { P P'1, P'2, ..., P' n′} and {L′1, L′2, ..., L′ m′ The second feature is associated with the second line segment. Using the distance d2 from the second target point corresponding to the second feature to the second line segment, it is determined whether the second feature belongs to the second line segment. The output of module 222 can be L′1: {P′ 11 , P′12 , ..., P′ 1i′}, L′2:{P′ 21 , P′ 22 , ……, P′ 2j′},……,L′ m′ :{P′ m′1 , P′ m′2 , ..., P′ m′k′}

[0133] The input to module 23 needs to be determined based on the actual scenario. For example, in a map creation application scenario, 23 indicates that the point feature matching module can be the SuperGlue algorithm model, and its input can be {P1, P2, ..., P...} n}、{D1,D2,……,D n′}, {P′1, P′2,…,P′ n′} and {D′1, D′2, ..., D′ n′ Its output can be {M1, M2, ..., M}; f For example, in a relocation application scenario, 23 represents the point feature matching module, which can be a bag-of-words model, and its input can be {D1, D2, ..., D...}. n′} and {D′1, D′2, ..., D′ n′ Its output can be {M1, M2, ..., M}; f}

[0134] The input to module 24 can be the output of modules 221, 222, and 23; the output of module 24 can be the degree of matching.

[0135] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0136] Based on the same inventive concept, this disclosure also provides a line segment matching device corresponding to the line segment matching method. Since the principle of the device in this disclosure for solving the problem is similar to the line segment matching method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0137] Reference Figure 3 The diagram shown is a schematic of a line segment matching device provided in an embodiment of this disclosure. The device includes: an acquisition module 301, a determination module 302, a first matching module 303, and a second matching module 304; wherein,

[0138] The acquisition module 301 is used to acquire a first image and a second image; the first image and the second image are images from the same environment;

[0139] The determining module 302 is used to extract multiple first point features in the first image and determine the first line segment to which each first point feature belongs, and to extract multiple second point features in the second image and determine the second line segment to which each second point feature belongs;

[0140] The first matching module 303 is used to match the first point feature in the first image with the second point feature in the second image to obtain a matching result;

[0141] The second matching module 304 is used to determine the matching degree of any first line segment and any second line segment based on the matching result, and to take the first line segment and the second line segment whose matching degree meets the preset conditions as mutually matched line segments.

[0142] In one optional implementation, the second matching module 304 is configured to, based on the matching result, filter target point features from the first point features included in the arbitrary first line segment that match the second point features included in the arbitrary second line segment; and determine the degree of matching between the arbitrary first line segment and the arbitrary second line segment based on the first number of target point features, the second number of first point features included in the arbitrary first line segment, and the third number of second point features included in the arbitrary second line segment.

[0143] In one optional implementation, the second matching module 304 is configured to determine a first ratio of the first quantity to the second quantity, and to determine a second ratio of the first quantity to the third quantity; and to determine the degree of matching between the arbitrary first line segment and the arbitrary second line segment based on the first ratio and the second ratio.

[0144] In one optional implementation, the second matching module 304 is configured to determine a target ratio based on the first ratio and the second ratio, and use the target ratio as the degree of matching; if the target ratio is greater than or equal to a first preset threshold, the first line segment and the second line segment are determined to be mutually matched line segments.

[0145] In an optional implementation, the determining module 302 is further configured to, after extracting multiple first point features in the first image and determining the first line segment to which each first point feature belongs, and extracting multiple second point features in the second image and determining the second line segment to which each first point feature belongs, determine a first set of first point features included in any first line segment, and determine a second set of second point features included in any second line segment.

[0146] The second matching module 304 is used to filter target point features that match the second point features included in any second line segment from the first point features included in any first line segment, based on the matching result, the first set and the second set.

[0147] In one optional embodiment, the line segment matching device further includes a filtering module 305;

[0148] The filtering module 305 is configured to, before matching the first point feature in the first image with the second point feature in the second image to obtain a matching result, determine the number of first point features in the first line segment, and remove the first line segment and the first point feature in the first line segment if the number of first point features is less than or equal to a second preset threshold; and determine the number of second point features in the second line segment, and remove the second line segment and the second point feature in the second line segment if the number of second point features is less than or equal to a third preset threshold.

[0149] In an optional implementation, the determining module 302 is further configured to, after extracting multiple first point features in the first image and multiple second point features in the second image, determine first feature description information corresponding to the multiple first point features and second feature description information corresponding to the multiple second point features;

[0150] The first matching module 303 is used to match the first point feature and the second point feature based on the first feature description information and the second feature description information to obtain the matching result.

[0151] In one optional implementation, the determining module 302 is configured to: determine the distance from a first target point corresponding to a first point feature to any first line segment for each of the plurality of first point features; designate the first point feature corresponding to a first target point whose distance is less than or equal to a fourth preset threshold as a point feature belonging to the arbitrary first line segment; and determine the distance from a second target point corresponding to a second point feature to any second line segment for each of the plurality of second point features; designate the second point feature corresponding to a second target point whose distance is less than or equal to a fifth preset threshold as a point feature belonging to the arbitrary second line segment.

[0152] The processing flow of each module in the line segment matching device and the interaction flow between each module can be referred to the relevant descriptions in the above-described line segment matching method embodiments, and will not be detailed here.

[0153] Based on the same technical concept, embodiments of this application also provide a computer device. (Refer to...) Figure 4 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application, including:

[0154] The system includes a processor 41, a memory 42, and a bus 43. The memory 42 stores machine-readable instructions executable by the processor 41. The processor 41 executes these machine-readable instructions, and when executed, performs the following steps: S101: Acquire a first image and a second image; the first image and the second image are images from the same environment; S102: Extract multiple first point features from the first image and determine the first line segment to which each first point feature belongs; and extract multiple second point features from the second image and determine the second line segment to which each second point feature belongs; S103: Match the first point features in the first image with the second point features in the second image to obtain a matching result; S104: Based on the matching result, determine the matching degree between any first line segment and any second line segment, and designate the first line segment and the second line segment whose matching degree meets a preset condition as mutually matched line segments.

[0155] The aforementioned memory 42 includes a main memory 421 and an external memory 422. The main memory 421, also known as internal memory, is used to temporarily store the computational data in the processor 41, as well as the data exchanged with external memory such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421. When the computer device is running, the processor 41 and the memory 42 communicate through the bus 43, so that the processor 41 executes the execution instructions mentioned in the above method embodiments.

[0156] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the line segment matching method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0157] This disclosure also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the line segment matching method described above. The computer program product can be any product capable of implementing the line segment matching method described above. Part or all of the solutions in the computer program product that contribute to the prior art can be embodied in the form of a software product (e.g., a software development kit, SDK). This software product can be stored in a storage medium, and the included computer instructions cause relevant devices or processors to execute part or all of the steps of the line segment matching method described above.

[0158] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In addition, the functional modules in the various embodiments of this disclosure can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0161] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A line segment matching method, characterized in that, include: Acquire a first image and a second image; the first image and the second image are images from the same environment; Multiple first point features and multiple first line segments are extracted from the first image, and the first line segment to which each first point feature belongs is determined. Similarly, multiple second point features and multiple second line segments are extracted from the second image, and the second line segment to which each second point feature belongs is determined. The first line segment is any line segment extracted from the first image, which includes multiple first target points, and the first point feature is the feature corresponding to the first target point in the first image. The second line segment is any line segment extracted from the second image, which includes multiple second target points, and the second point feature is the feature corresponding to the second target point in the second image. Determine the number of first point features in the first line segment; if the number of first point features is less than or equal to a second preset threshold, remove the first line segment and the first point features within the first line segment; and... Determine the number of second point features in the second line segment. If the number of second point features is less than or equal to a third preset threshold, remove the second line segment and the second point features in the second line segment. The first feature in the first image is matched with the second feature in the second image to obtain the matching result; Based on the matching results, target point features that match the second point features included in any second line segment are selected from the first point features included in any first line segment; a first ratio of the first number of target point features to the second number of first point features included in any first line segment is determined, and a second ratio of the first number to the third number of second point features included in any second line segment is determined; based on the first ratio and the second ratio, the matching degree between the arbitrary first line segment and the arbitrary second line segment is determined, and the first line segment and the second line segment whose matching degree meets the preset conditions are regarded as mutually matched line segments.

2. The method according to claim 1, characterized in that, Determining the matching degree between any first line segment and any second line segment based on the first ratio and the second ratio includes: Based on the first ratio and the second ratio, a target ratio is determined, and the target ratio is used as the degree of matching. The step of designating the first line segment and the second line segment whose matching degree meets the preset conditions as mutually matched line segments includes: If the target ratio is greater than or equal to a first preset threshold, the first line segment and the second line segment are determined to be mutually matched line segments.

3. The method according to claim 1, characterized in that, After extracting multiple first point features from the first image and determining the first line segment to which each first point feature belongs, and extracting multiple second point features from the second image and determining the second line segment to which each second point feature belongs, the method further includes: Determine a first set of first point features included in any first line segment, and determine a second set of second point features included in any second line segment; The step of filtering target point features from the first point features included in any first line segment that match the second point features included in any second line segment, based on the matching result, includes: Based on the matching results, the first set, and the second set, target point features that match the second point features included in any second line segment are selected from the first point features included in any first line segment.

4. The method according to claim 1, characterized in that, After extracting multiple first-point features from the first image and multiple second-point features from the second image, the method further includes: Determine the first feature description information corresponding to the multiple first point features and the second feature description information corresponding to the multiple second point features; The step of matching the first feature point in the first image with the second feature point in the second image to obtain a matching result includes: Based on the first feature description information and the second feature description information, the first point feature and the second point feature are matched to obtain the matching result.

5. The method according to claim 1, characterized in that, Determining the first line segment to which each first point feature belongs includes: For each of the plurality of first point features, determine the distance from the first target point corresponding to the first point feature to any first line segment; and take the first point feature corresponding to the first target point whose distance is less than or equal to a fourth preset threshold as the point feature belonging to the arbitrary first line segment. Determining the second line segment to which each second point feature belongs includes: For each of the plurality of second point features, determine the distance from the second target point corresponding to the second point feature to any second line segment; and take the second point feature corresponding to the second target point whose distance is less than or equal to a fifth preset threshold as the point feature belonging to the arbitrary second line segment.

6. A line segment matching device, characterized in that, include: The acquisition module is used to acquire a first image and a second image; the first image and the second image are images from the same environment; The determining module is configured to extract multiple first point features and multiple first line segments from the first image, and determine the first line segment to which each first point feature belongs; and to extract multiple second point features and multiple second line segments from the second image, and determine the second line segment to which each second point feature belongs; wherein, the first line segment is any line segment extracted from the first image, the first image includes multiple first target points, and the first point feature is the feature corresponding to the first target point in the first image; the second line segment is any line segment extracted from the second image, the second image includes multiple second target points, and the second point feature is the feature corresponding to the second target point in the second image; the module determines the number of first point features in the first line segment, and if the number of first point features is less than or equal to a second preset threshold, discards the first line segment and the first point features in the first line segment; and determines the number of second point features in the second line segment, and if the number of second point features is less than or equal to a third preset threshold, discards the second line segment and the second point features in the second line segment. The first matching module is used to match the first point feature in the first image with the second point feature in the second image to obtain a matching result; The second matching module is configured to, based on the matching result, filter target point features from the first point features included in any first line segment that match the second point features included in any second line segment; determine a first ratio between the first number of target point features and the second number of first point features included in any first line segment, and determine a second ratio between the first number and the third number of second point features included in any second line segment; determine the degree of matching between any first line segment and any second line segment based on the first ratio and the second ratio, and designate the first line segment and the second line segment whose degree of matching meets a preset condition as mutually matched line segments.

7. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the line segment matching method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the line segment matching method as described in any one of claims 1 to 5.

Images