A method for matching line segments in an image, an image positioning method and an electronic device

Abstract

The application provides a line segment matching method in images, an image positioning method and an electronic device, and relates to the technical field of image recognition. The method comprises the following steps: acquiring descriptor information and geometric feature information of each line segment in a first line segment set corresponding to a first image and a second line segment set corresponding to a second image; and determining mutually matched line segments in the first line segment set and the second line segment set according to the descriptor information and the geometric feature information of each line segment. In the process of determining matched line pairs, the application not only considers the descriptor information, but also considers the geometric feature information of the line segments, thereby improving the matching efficiency and effectively improving the precision of the matching result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a line segment matching method in an image, an image positioning method and an electronic device. BACKGROUND

[0002] Image matching is a key task in the field of computer vision and image processing, and plays a very important role in the fields of remote sensing image analysis, visual navigation, target recognition and the like. There are a large number of line features in actual scenes, which often exist in the boundaries, textures and the like of objects and can reflect the structural information of real scenes. The matching of lines in images is very much applied in the fields of visual SLAM (simultaneous localization and mapping) and three-dimensional reconstruction, and therefore, how to accurately match the line features in two images is particularly important.

[0003] The existing image line segment matching algorithm is based on a traditional visual method, which first calculates the descriptor of each line segment in an image, such as an LBD descriptor and an MSLD descriptor. The final matching result is obtained based on the distance of the descriptors.

[0004] The prior art only uses the descriptor of a line for matching, and ignores the geometric information of the line in an image. When there are repeated textures such as boundary lines of floor tiles in the image, only using the descriptor will result in erroneous matching. In addition, the geometric information of the line often has consistency in matching, such as two lines with similar positions on an image, and the two line segments matched with them on another image also usually have similar positions, or two mutually perpendicular lines on an image, and the two line segments matched with them on another image also usually are mutually perpendicular. Ignoring the geometric information of the matching will cause the repeated matching of lines with consistency, and reduce the matching efficiency of the line segments in the image. SUMMARY

[0005] In view of this, the purpose of the present application is to provide a line segment matching method in an image, an image positioning method and an electronic device to improve the accuracy and matching efficiency of line segment matching.

[0006] In a first aspect, an embodiment of the present application provides a line segment matching method in an image, which comprises: obtaining the descriptor information and geometric feature information of each line segment in a first line segment set corresponding to a first image and a second line segment set corresponding to a second image; wherein the geometric feature information comprises at least one of the position, direction and length of the line segment in the first image and the second image, respectively; and determining the line segments matched with each other in the first line segment set and the second line segment set according to the descriptor information and the geometric feature information of each line segment.

[0007] Further, the step of determining the matching line segments in the first line segment set and the second line segment set according to the description sub-information and the geometric feature information of each line segment comprises: adding the geometric feature information corresponding to the line segment to the description sub-information corresponding to the line segment to obtain initial fusion features corresponding to the line segment; and determining the matching line segments in the first line segment set and the second line segment set according to the initial fusion features.

[0008] Further, the step of adding the geometric feature information corresponding to the line segment to the description sub-information corresponding to the line segment to obtain initial fusion features corresponding to the line segment comprises: determining a geometric feature vector with the same dimension as the description sub-information of the line segment according to the geometric feature information of the line segment; and adding the geometric feature vector and the description sub-information of the line segment to obtain the initial fusion features of the line segment.

[0009] Further, the step of determining the matching line segments in the first line segment set and the second line segment set according to the initial fusion features comprises: determining an attention graph corresponding to each line segment by using the initial fusion features of the line segment and a graph attention neural network; and determining the matching line segments in the first line segment set and the second line segment set according to the attention graph.

[0010] Further, the step of determining the feature representation vector of each line segment according to the attention graph comprises: calculating an attention coefficient of each edge in the attention graph; performing normalization processing on each attention coefficient to obtain a normalized attention coefficient of each edge; performing feature aggregation on the initial fusion features of each line segment and the normalized attention coefficient of the edge corresponding to the line segment to obtain aggregated features of the line segment; updating the initial fusion features by using the aggregated features to obtain updated fusion features; and determining the feature representation vector of the line segment by using the updated fusion features.

[0011] Further, the step of determining the feature representation vector of each line segment according to the attention graph comprises: determining a module feature representation vector of each line segment by using each attention module in the graph neural network; wherein the graph neural network comprises at least two attention modules; and combining the plurality of module feature representation vectors corresponding to each line segment to obtain the feature representation vector of the line segment.

[0012] Further, the step of determining the feature representation vector of each straight line segment according to the attention map comprises: processing the attention map through a current attention network layer to obtain a network layer feature of each straight line segment; determining whether the current attention network layer is the last layer of the graph neural network, if yes, determining the network layer feature as the feature representation vector of the straight line segment; wherein the graph neural network comprises at least two attention network layers, and each attention network layer comprises at least one attention module; if no, updating the attention map through the network layer feature, and continuing to process the updated attention map through a next attention network layer.

[0013] Further, the graph neural network comprises a self-attention network layer and a cross-attention network layer, the self-attention network layer and the cross-attention network layer are connected alternately; the attention map input into the self-attention network layer is a self-attention map, and the attention map input into the cross-attention network layer is a cross-attention map; wherein an edge in the self-attention map is a connecting line of any two straight line segments in the first straight line segment set or a connecting line of any two straight line segments in the second straight line segment set; an edge in the cross-attention map is a connecting line of any one straight line segment in the first straight line segment set and any one straight line segment in the second straight line segment set.

[0014] Further, the graph neural network is obtained by the following method: obtaining the descriptor information and the geometric feature information of each sample straight line segment in a first sample straight line segment set corresponding to a first sample image and a second sample straight line segment set corresponding to a second sample image; for each sample straight line segment, the descriptor information and the geometric feature information of the sample straight line segment are fused into an initial fusion feature of the sample straight line segment, and an attention map corresponding to the sample straight line segment is determined according to the initial fusion feature of the sample straight line segment; a predicted feature representation vector of each sample straight line segment is determined according to the attention map corresponding to each sample straight line segment through a graph neural initial network; a predicted score matrix is determined according to the predicted feature representation vector; each element in the predicted score matrix represents a similarity score of two sample straight line segments not belonging to the same sample image; a predicted assignment matrix is obtained by solving an optimization problem based on the predicted score matrix; each element in the predicted assignment matrix represents a matching probability between sample straight line segments in the first sample image and the second sample image respectively; a loss value is determined according to the predicted assignment matrix, the label features corresponding to the first sample image and the second sample image respectively, and the graph neural initial network is trained according to the loss value until a training stop condition is met, and a trained graph neural network is obtained.

[0015] In a second aspect, the embodiments of the present application further provide an image positioning method, which comprises: performing straight line segment matching on two images to be positioned by using the image straight line segment matching method of the first aspect, to determine straight line segments matched with each other in the images to be positioned; and determining the position of one of the images to be positioned in the other image to be positioned according to the position information of the straight line segments matched with each other.

[0016] In a third aspect, the embodiments of the present application further provide an electronic device, which comprises a processor and a memory, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the image straight line segment matching method of the first aspect and the image positioning method of the second aspect.

[0017] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the image straight line segment matching method of the first aspect and the image positioning method of the second aspect.

[0018] In a fifth aspect, the embodiments of the present application further provide a computer program product, which comprises a computer program, and the computer program, when executed by a processor, implements the image straight line segment matching method of the first aspect and the image positioning method of the second aspect.

[0019] The image straight line segment matching method, the image positioning method and the electronic device provided by the embodiments of the present application determine the straight line segments matched with each other in the first straight line segment set and the second straight line segment set according to the descriptor information and the geometric feature information of each straight line segment, wherein the descriptor information and the geometric feature information of each straight line segment are obtained. In the process of determining the matched straight line pairs, the embodiments of the present application not only consider the descriptor information, but also consider the geometric feature information of the straight line segments, thereby improving the matching efficiency and effectively improving the precision of the matching result.

[0020] Other features and advantages of the present disclosure will be described in the following description, or can be learned from the description, or can be determined without any doubt, or can be known by implementing the above-mentioned technologies of the present disclosure.

[0021] In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are specifically described with reference to the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0023] Figure 1 A structural schematic diagram of an electronic system provided for an embodiment of the present application is shown in the figure.

[0024] Figure 2 A flowchart of a straight line segment matching method in an image provided for an embodiment of the present application is shown in the figure.

[0025] Figure 3 A flowchart of another straight line segment matching method in an image provided for an embodiment of the present application is shown in the figure.

[0026] Figure 4 A structural schematic diagram of a graph neural network provided for an embodiment of the present application is shown in the figure.

[0027] Figure 5 A flowchart of a training method of a graph neural network provided for an embodiment of the present application is shown in the figure.

[0028] Figure 6 A flowchart of an image positioning method provided for an embodiment of the present application is shown in the figure.

[0029] Figure 7 A straight line segment matching device in an image provided for an embodiment of the present application is shown in the figure.

[0030] Figure 8 An image positioning device provided for an embodiment of the present application is shown in the figure.

[0031] Figure 9 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be clearly described below with reference to the drawings.

[0033] In recent years, important progress has been made in the research of computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence. Artificial intelligence (AI) is a new science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks and many other technology categories. Computer vision, as an important branch of artificial intelligence, is specifically to enable machines to recognize the world. Computer vision technology generally includes face recognition, liveness detection, fingerprint recognition and anti-forgery verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, character recognition, video processing, video content recognition, behavior recognition, three-dimensional reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning, and other technologies. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as security, city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile imaging, cloud services, smart home, wearable devices, driverless vehicles, autonomous driving, intelligent medical care, face payment, face unlocking, fingerprint unlocking, face and certificate verification, smart screens, smart televisions, cameras, mobile Internet, network live streaming, beauty, makeup, medical cosmetology, intelligent temperature measurement, and other fields.

[0034] In the current straight line segment matching method, the geometric characteristics of the straight line segment are not considered, resulting in low accuracy of straight line segment matching. Therefore, the embodiment of the present application provides a straight line segment matching method in an image, an image positioning method and an electronic device, which can effectively improve the accuracy and matching efficiency of straight line segment matching in an image.

[0035] Referring to Figure 1 the structural schematic diagram of an electronic system 100. The electronic system can be used to implement the straight line segment matching method in an image, the image positioning method and the device according to the embodiment of the present application.

[0036] As Figure 1 shown in the structural schematic diagram of an electronic system, the electronic system 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108 and one or more image acquisition devices 110, which are interconnected through a bus system 112 and / or other forms of connection mechanism (not shown). It should be noted that Figure 1 The components and structure of the electronic system 100 shown are only exemplary and are not limiting. The electronic system can also have other components and structures as needed.

[0037] The processing device 102 can be a server, a smart terminal, or a device including a central processing unit (CPU) or other forms of processing units having data processing capability and / or instruction execution capability, and can process data of other components in the electronic system 100 and control other components in the electronic system 100 to perform the straight line segment matching in an image and the image positioning.

[0038] The storage 104 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM), cache memory, and / or the like. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer readable storage media, and the processing device 102 can run the program instructions to implement the client functions (implemented by the processing device) in the embodiments of the present application below and / or other desired functions. Various application programs and various data, such as various data used and / or generated by the application programs, and the like, can also be stored in the computer readable storage media.

[0039] The input device 106 can be a device used by a user to input instructions, and can include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0040] The output device 108 can output various information (e.g., images or sounds) to the outside (e.g., a user), and can include one or more of a display, a speaker, and the like.

[0041] The image acquisition device 110 can acquire an image containing straight line segment information, and store the image in the storage 104 for use by other components.

[0042] Exemplarily, the various devices used to implement the straight line segment matching method in an image, the image positioning method, and the electronic device according to the embodiments of the present application can be integrally arranged or dispersedly arranged, such as integrally arranging the processing device 102, the storage 104, the input device 106, and the output device 108 in one body, and arranging the image acquisition device 110 at a designated position where an image can be acquired. When the various devices in the above electronic system are integrally arranged, the electronic system can be implemented as a smart terminal such as a mobile robot, a camera, a smart phone, a tablet computer, a computer, a vehicle-mounted terminal, and the like.

[0043] Figure 2 A flowchart of a straight line segment matching method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the straight line segment matching method according to the embodiment of the present application can include the following steps. Figure 2As shown, the method comprises:

[0044] S202: Obtain the descriptor information and geometric feature information of each straight line segment in the first set of straight line segments corresponding to the first image and the second set of straight line segments corresponding to the second image; wherein the geometric feature information comprises at least one of the position, direction and length of the straight line segment in the first image and the second image respectively;

[0045] The set of straight line segments can be a vector with a length same as the number of straight line segments contained in the image, and each straight line segment is represented by the coordinates of two end points of the straight line segment in the image. The descriptor corresponding to each straight line segment in the set of straight line segments can be obtained by using an existing method for extracting descriptors, for example, the descriptor can be obtained by using the LSD (LSD, Line Segment Detector, line detection segmentation algorithm) algorithm, the LSD algorithm can obtain a detection result with sub-pixel level accuracy within linear time, and the descriptor can also be obtained by using the LBD algorithm, and the present application does not limit the method for obtaining the descriptor. Each descriptor is represented by a fixed dimension vector, and in some methods, the descriptor is encoded into a pixel gradient histogram of a neighborhood of the straight line segment. The descriptor does not contain position information, but only uses the brightness information of the pixels.

[0046] The geometric features of the straight line segment include the position, direction and length of the straight line segment, wherein the position of the straight line segment in the first image and the second image can be represented by the center point coordinates of the straight line segment in the first image, or can be represented by the starting coordinates of the straight line segment in the first image and the second image. The direction of the straight line segment can be the included angle between the straight line segment and the horizontal line or the vertical line of the image. The above information of the straight line segment is connected to form the geometric feature corresponding to the straight line segment.

[0047] It should be noted that in the process of obtaining the descriptor information of the straight line segment, in order to obtain the gradient information of each pixel neighborhood, the starting point and end point coordinates and other geometric information of the straight line segment are used, but these geometric information is an intermediate process for obtaining the descriptor information, and the finally obtained descriptor information still only reflects the appearance information of the straight line segment, and does not contain related geometric information.

[0048] S204: Determine the straight line segments matched with each other in the first set of straight line segments and the second set of straight line segments according to the descriptor information and geometric feature information of each straight line segment.

[0049] The descriptor information of the straight line segment is used to describe some attributes of the straight line segment, such as shape, size, etc., but cannot represent the geometric position relationship between the straight line segment and the image. In the process of determining the matched straight line segments, in addition to considering the descriptor information, the geometric feature information of the straight line segment is also considered, so that the straight line segments that cannot be matched in the set relationship can be excluded first, the matching time is saved, and the accuracy of the matching result is improved.

[0050] The method for matching line segments in images provided by the present application comprises the following steps: obtaining the descriptor information and geometric feature information of each line segment in a first line segment set corresponding to a first image and a second line segment set corresponding to a second image; and determining the line segments in the first line segment set and the second line segment set that match each other according to the descriptor information and geometric feature information of each line segment. In the process of determining the matched line pairs, the present application not only considers the descriptor information, but also considers the geometric feature information of the line segments, thereby improving the matching efficiency and effectively improving the precision of the matching results.

[0051] In some possible implementation manners, step S204 in the above embodiment can be specifically as follows:

[0052] (1) For each line segment, the geometric feature information corresponding to the line segment is added to the descriptor information corresponding to the line segment to obtain the initial fusion feature corresponding to the line segment.

[0053] For each line segment, the method for determining the initial fusion feature can be as follows: according to the geometric feature information of the line segment, a geometric feature vector with the same dimension as the descriptor information of the line segment is determined; and the geometric feature vector and the descriptor information of the line segment are added to obtain the initial fusion feature of the line segment.

[0054] Specifically, the geometric feature of the line segment can be converted in dimension to a vector with the same dimension as the descriptor of the line segment, which is referred to as a geometric feature vector. For example, the geometric feature can be mapped to the same dimension as the descriptor by an MLP (multi-layer perception).

[0055] After the geometric feature vector is obtained, the geometric feature vector and the descriptor information with the same dimension are added to obtain the initial fusion feature corresponding to the line segment. Specifically, the fusion can be performed by using the following formula, i.e., h i =d i +MLP(g i ), wherein h i represents the initial fusion feature of the i-th line segment, d i represents the descriptor of the i-th line segment, g i is the geometric feature of the i-th line segment, and MLP contains learnable parameters, and all the initial fusion features share one set of MLP parameters in the process of obtaining.

[0056] It should be noted that, since there are some straight line segments in one image that do not have a matching straight line segment in the other image, in some embodiments, an invalid straight line segment can be set and added to the straight line segment set, and the initial fusion feature of the invalid straight line segment can be a randomly set feature vector, for example, each element in the feature vector is set to 0. Based on this, when a straight line segment does not have a matching straight line segment in the other image, the straight line segment is matched with the invalid straight line segment in the other image. The dimension of the initial fusion feature of the invalid straight line segment is the same as that of the initial fusion feature of other straight line segments, and the method for obtaining the fusion feature of the invalid straight line segment can use a machine learning method, and the parameters used in the process of adding the invalid straight line segment in the two images are shared. By adding the invalid straight line segment, the straight line segments that do not have a matching relationship can be effectively found out, and the straight line segments that do not have a matching relationship are prevented from participating in the matching of other straight line segments to affect the matching accuracy of other straight line segments, so that the accuracy of the matching result of the straight line segments can be further improved.

[0057] (2) According to the initial fusion feature, determine the straight line segments matched with each other in the first straight line segment set and the second straight line segment set.

[0058] For each straight line segment, the descriptor information and the geometric feature information of the straight line segment are fused into the initial fusion feature of the straight line segment, and the attention graph corresponding to the straight line segment is determined according to the initial fusion feature of the straight line segment; wherein the node in the attention graph is the straight line segment, and the edge is the connection between the straight line segment and another straight line segment;

[0059] Wherein, each node in the attention graph represents a straight line segment, and the edge represents the connection between any two straight line segments. The feature corresponding to each straight line segment is the initial fusion feature obtained by fusing the descriptor information and the geometric feature information. It should be noted that when the nodes in the attention graph only include the straight line segments in the first image, or when the nodes in the attention graph only include the straight line segments in the second image, the attention graph is a self-attention graph, and the edges in the self-attention graph represent the connection between the nodes represented by any two straight line segments in the first image or the connection between the nodes represented by any two straight line segments in the second image. When the nodes in the attention graph include the straight line segments in the first image and the straight line segments in the second image, and the edge is the connection between the node corresponding to any one straight line segment in the first image and the node corresponding to any one straight line segment in the second image, the attention graph is called a cross-attention graph.

[0060] Please refer to Figure 3 , Figure 3 The flowchart of another method for matching straight line segments in images provided by the embodiment of the application is shown, and the method specifically includes:

[0061] S302: Obtain the descriptor information and the geometric feature information of each straight line segment in the first set of straight line segments corresponding to the first image and the second set of straight line segments corresponding to the second image;

[0062] The first image and the second image are two images to be matched containing a set of straight line segments, the first image can be denoted as A, which contains M straight line segments, and the second image can be denoted as B, which contains N straight line segments.

[0063] After the center c, the angle θ and the length l of each straight line segment are calculated, they are connected to form the geometric feature of the straight line segment.

[0064] S304: For each straight line segment, add the geometric feature information corresponding to the straight line segment to the descriptor information corresponding to the straight line segment to obtain the initial fusion feature corresponding to the straight line segment;

[0065] S306: Determine the attention map corresponding to each straight line segment according to the initial fusion feature corresponding to the straight line segment and the graph attention neural network.

[0066] S308: Determine the straight line segments matched with each other in the first set of straight line segments and the second set of straight line segments according to the attention map.

[0067] The method provided by the embodiment of the application can determine the straight line segments matched with each other in the first set of straight line segments and the second set of straight line segments based on the cross-attention map, or determine the straight line segments matched with each other in the first set of straight line segments and the second set of straight line segments based on the self-attention map and the cross-attention map.

[0068] In some possible implementation manners, the above steps can be specifically implemented as follows:

[0069] Step 1: Determine the feature representation vector of each straight line segment according to the attention map;

[0070] The feature representation vector can be determined by the graph neural network, and the process of obtaining the feature representation vector by the graph neural network will be described in detail below, which will not be repeated here.

[0071] Step 2: Determine the matching probability of each straight line segment in the first set of straight line segments and each straight line segment in the second set of straight line segments according to the feature representation vector.

[0072] Specifically, the similarity of each straight line segment in the first set of straight line segments and each straight line segment in the second set of straight line segments can be determined according to the feature representation vector, the similarity can be the distance between the feature representation vectors corresponding to the two straight line segments, and further, the matching probability of the two straight line segments can be determined according to the similarity of the two straight line segments.

[0073] In some embodiments, the matching probability can be obtained in the following manner:

[0074] (1) determining a similarity score between each line segment in the first line segment set and each line segment in the second line segment set through the feature representation vectors, and determining a score matrix according to the similarity scores;

[0075] Specifically, the similarity score of line segment i and line segment j is where Sij represents the element in the ith row and jth column of the score matrix S, i,j represents the feature representation vector of the ith line segment in the first image, represents the feature representation vector of the jth line segment in the second image, and T represents transposition. The score matrix is used to reflect the similarity of two nodes (i.e., line segments), and the higher the similarity, the higher the score, and the more likely the two line segments form a match.

[0076] (2) determining a assignment matrix corresponding to the optimal solution based on the score matrix; each element in the assignment matrix is used to represent the matching probability of each line segment in the first line segment set and each line segment in the second line segment set.

[0077] where it is necessary to solve the assignment matrix P ∈ [0, 1] (N+1)×(M+1) , each element P i,j in the assignment matrix represents the probability of matching the ith line segment in the first image and the jth line segment in the second image, N represents the number of line segments in the first image, and M represents the number of line segments in the second image. According to the above description, an invalid line segment is added to each image, and therefore, the size of the assignment matrix is (N+1) x (M+1). The assignment matrix P has the following constraints, and The optimal assignment matrix P is obtained by solving the optimization problem The above constrained optimization problem can be solved in a differentiable manner by Sinkhorn algorithm.

[0078] Step 3: determining the line segments in the first line segment set and the second line segment set that match each other according to the matching probability.

[0079] It can be understood that the sum of all matching probabilities corresponding to each line segment in the assignment matrix is 1. For example, there are two line segments a1 and a2 and an invalid line segment a3 in the first image, and there are three line segments b1, b2 and b3 and an invalid line segment b4 in the second image. The assignment matrix obtained by the matching method provided in the embodiments of the present application is ​It can be seen that the four probabilities corresponding to the straight line segment a1 in the assignment matrix are as follows: the matching probability of a1 and b1 is 0.9, the matching probability of a1 and b2 is 0.1, the matching probability of a1 and b3 is 0, and the matching probability of a1 and b4 is 0. The sum of the four probabilities is equal to 1, and the same is true for other straight line segments.

[0080] Further, in some embodiments, a probability threshold can be preset, and a pair of straight line segments greater than the probability threshold is regarded as a pair of matching straight line segments. Continuing the previous example, the probability threshold is 0.6, and the pairs of straight line segments with a matching probability greater than 0.6 are a1 and b1, a2 and b3. Although the matching probability of a3 and b2 is also greater than the probability threshold, a3 is an invalid straight line segment, which means that there is no straight line segment matching b2 in the first image. Therefore, the final matching result of the straight line segments in the first image and the second image is that a1 and b1 and a2 and b3 match each other.

[0081] In the step S310, the feature representation vector of each straight line segment is determined according to the attention graph. Specifically, the feature representation vector of the straight line segment can be determined by processing the attention graph through a graph neural network.

[0082] In some embodiments, the feature representation vector can be determined in the following manner:

[0083] (1) Calculate the attention coefficient of each edge in the attention graph;

[0084] Specifically, the attention coefficient can be determined by e ij =a(Wh i ,Wh j ), where e ij represents the attention coefficient of the edge formed by the node i and the node j in the attention graph, h i and h j represent the initial fusion features of the node i and the node j, and W is a learnable parameter. a is an attention mechanism. Optionally, a single-layer feedforward neural network is used in the present application, that is, a(Wh i ,Wh j )=σ(a T [Wh i ||Wh j ]), where W is a learnable parameter, a T is a weight vector, and σ is an activation function.

[0085] (2) Normalize each attention coefficient to obtain the normalized attention coefficient of each edge;

[0086] For each edge in the attention graph, the attention coefficient is normalized, that is, where αij is a normalized attention coefficient of the connection between nodes i and j, e ij is an attention coefficient of the connection between nodes i and j, is a neighbor node of node i, which is selected differently according to whether the current edge belongs to a self-attention graph or a cross-attention graph.

[0087] (3) aggregating the initial fusion feature of each straight line segment with the normalized attention coefficient of the edge corresponding to the straight line segment to obtain the aggregated feature of the straight line segment;

[0088] For each node in the feature graph, feature aggregation is performed, and specifically, the feature aggregation can be performed using the following formula: where h i ′ represents the aggregated feature of node i, i.e., the aggregated feature of the straight line segment represented by node i, is a neighbor node of node i, W is a learnable parameter, and h j is the initial fusion feature of node j, and a ij is a normalized attention coefficient of the connection between nodes i and j, and σ is an activation function.

[0089] (4) updating the initial fusion feature by using the aggregated feature to obtain an updated fusion feature, and determining the feature representation vector of the straight line segment by using the updated fusion feature.

[0090] For each node in the attention graph, feature updating is performed, and the initial fusion feature of the node is updated by using the aggregated aggregated feature. There are many updating methods, and optionally, the following updating method can be used: where represents the updated fusion feature of node i, represents the initial fusion feature of node i before updating, and h i ′ represents the aggregated feature of node i, || represents a connection operation, W t is a learnable parameter, and σ is an activation function.

[0091] In some other embodiments, the above-mentioned graph neural network can adopt a multi-head mechanism, i.e., the graph neural network includes at least two attention modules, and the method for determining the feature representation vector of each straight line segment by using the multi-head graph neural network can be specifically:

[0092] (1) determining a module feature representation vector of each straight line segment by using each attention module in the graph neural network;

[0093] The plurality of attention modules are parallel attention modules, the plurality of attention modules are of the same structure, but the parameters of each attention module are different, and the specific parameters can be manually set according to experience or obtained through learning of sample images. All nodes in the attention graph are input into the plurality of attention modules, the plurality of attention modules are processed in parallel, each module obtains a feature representation vector of each straight line segment, which can be referred to as a module feature representation vector corresponding to the straight line segment and the attention module.

[0094] (2) The plurality of module feature representation vectors corresponding to each straight line segment are combined to obtain a feature representation vector of the straight line segment.

[0095] Specifically, after each module obtains a module feature representation vector, the plurality of module feature representation vectors corresponding to the same straight line segment are combined to obtain a feature representation vector corresponding to the straight line segment, and the specific combination manner can be to take an average or other combination manners, which are not limited in the application.

[0096] In order to increase the communication and propagation between the features of the nodes in a targeted manner, the feature representation vectors can be extracted in a targeted manner according to the categories of the attention graphs, that is, the feature representation vectors of the nodes in the self-attention graph are extracted in one stage, and the feature representation vectors of the nodes in the cross-attention graph are extracted in another stage, so that the extracted feature representation vectors can more fully reflect the relationship between the nodes in each stage.

[0097] Based on this, in some possible embodiments, the above-mentioned graph neural network can further include a plurality of attention layers, each attention layer includes at least one attention module, and the feature representation vector of the straight line segment can be determined by the following method:

[0098] (1) The attention graph is processed by the current attention network layer to obtain a network layer feature of each straight line segment;

[0099] Specifically, the graph neural network includes a plurality of attention network layers, for each attention network layer, one attention module or a plurality of attention modules can be included, each attention module extracts a feature representation vector by using the above-mentioned method provided in the embodiments of the application, and when there are a plurality of attention modules, the plurality of module feature representation vectors are combined to obtain a feature representation vector corresponding to the straight line segment, and the feature representation vector is used as a network layer feature corresponding to the attention network layer.

[0100] (2) It is judged whether the current attention network layer is the last layer of the graph neural network, if yes, the network layer feature is determined as the feature representation vector of the straight line segment; wherein the graph neural network includes at least two attention network layers, and each attention network layer includes at least one attention module;

[0101] (3) If no, update the attention map through the network layer feature, and continue to process the updated attention map through the next attention network layer.

[0102] Specifically, taking the case of two layers of the graph attention network, if the current attention network layer is the first layer, the network layer feature corresponding to the attention network layer is obtained, which can be taken as the initial fusion feature of each straight line segment, input into the second attention network layer, and the network layer feature corresponding to the second attention network layer is obtained, which is the final feature representation vector corresponding to the straight line segment.

[0103] In some embodiments, the graph neural network includes self-attention network layers and cross-attention network layers, and the self-attention network layers and the cross-attention network layers are connected alternately; as shown in Figure 4 Figure 4 is a structural schematic diagram of a graph neural network provided by an embodiment of the present application.

[0104] Among them, the left multi-head network is a self-attention network layer, and the right multi-head network is a cross-attention network layer. Each network layer includes 3 attention modules.

[0105] Among them, the attention map input into the self-attention network layer is a self-attention map, and the attention map input into the cross-attention network layer is a cross-attention map.

[0106] Among them, the edges in the self-attention map are the connecting lines of any two straight line segments in the first straight line segment set or the connecting lines of any two straight line segments in the second straight line segment set; and the edges in the cross-attention map are the connecting lines of any one straight line segment in the first straight line segment set and any one straight line segment in the second straight line segment set.

[0107] It can be understood that when the graph neural network includes multiple self-attention network layers and cross-attention network layers, the self-attention network layers and the cross-attention network layers can be connected alternately or sequentially, for example, the graph neural network includes 3 self-attention network layers, denoted as Self1-Self3, and 5 cross-attention network layers, denoted as Cross1-Cross5. The graph neural network can adopt the form of Self1-Self2-Self3-Cross1-Cross2-Cross3-Cross4-Cross5, or the form of Self1-Cross1-Self2-Cross2-Self3-Cross3-Cross4-Cross5.

[0108] ​For ease of understanding, the following describes how to match straight line segments in images by using the straight line segment matching method provided by the embodiments of the present application in combination with actual application scenarios:

[0109] (1) Given two images A and B, and a set of straight line segments in the images, and a set of descriptors of the straight line segments M and N are the numbers of straight line segments in images A and B, respectively.

[0110] (2) Obtain the geometric feature g = c corresponding to each straight line segment x ||c y ||cos(θ)||l, and || represents a connection operator.

[0111] (3) Map the geometric feature of each straight line segment through an MLP to obtain a geometric feature of the same dimension as the descriptor, and fuse the transformed geometric feature and the descriptor to obtain a fused feature representation of each straight line segment.

[0112] (4) Construct a graph neural network, which includes three self-attention layers and three cross-attention layers, wherein the self-attention layers and the cross-attention layers are arranged alternately and connected end to end, and each attention layer includes two attention modules.

[0113] (5) In each attention module of the self-attention layer, calculate the attention coefficient corresponding to each edge, and normalize the attention coefficient to obtain a normalized attention coefficient.

[0114] (6) For each point, perform feature aggregation based on the normalized attention coefficient to obtain an aggregated first feature vector.

[0115] (7) Combine the feature vectors of each node obtained by the two attention modules in the current self-attention network layer to obtain a second feature vector corresponding to each node of the current self-attention network layer.

[0116] (8) Input the second feature vector into the next cross-attention network layer to obtain a third feature vector corresponding to each node of the current cross-attention network layer using the above method.

[0117] (9) Repeat the above process until the output of the last cross-attention network layer is obtained.

[0118] (10) According to the final output feature vector, calculate a score matrix of the similarity score between any one straight line segment in the first image and any one straight line segment in the second image.

[0119] (11) Based on the score matrix, solve an optimization problem to obtain an optimal assignment matrix.

[0120] In order to make the feature representation vector obtained by the graph neural network more accurately represent the characteristics of the line segment, the above-mentioned graph neural network is trained by Figure 5 The training method of the graph neural network shown in the figure is trained to obtain:

[0121] S502: Obtain the descriptor information and geometric feature information of each sample line segment in the first sample line segment set corresponding to the first sample image and the second sample line segment set corresponding to the second sample image;

[0122] Wherein, the first sample image and the second sample image contain a line segment set, and at the same time, the first sample image and the second sample image are marked with actually matched line segments. For example, the true matching label can be represented by a label matrix , which indicates that the i-th line segment in the first sample image (A) is actually matched with the j-th line segment in the second sample image (B).

[0123] S504: For each sample line segment, the descriptor information and geometric feature information of the sample line segment are fused into the initial fusion feature of the sample line segment, and the attention graph corresponding to the sample line segment is determined according to the initial fusion feature of the sample line segment;

[0124] S506: According to the attention graph corresponding to each sample line segment, determine the predicted feature representation vector of each sample line segment through the graph neural network;

[0125] S508: According to the predicted feature representation vector, determine the predicted score matrix; each element in the predicted score matrix represents the similarity score of two sample line segments not belonging to the same sample image;

[0126] S510: Obtain the predicted assignment matrix based on the predicted score matrix; each element in the predicted assignment matrix represents the matching probability between the sample line segments in the first sample image and the second sample image, respectively;

[0127] S512: Determine the loss value according to the predicted assignment matrix, the label feature corresponding to the first sample image and the second sample image, respectively, and train the graph neural network according to the loss value until the training stopping condition is met, to obtain the trained graph neural network.

[0128] Specifically, assuming that the first sample image A contains M straight line segments, the second sample image B contains N straight line segments, an invalid straight line segment is added to the first sample image, and an invalid straight line segment is also added to the second sample image, then the straight line segment set of the first sample image contains M+1 elements, the straight line segment set of the second sample image contains N+1 elements, after the above steps, the obtained prediction assignment matrix is a (M+1) x (N+1) two-dimensional vector, after obtaining the prediction assignment matrix each time, the prediction assignment matrix is subjected to loss value calculation with the label information to obtain the loss value of the current network, the label information is as above, and can be a label matrix Based on this, the following loss function can be used to determine the loss value, Loss = -∑ (i,j)∈GT logP i,j -∑ i∈A logP i,N+1 -∑ j∈B logP M+1,j Wherein, Pi,j represents the i-th straight line segment in the first sample image and the j-th straight line segment in the second sample image matched with each other in the label matrix, Pi,N+1 represents the matching probability of the i-th straight line segment in the first sample image and the invalid straight line segment in the second sample image, and PM+1,j represents the matching probability of the j-th straight line segment in the second sample image and the invalid straight line segment in the first sample image.

[0129] The graph neural network is iteratively trained through the loss value, and when a training stop condition is met, the training is stopped, and the trained graph neural network is obtained.

[0130] Figure 6 is a flowchart of an image positioning method provided by an embodiment of the present application, referring to Figure 6 The method comprises the following steps.

[0131] S602: straight line segments in two images to be positioned are matched by using a straight line segment matching method provided by an embodiment of the present application, and the straight line segments matched with each other in the images to be positioned are determined.

[0132] Specifically, the straight line segments matched with each other can be represented in the form of vectors, or can be directly labeled in the two images to be positioned.

[0133] S604: the position of one of the images to be positioned in the other image to be positioned is determined according to the position information of the straight line segments matched with each other.

[0134] After the information of the matched straight line segments is obtained, the position of one image to be positioned in another image can be determined according to the positions of the matched straight line segments in the respective images. For example, in the process of moving the mobile camera device, the position of the mobile camera device in the map needs to be monitored in real time. At a certain moment, the current environment image of the device is obtained by the mobile camera device, and then the environment image and the map image need to be matched. Specifically, the matched straight line segments can be determined by the above straight line segment matching method provided by the embodiment of the present application, the matched environment or target object is determined based on the matched straight line segments, and then the position of the mobile camera device in the map is determined.

[0135] The position of one image in another image can be obtained by the above method. Since the appearance attribute and the geometric characteristics of the straight line segment are fully considered in the matching process, the matching accuracy and efficiency are effectively improved.

[0136] Based on the above method embodiment, the embodiment of the present application further provides an image straight line segment matching device, as shown in Figure 7 The device comprises:

[0137] The information acquisition module 702 is configured to acquire the descriptor information and the geometric feature information of each straight line segment in the first straight line segment set corresponding to the first image and the second straight line segment set corresponding to the second image. The geometric feature information includes at least one of the position, direction and length of the straight line segment in the first image and the second image.

[0138] The determination module 704 is configured to determine the matched straight line segments in the first straight line segment set and the second straight line segment set according to the descriptor information and the geometric feature information of each straight line segment.

[0139] The above image straight line segment matching device provided by the present application acquires the descriptor information and the geometric feature information of each straight line segment in the first straight line segment set corresponding to the first image and the second straight line segment set corresponding to the second image, and determines the matched straight line segments in the first straight line segment set and the second straight line segment set according to the descriptor information and the geometric feature information of each straight line segment. In the process of determining the matched straight line pairs, the present application not only considers the descriptor information, but also considers the geometric feature information of the straight line segment, thereby improving the matching efficiency and effectively improving the accuracy of the matching result.

[0140] The above process of determining the matched straight line segments in the first straight line segment set and the second straight line segment set according to the descriptor information and the geometric feature information of each straight line segment comprises: for each straight line segment, adding the geometric feature information corresponding to the straight line segment to the descriptor information corresponding to the straight line segment to obtain the initial fusion feature corresponding to the straight line segment; and determining the matched straight line segments in the first straight line segment set and the second straight line segment set according to the initial fusion feature.

[0141] The process of adding the geometric feature information corresponding to the straight line segment into the description sub-information corresponding to the straight line segment to obtain the initial fusion feature corresponding to the straight line segment includes: determining a geometric feature vector with the same dimension as the description sub-information of the straight line segment according to the geometric feature information of the straight line segment; and adding the geometric feature vector and the description sub-information of the straight line segment to obtain the initial fusion feature of the straight line segment.

[0142] The process of determining the straight line segments matched with each other in the first straight line segment set and the second straight line segment set according to the initial fusion feature includes: determining an attention graph corresponding to each straight line segment by using the initial fusion feature corresponding to the straight line segment and the graph attention neural network; and determining the straight line segments matched with each other in the first straight line segment set and the second straight line segment set according to the attention graph.

[0143] The process of determining the feature representation vector of each straight line segment according to the attention graph includes: calculating an attention coefficient of each edge in the attention graph; performing normalization processing on each attention coefficient to obtain a normalized attention coefficient of each edge; performing feature aggregation on the initial fusion feature of each straight line segment and the normalized attention coefficient corresponding to the edge of the straight line segment to obtain an aggregated feature of the straight line segment; updating the initial fusion feature by using the aggregated feature to obtain an updated fusion feature; and determining the feature representation vector of the straight line segment by using the updated fusion feature.

[0144] The process of determining the feature representation vector of each straight line segment according to the attention graph includes: determining a module feature representation vector of each straight line segment by using each attention module in the graph neural network; wherein the graph neural network includes at least two attention modules; and combining a plurality of module feature representation vectors corresponding to each straight line segment to obtain the feature representation vector of the straight line segment.

[0145] The process of determining the feature representation vector of each straight line segment according to the attention graph includes: processing the attention graph by using a current attention network layer to obtain a network layer feature of each straight line segment; determining whether the current attention network layer is the last layer of the graph neural network, and if so, determining the network layer feature as the feature representation vector of the straight line segment; wherein the graph neural network includes at least two attention network layers, and each attention network layer includes at least one attention module; and if not, updating the attention graph by using the network layer feature, and continuing to process the updated attention graph by using a next attention network layer.

[0146] The graph neural network comprises a self-attention network layer and a cross-attention network layer, and the self-attention network layer and the cross-attention network layer are alternately connected; the attention graph input into the self-attention network layer is a self-attention graph, and the attention graph input into the cross-attention network layer is a cross-attention graph; wherein the edges in the self-attention graph are the connecting lines of any two straight line segments in the first straight line segment set or the connecting lines of any two straight line segments in the second straight line segment set; and the edges in the cross-attention graph are the connecting lines of any one straight line segment in the first straight line segment set and any one straight line segment in the second straight line segment set.

[0147] The graph neural network is obtained by the following method: obtaining the descriptor information and the geometric feature information of each sample straight line segment in a first sample straight line segment set corresponding to a first sample image and a second sample straight line segment set corresponding to a second sample image; for each sample straight line segment, the descriptor information and the geometric feature information of the sample straight line segment are fused into initial fusion features of the sample straight line segment, and an attention graph corresponding to the sample straight line segment is determined according to the initial fusion features of the sample straight line segment; the prediction feature representation vector of each sample straight line segment is determined through a graph neural initial network according to the attention graph corresponding to the sample straight line segment; a prediction score matrix is determined according to the prediction feature representation vector; each element in the prediction score matrix represents the similarity score of two sample straight line segments not belonging to the same sample image; a prediction assignment matrix is obtained by solving an optimization problem based on the prediction score matrix; each element in the prediction assignment matrix represents the matching probability between the sample straight line segments in the first sample image and the second sample image respectively; a loss value is determined according to the prediction assignment matrix, the label features corresponding to the first sample image and the second sample image respectively, and the graph neural initial network is trained according to the loss value until a training stop condition is met, and a trained graph neural network is obtained.

[0148] Based on the above method embodiment, the embodiment of the present application further provides an image positioning device, as shown in Figure 8 The device comprises:

[0149] The matching module 802 is configured to perform straight line segment matching on two to-be-positioned images by using any straight line segment matching method provided in the embodiments of the present application, and determine the straight line segments in the to-be-positioned images that match each other.

[0150] The positioning module 804 is configured to determine the position of one of the to-be-positioned images in the other to-be-positioned image according to the position information of the straight line segments that match each other.

[0151] Through the above device, the position of one image in another image can be obtained. Since the appearance attributes and geometric characteristics of the straight line segments are fully considered in the matching process, the matching accuracy and efficiency are effectively improved.

[0152] The image line segment matching device and the image positioning device provided by the embodiments of the present application have the same implementation principle, technical effects and the above-mentioned method embodiments. For brief description, the above-mentioned device embodiments are not mentioned in the following, and the corresponding content can be referred to the above-mentioned image line segment matching method and image positioning method embodiments.

[0153] The embodiments of the present application further provide an electronic device, as shown in the accompanying drawings, which is a structural schematic diagram of the electronic device. The electronic device comprises a processor 901 and a memory 902. The memory 902 stores computer executable instructions which can be executed by the processor 901. The processor 901 executes the computer executable instructions to implement the above-mentioned image line segment matching method and image positioning method. Figure 9

[0154] In the embodiment shown in the accompanying drawings, the electronic device further comprises a bus 903 and a communication interface 904. The processor 901, the communication interface 904 and the memory 902 are connected through the bus 903. Figure 9

[0155] The memory 902 can include a high-speed random access memory (RAM) and can further include a non-volatile memory, for example, at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 904 (which can be wired or wireless). The communication connection can use the Internet, a wide area network, a local area network, a metropolitan area network, etc. The bus 903 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 903 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one bidirectional arrow is used in the accompanying drawings to represent the bus, but it does not mean that there is only one bus or only one type of bus. Figure 9

[0156] ​​​The processor 801 can be an integrated circuit chip having a processing capability of signals. In the implementation process, the steps of the above method can be completed by the integrated logic circuit of hardware in the processor 901 or the instructions in the form of software. The processor 901 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the storage, and the processor 901 reads the information in the storage, and combines the hardware to complete the steps of the straight line segment matching method in the image and the image positioning method of the foregoing embodiments.

[0157] The embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to implement the straight line segment matching method in the image and the image positioning method. The specific implementation can be referred to the foregoing method embodiments, and will not be described here.

[0158] The computer program product of the straight line segment matching method in the image, the image positioning method and the electronic equipment provided by the embodiment of the present application includes a computer readable storage medium storing program codes. The instructions included in the program codes can be used to execute the method in the foregoing method embodiments. The specific implementation can be referred to the method embodiments, and will not be described here.

[0159] Unless otherwise specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0160] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0161] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0162] Finally, it should be noted that: the above embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art within the technical range disclosed by the present application can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of matching line segments in an image, characterized by, The method comprises: obtaining the descriptor information and the geometric feature information of each line segment in a first set of line segments corresponding to a first image and a second set of line segments corresponding to a second image; wherein the geometric feature information comprises at least one of the position, direction and length of the line segment in the first image and the second image respectively; for each line segment, adding the geometric feature information corresponding to the line segment to the descriptor information corresponding to the line segment to obtain an initial fusion feature corresponding to the line segment; determining the line segments that match each other in the first set of line segments and the second set of line segments according to the initial fusion feature; wherein the step of adding the geometric feature information corresponding to the line segment to the descriptor information corresponding to the line segment to obtain an initial fusion feature corresponding to the line segment comprises: determining a geometric feature vector with the same dimension as the descriptor information of the line segment according to the geometric feature information of the line segment; adding the geometric feature vector and the descriptor information of the line segment to obtain the initial fusion feature of the line segment; the step of determining the line segments that match each other in the first set of line segments and the second set of line segments according to the initial fusion feature comprises: determining an attention graph corresponding to each line segment according to the initial fusion feature corresponding to the line segment and a graph attention neural network; determining the line segments that match each other in the first set of line segments and the second set of line segments according to the attention graph.

2. The method of claim 1, wherein, The step of determining the line segments that match each other in the first set of line segments and the second set of line segments according to the attention graph comprises: determining a feature representation vector of each line segment according to the attention graph; determining the matching probability of each line segment in the first set of line segments and each line segment in the second set of line segments according to the feature representation vector; determining the line segments that match each other in the first set of line segments and the second set of line segments according to the matching probability.

3. The method of claim 2, wherein, The step of determining the matching probability of each line segment in the first set of line segments and each line segment in the second set of line segments according to the feature representation vector comprises: determining a similarity score between each line segment in the first set of line segments and each line segment in the second set of line segments through the feature representation vector, and determining a score matrix according to the similarity score; determining a assignment matrix corresponding to an optimal solution based on the score matrix; each element in the assignment matrix is used to represent the matching probability of each line segment in the first set of line segments and each line segment in the second set of line segments.

4. The method of claim 2, wherein, The step of determining a feature representation vector of each line segment according to the attention graph comprises: calculating an attention coefficient of each edge in the attention graph; normalizing each attention coefficient to obtain a normalized attention coefficient of the edge; performing feature aggregation on the initial fusion feature of each line segment and the normalized attention coefficient of the edge corresponding to the line segment to obtain an aggregated feature of the line segment; The initial fusion feature is updated by the aggregation feature to obtain an updated fusion feature, and the updated fusion feature is used to determine a feature representation vector of the straight line segment.

5. The method of claim 2, wherein, The step of determining the feature representation vector of each straight line segment according to the attention graph comprises: Each module feature representation vector of each straight line segment is determined by each attention module in the graph neural network; wherein the graph neural network comprises at least two attention modules; The module feature representation vectors corresponding to each straight line segment are combined to obtain a feature representation vector of the straight line segment.

6. The method of claim 2, wherein, The step of determining the feature representation vector of each straight line segment according to the attention graph comprises: A network layer feature of each straight line segment is obtained by processing the attention graph through a current attention network layer; If the current attention network layer is the last layer of the graph neural network, the network layer feature is determined as the feature representation vector of the straight line segment; wherein the graph neural network comprises at least two attention network layers, and each attention network layer comprises at least one attention module; If not, the attention graph is updated by the network layer feature, and the updated attention graph is processed by the next attention network layer.

7. The method of claim 6, wherein, The graph neural network comprises a self-attention network layer and a cross-attention network layer, and the self-attention network layer and the cross-attention network layer are connected alternately; The input of the attention graph in the self-attention network layer is a self-attention graph, and the input of the attention graph in the cross-attention network layer is a cross-attention graph; Wherein, the edge in the self-attention graph is the connection line of any two straight line segments in the first straight line segment set or the connection line of any two straight line segments in the second straight line segment set; the edge in the cross-attention graph is the connection line of any one straight line segment in the first straight line segment set and any one straight line segment in the second straight line segment set.

8. The method according to any one of claims 5-7, characterized in that, The graph neural network is obtained by the following method: The descriptor information and the geometric feature information of each sample straight line segment in the first sample straight line segment set corresponding to the first sample image and the second sample straight line segment set corresponding to the second sample image are obtained; For each sample straight line segment, the descriptor information and the geometric feature information of the sample straight line segment are fused into the initial fusion feature of the sample straight line segment, and the attention graph corresponding to the sample straight line segment is determined according to the initial fusion feature of the sample straight line segment; According to the attention graph corresponding to each sample straight line segment, a predicted feature representation vector of each sample straight line segment is determined by a graph neural network; According to the predicted feature representation vector, a predicted score matrix is determined; each element in the predicted score matrix represents the similarity score of the sample straight line segments not belonging to the same sample image; Based on the predicted score matrix, an optimal problem is solved to obtain a predicted assignment matrix; each element in the predicted assignment matrix represents the matching probability between the sample straight line segments in the first sample image and the second sample image, respectively. A loss value is determined according to the predicted assignment matrix, and label features corresponding to the first sample image and the second sample image, respectively, and the graph neural network is trained according to the loss value until a training stop condition is met, to obtain the trained graph neural network.

9. An image positioning method characterized by, The method comprises: performing line segment matching on two to-be-positioned images by the method of any one of claims 1-8 to determine mutually matched line segments in the to-be-positioned images; determining the position of one of the to-be-positioned images in the other to-be-positioned image according to position information of the mutually matched line segments.

10. An electronic device, comprising: A processor and a memory are included, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method of any one of claims 1-8, or the processor executes the computer executable instructions to implement the method of claim 9.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions cause the processor to implement the method of any one of claims 1-8, or the computer executable instructions cause the processor to implement the method of claim 9.

12. A computer program product, characterised in that, The computer program product comprises a computer program, and when the computer program is executed by the processor, the computer program implements the method of any one of claims 1-8, or when the computer program is executed by the processor, the computer program implements the method of claim 9.

Images