A UAV visual relocalization method based on object semantics

Abstract

The present invention provides a method for visual relocation of unmanned aerial vehicles based on object semantics, including: constructing a visual relocation model of unmanned aerial vehicles: an image processing and representation module, a scene matching module and a relocation module. The current scene is photographed by a camera of the unmanned aerial vehicle, and the semantic information of the object contained in the picture is obtained by using the YOLOv3 target detection network in the image processing module, and a topological map is constructed using the semantic information as a sparse description of the environment; in the case of prior knowledge of the map, an accompanying map of the picture to be matched is constructed, and a random walk algorithm is used to match the semantic features and the scene; the EPnP algorithm is used to solve the position and posture of the unmanned aerial vehicle to complete the relocation. The present invention can realize real-time relocation of unmanned aerial vehicles with strong robustness when the scene lighting conditions change dynamically, and is particularly suitable for automation tasks such as industrial inspections.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning, and in particular to a method for visual repositioning of an unmanned aerial vehicle based on object semantics. Background Art

[0002] Positioning is a critical issue in the field of drones. When performing automated tasks such as industrial inspections, drones need to have accurate self-positioning capabilities and be robust to common interference factors during the work process. With the development of computer vision, the visual scene matching technology based on camera sensors has become increasingly mature and can provide high-quality services for autonomous flight of drones. Visual sensors have the advantages of low cost, small size, light weight, the ability to provide rich online environmental information, and non-contact. A suitable visual positioning system can be combined with GPS and INS systems as a supplementary positioning system for drones.

[0003] The drone needs to keep track of its position during operation. When the position tracking fails, it needs to be relocated to restore the current position. Traditional visual positioning algorithms rely on the similarity between environmental representations, use artificially designed underlying geometric visual features as the basis for calculating image similarity, and then use feature matching to achieve scene matching. When the environmental conditions remain unchanged, the drone relocation can achieve good results. However, in the real world, there are often complex environmental changes such as changes in lighting conditions, weather changes, and seasons, which cause some key features in the image to be strengthened or weakened, greatly reducing the accuracy of feature matching. At this time, the relocation accuracy of this traditional method will drop significantly, and expensive map maintenance costs are required to maintain performance. Summary of the invention

[0004] Purpose of the invention: The present invention provides a UAV visual relocalization method based on object semantics to solve the technical problem that the existing UAV visual relocalization method cannot well cope with the lighting changes in the environment, resulting in insufficient relocalization accuracy.

[0005] The present invention comprises: step 1, constructing a UAV visual relocation model, wherein the UAV visual relocation model comprises an image processing and representation module, a scene matching module and a relocation module;

[0006] Step 2, constructing a semantic topology map through image processing and representation modules;

[0007] Step 3, the scene matching module constructs a companion graph based on the common semantic information of the image pair to be matched, comprehensively considers the semantic information differences, node position differences and topological structure differences of the image to be matched, uses a random walk algorithm to complete the mapping of semantic feature point pairs, and calculates the similarity between the camera sensor input image and the image in the map library, thereby completing the scene matching of the drone;

[0008] Step 4: Use the relocation module to solve the position and posture of the drone and complete the relocation.

[0009] Step 2 includes: taking a picture of the current scene by a drone camera, the image processing and representation module uses the YOLOv3 target detection network to obtain the object semantic information contained in the scene picture (including the object category label and the pixel position information of the object in the image), the object semantic information includes the object semantic label and the pixel position information of the object in the image, constructing a semantic topology map with the object semantic label as a node, and establishing a scene map library as an internal representation of the environment for comparison with the input visual data.

[0010] Step 3 includes: using the weight matrix W of the semantic topology map to measure the matching degree between the candidate pairs, where the elements W on the diagonal of the semantic topology map are ia；ia Represents the matching weight between nodes, and the off-diagonal elements W ia；jb Represents the matching weight between edges. The larger the weight, the higher the matching degree. The probability transfer matrix P = D -1 W,D is a A diagonal matrix, where D ii are the diagonal elements of D, W ij Represents the element in the i-th row and j-th column of the weight matrix W;

[0011] Set the two images to be matched as G M and G N ,picture G M and G N The accompanying diagram of It's G M The nodes in yes The edge between It's G N The nodes in yes The edge between

[0012] Let X (t)T is the probability distribution of the random walker reaching each node on the companion graph at time t, X (t+1)T is the probability distribution of the random walker reaching each node on the companion graph at time t+1, and the probability transfer matrix is ​​P. Then the Markov chain of the random walk is expressed as:

[0013] X (t+1)T =X (t)T P

[0014] Perform a random walk on the companion graph until the probability distribution of the random walker reaching each node on the companion graph converges, and obtain the assignment matrix X', thereby determining the two graphs G to be matchedM and G N The matching relationship of the midpoint.

[0015] In step 3, the semantic information differences, node position differences, and topological structure differences of the graphs to be matched are comprehensively considered in the following way:

[0016] Considering semantic information differences: a pair of nodes with the same semantic label and Will form the accompanying graph New Node A pair of nodes with different semantic labels is defined as a conflicting matching pair and will not appear in the companion graph; when calculating image similarity, the movement frequency of objects in the scene is taken into account, and objects with low movement frequency are given larger weight values ​​during calculation.

[0017] Consider the difference in node positions: remember dis ia For Node With Node The Euclidean distance between ia The smaller, With Node The higher the matching degree, the higher the ij For Node With Node The Euclidean distance between ij For Node With Node The Euclidean distance between ij with dis ab The smaller the difference, and The higher the matching degree;

[0018] The weight matrix and probability transfer matrix of non-conflicting matching pairs are initialized using the following formula:

[0019]

[0020]

[0021] Among them, when i≠a and j≠b, the element W of the weight matrix ia；jb represent and The matching weight of the weight matrix is ​​W when i = a and j = b. ia；jb represent and The matching weight of is the adjustment factor;

[0022] Consider topological differences: according to the pre-set step size and accompanying graph The number of times each node is used as a departure node in the figure Perform a random walk on .

[0023] In step 3, the similarity s between the camera sensor input image and the image in the map library is calculated using the following formula:

[0024]

[0025] Among them, m i Accompanying graph represents the normalized weight of the ith matching pair, n i The value range of n is (0,1), i represents the weight of the i-th unmatched object in the camera sensor input image and the image in the map library, m i The value of is determined by random walk, n i It is determined according to the movement frequency of the objects in the scene, Δ i is the object pixel position deviation between the camera sensor input image and the i-th matching pair in the image in the map library, q is the number of matching pairs, and d is an adjustment factor determined by the number of objects in the image and the acquisition frequency of the camera when building the map library; the image in the map library with the highest similarity to the input image is taken as the result of scene matching.

[0026] Step 4 includes:

[0027] The drone camera is set as a pinhole model and the internal parameters are known. The image currently captured by the drone is set to G1. After scene matching, it corresponds to the image G2 in the map library. There are n (n≥4) semantic feature matching pairs. The 2D position of the semantic feature points in G1 in the camera coordinate system is known, and the 3D position of the semantic feature points in G2 in the world coordinate system is known. Using the EPnP algorithm, the n 3D points are represented as the weighted sum of four non-coplanar virtual control points, and then the coordinates of the four virtual control points in the camera coordinate system are calculated to restore the drone's pose R, t, where R is the rotation matrix containing the pose information, and t is the translation vector containing the position information; let the world coordinates of the four virtual control points be The matrix C is:

[0028]

[0029] In step 4, let the 3D coordinates of the semantic feature points and the four virtual control points in G2 in the world coordinate system and They are:

[0030]

[0031]

[0032] Coordinates of semantic feature points in the world coordinate system Represented as coordinates of 4 virtual control points The weighted sum of:

[0033]

[0034] Among them, α ij is the weighting factor.

[0035] In step 4, let represents the coordinates of the jth virtual control point in the camera coordinate system, and the coordinates of the i-th semantic feature point in G1 in the camera coordinate system are The relationship between them is as follows:

[0036]

[0037]

[0038] In step 4, after obtaining the 3D coordinates of the control points in the camera coordinate system, the world coordinate centroids of the semantic feature points are calculated separately. and the camera coordinate center Get matrix A, matrix B and matrix H:

[0039]

[0040]

[0041]

[0042]

[0043] H=B T A.

[0044] In step 4, SVD singular value decomposition is performed on H to recover the drone's position R and t:

[0045] H=U∑V T ,

[0046] R=UV T ,

[0047]

[0048] Beneficial effects: The present invention uses YOLOv3 with the ability to segment the foreground and background to complete target detection, and can obtain the semantic information of objects in the scene image. Compared with geometric visual features, this object-level semantics belongs to high-level features, has good sparsity and has strong anti-interference ability for changes in illumination in the environment, and can ensure the robustness of drone relocation. The present invention abstracts the image into a semantic topological graph, and indirectly completes the calculation of image similarity by comparing its structure, which simplifies the comparison process of environmental information and can meet the needs of real-time visual relocation of drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0050] Figure 1 It is a schematic diagram of the UAV visual relocalization model proposed by the present invention;

[0051] Figure 2 The present invention is based on Figure 1 The workflow of the object semantics-based UAV visual relocalization method proposed by the relocalization model;

[0052] Figure 3 This is the image similarity calculation process described in the present invention. DETAILED DESCRIPTION

[0053] Figure 1 The figure shows the UAV visual relocalization model proposed in the present invention, which includes an image processing and representation module, a scene matching module and a relocalization module.

[0054] This embodiment provides a method for visual relocalization of a drone based on object semantics. The execution process of the method is as follows: Figure 2 As shown, the following steps are included:

[0055] Step 1, constructing a UAV visual relocalization model, wherein the UAV visual relocalization model includes an image processing and representation module, a scene matching module and a relocalization module;

[0056] Step 2, constructing a semantic topology map through image processing and representation modules;

[0057] Step 3, the scene matching module constructs a companion graph based on the common semantic information of the image pair to be matched, comprehensively considers the semantic information differences, node position differences and topological structure differences of the image to be matched, uses a random walk algorithm to complete the mapping of semantic feature point pairs, and calculates the similarity between the camera sensor input image and the image in the map library, thereby completing the scene matching of the drone;

[0058] Step 4: Use the relocation module to solve the position and posture of the drone and complete the relocation.

[0059] Step 2 includes: taking a picture of the current scene by a drone camera, the image processing and representation module uses the YOLOv3 target detection network to obtain the object semantic information contained in the scene picture, the object semantic information includes the object semantic label and the pixel position information of the object in the image, and constructs a semantic topology map with the object semantic label as a node, and establishes a scene map library as an internal representation of the environment for comparison with the input visual data.

[0060] Step 3 includes: using the weight matrix W of the semantic topology map to measure the matching degree between the candidate pairs, where the elements W on the diagonal of the semantic topology map are ia；ia Represents the matching weight between nodes, and the off-diagonal elements W ia；jb Represents the matching weight between edges. The larger the weight, the higher the matching degree. The probability transfer matrix P = D -1 W,D is a A diagonal matrix, where D ii are the diagonal elements of D, W ij Represents the element in the i-th row and j-th column of the weight matrix W;

[0061] Set the two images to be matched as G M and G N ,picture G M and G N The accompanying diagram of It's G M The nodes in yes The edge between It's G N The nodes in yes The edge between

[0062] Let X (t)T is the probability distribution of the random walker reaching each node on the companion graph at time t, X (t+1)T is the probability distribution of the random walker reaching each node on the companion graph at time t+1, and the probability transfer matrix is ​​P. Then the Markov chain of the random walk is expressed as:

[0063] X (t+1)T =X (t)T P

[0064] Perform a random walk on the companion graph until the probability distribution of the random walker reaching each node on the companion graph converges, and obtain the assignment matrix X', thereby determining the two graphs G to be matched M and G N The matching relationship of the midpoint.

[0065] In step 3, the semantic information differences, node position differences, and topological structure differences of the graphs to be matched are comprehensively considered in the following way:

[0066] Considering semantic information differences: a pair of nodes with the same semantic label and Will form the accompanying graph New Node A pair of nodes with different semantic labels is defined as a conflicting matching pair and will not appear in the companion graph;

[0067] Consider the difference in node positions: remember dis ia For Node With Node The Euclidean distance between ia The smaller, With Node The higher the matching degree, the higher the ij For Node With Node The Euclidean distance between ij For Node With Node The Euclidean distance between ij with dis ab The smaller the difference, and The higher the matching degree;

[0068] The weight matrix and probability transfer matrix of non-conflicting matching pairs are initialized using the following formula:

[0069]

[0070]

[0071] Among them, when i≠a and j≠b, the element W of the weight matrix ia；jb represent and The matching weight of the weight matrix is ​​W when i = a and j = b. ia；jb represent and The matching weight of is the adjustment factor;

[0072] Consider topological differences: according to the pre-set step size and accompanying graph The number of times each node is used as a departure node in the figure Perform a random walk on .

[0073] In step 3, the similarity s between the camera sensor input image and the image in the map library is calculated using the following formula:

[0074]

[0075] Among them, m i Accompanying graph represents the normalized weight of the ith matching pair, n i The value range of n is (0,1), i represents the weight of the i-th unmatched object in the camera sensor input image and the image in the map library, m i The value of is determined by random walk, n i It is determined according to the movement frequency of the objects in the scene, Δ i is the object pixel position deviation between the camera sensor input image and the i-th matching pair in the image in the map library, q is the number of matching pairs, and d is an adjustment factor determined by the number of objects in the image and the acquisition frequency of the camera when building the map library; the image in the map library with the highest similarity to the input image is taken as the result of scene matching.

[0076] Step 4 includes:

[0077] The drone camera is set as a pinhole model and the internal parameters are known. The image currently captured by the drone is G1. After scene matching, it corresponds to the image G2 in the map library. There are n semantic feature matching pairs. The 2D position of the semantic feature points in G1 in the camera coordinate system is known, and the 3D position of the semantic feature points in G2 in the world coordinate system is known. Using the EPnP algorithm, the n 3D points are represented as the weighted sum of 4 non-coplanar virtual control points, and then the coordinates of the 4 virtual control points in the camera coordinate system are calculated to restore the drone's posture R and t. R is the rotation matrix containing the posture information, and t is the translation vector containing the position information. The 4 virtual control points in the world coordinate system can make the matrix C reversible. Let the world coordinates of the 4 virtual control points be The matrix C is:

[0078]

[0079] In step 4, let the 3D coordinates of the semantic feature points and the four virtual control points in G2 in the world coordinate system and They are:

[0080]

[0081]

[0082] for Specific coordinates in the world coordinate system;

[0083] for Specific coordinates in the world coordinate system;

[0084] Coordinates of semantic feature points in the world coordinate system Represented as coordinates of 4 virtual control points The weighted sum of:

[0085]

[0086] Among them, α ij is the weighting factor.

[0087] In step 4, let represents the coordinates of the jth virtual control point in the camera coordinate system, are the world coordinates of the four virtual control points, and the coordinates of the i-th semantic feature point in G1 in the camera coordinate system are The relationship between them is as follows:

[0088]

[0089]

[0090] In step 4, after obtaining the 3D coordinates of the control points in the camera coordinate system, the world coordinate centroids of the semantic feature points are calculated separately. and the camera coordinate center Get matrix A, matrix B and matrix H:

[0091]

[0092]

[0093]

[0094]

[0095] H=B T A.

[0096] In step 4, perform SVD (Singular Value Decomposition) on H to recover the drone's position R and t:

[0097] H=U∑V T ,

[0098] R=UV T ,

[0099]

[0100] The drone used in the present invention is DJI M200, and its specific parameters are shown in Table 1 below:

[0101] Table 1

[0102]

[0103] The present invention implements a target detector based on YOLOv3 under the Ubuntu 18.04 system, and uses a drone to shoot 800 pictures at a rate of 10fps in an indoor scene to build a scene map library. The optimal image similarity threshold of the object semantics-based relocation method proposed in the present invention is 0.375. When the scene lighting conditions remain unchanged, its accuracy and recall rates can reach 99.23% and 97.23%; when the scene lighting conditions change, its accuracy and recall rates can reach 93.18% and 87.23%, which is 52.44% higher than the visual relocation method based on the bag-of-words model, and the recall rate is 54.88% higher than the visual relocation method based on the bag-of-words model. Under the same scene map library, the average running time required for the method proposed in the present invention to complete a single scene match is 0.027s faster than that of the bag-of-words model algorithm. When the scene lighting conditions change dynamically, the method proposed in the present invention can achieve strong robustness in real-time relocation of drones, providing a guarantee for drones to perform tasks.

[0104] The present invention provides a method for visual relocation of drones based on object semantics. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A UAV visual relocalization method based on object semantics, characterized in that: The following steps are involved: Step 1, constructing a UAV visual relocalization model, wherein the UAV visual relocalization model includes an image processing and representation module, a scene matching module and a relocalization module; Step 2, constructing a semantic topology map through image processing and representation modules; Step 3, the scene matching module constructs a companion graph based on the common semantic information of the image pair to be matched, and completes the scene matching of the drone; Step 4: Solve the position and posture of the drone through the relocation module to complete the relocation; Step 3 includes: using the weight matrix W of the semantic topology map to measure the matching degree between the candidate pairs, where the elements W on the diagonal of the semantic topology map are ia；ia Represents the matching weight between nodes, and the off-diagonal elements W ia；jb Represents the matching weight between edges; probability transfer matrix P = D -1 W,D is a A diagonal matrix, where D ii are the diagonal elements of D, W ij Represents the element in the i-th row and j-th column of the weight matrix W; Set the two images to be matched as G M and G N ,picture G M and G N The accompanying diagram of It's G M The nodes in yes The edge between It's G N The nodes in yes The edge between Let X (t)T is the probability distribution of the random walker reaching each node on the companion graph at time t, X (t+1)T is the probability distribution of the random walker reaching each node on the companion graph at time t+1, and the probability transfer matrix is ​​P. Then the Markov chain of the random walk is expressed as: X (t+1)T =X (t)T P Perform a random walk on the companion graph until the probability distribution of the random walker reaching each node on the companion graph converges, and obtain the assignment matrix X', thereby determining the two graphs G to be matched M and G N The matching relationship of the midpoints; In step 3, the semantic information differences, node position differences, and topological structure differences of the graphs to be matched are comprehensively considered in the following way: Considering semantic information differences: a pair of nodes with the same semantic label and Will form the accompanying graph New Node A pair of nodes with different semantic labels is defined as a conflicting matching pair and will not appear in the companion graph; Consider the difference in node positions: remember dis ia For Node With Node The Euclidean distance between ia The smaller, With Node The higher the matching degree, the higher the ij For Node With Node The Euclidean distance between ij For Node With Node The Euclidean distance between ij with dis ab The smaller the difference, and The higher the matching degree; The weight matrix and probability transfer matrix of non-conflicting matching pairs are initialized using the following formula: Among them, when i≠a and j≠b, the element W of the weight matrix ia；jb represent and The matching weight of the weight matrix is ​​W when i = a and j = b. ia；jb represent and The matching weight of is the adjustment factor; Consider topological differences: according to the pre-set step size and accompanying graph The number of times each node is used as a departure node in the figure Perform a random walk on In step 3, the similarity s between the camera sensor input image and the image in the map library is calculated using the following formula: Among them, m i Accompanying graph represents the normalized weight of the ith matching pair, n i The value range of n is (0,1), i represents the weight of the i-th unmatched object in the camera sensor input image and the image in the map library, m i The value of is determined by random walk, n i It is determined according to the movement frequency of the objects in the scene, V i is the object pixel position deviation between the camera sensor input image and the i-th matching pair in the image in the map library, q is the number of matching pairs, and d is an adjustment factor determined by the number of objects in the image and the acquisition frequency of the camera when building the map library; the image in the map library with the highest similarity to the input image is taken as the result of scene matching.

2. The method according to claim 1, characterized in that Step 2 includes: taking a picture of the current scene by a drone camera, the image processing and representation module uses the YOLOv3 target detection network to obtain the object semantic information contained in the scene picture, the object semantic information includes the object semantic label and the pixel position information of the object in the image, and constructs a semantic topology map with the object semantic label as a node, and establishes a scene map library as an internal representation of the environment for comparison with the input visual data.

3. The method according to claim 2, characterized in that Step 4 includes: The drone camera is set as a pinhole model and the internal parameters are known. The image currently captured by the drone is set to G1. After scene matching, it corresponds to the image G2 in the map library. There are n semantic feature matching pairs. The 2D position of the semantic feature points in G1 in the camera coordinate system is known, and the 3D position of the semantic feature points in G2 in the world coordinate system is known. Using the EPnP algorithm, the n 3D points are represented as the weighted sum of four non-coplanar virtual control points, and then the coordinates of the four virtual control points in the camera coordinate system are calculated to restore the drone's posture R and t. R is the rotation matrix containing the posture information, and t is the translation vector containing the position information. Let the world coordinates of the four virtual control points be The matrix C is:

4. The method according to claim 3, characterized in that In step 4, let the 3D coordinates of the semantic feature points and the four virtual control points in G2 in the world coordinate system and They are: Coordinates of semantic feature points in the world coordinate system Represented as coordinates of 4 virtual control points The weighted sum of: Among them, α ij is the weighting factor.

5. The method according to claim 4, characterized in that In step 4, let represents the coordinates of the jth virtual control point in the camera coordinate system, and the coordinates of the i-th semantic feature point in G1 in the camera coordinate system are The relationship between them is as follows:

6. The method according to claim 5, characterized in that In step 4, after obtaining the 3D coordinates of the control points in the camera coordinate system, the world coordinate centroid of the semantic feature points is calculated respectively. and the camera coordinate center Get matrix A, matrix B and matrix H: H=B T A。 7. The method according to claim 6, characterized in that In step 4, SVD singular value decomposition is performed on H to recover the drone's position R and t: H=U∑V T , R=UV T ,

Images