A panoramic segmentation-based target positioning method

Abstract

The application provides a target positioning method based on panoramic segmentation, and belongs to the field of target positioning. Firstly, an image acquisition sensor is used to acquire a real scene image of a target area, and an initial pose of the sensor is acquired; a rendering image is generated according to the initial pose of the sensor and a three-dimensional geographical space model which has been established; then, panoramic segmentation is performed on the real scene image and the rendering image, corner point detection and matching are performed on the segmentation results, and matched point pairs of the two images are obtained; the real pose of the sensor is calculated by using the matched point pairs and the initial pose of the sensor; finally, the real pose of the sensor is used to realize target positioning in a real scene. Compared with the matching method in the prior art which uses a deep learning method, the application does not need a large number of samples, uses a corner point detection method to determine the matched point pairs of the two images, improves the work efficiency, improves the matching precision between the heterogeneous images, and uses the real pose of the sensor to realize target positioning, thereby improving the target positioning precision.

Description

TECHNICAL FIELD

[0001] The application relates to a target positioning method based on panoramic segmentation and belongs to the field of target positioning. BACKGROUND

[0002] With the rapid development of technologies such as the Internet of Things and geographic space science, a digital wave is sweeping the world. As an intersection field of geographic space science and artificial intelligence, geographic space intelligence shows great vitality and potential. Thanks to the enhancement of software comprehensive computing power, the continuous progress of data and model sharing, and the improvement of the performance and cost reduction of various sensors, geographic space intelligent perception has attracted more and more research investment. At present, rich research results have been achieved in three-dimensional reconstruction of geographic space, which can well express static scenes. However, the world is dynamic. Only by dynamically perceiving and understanding dynamic geographic space can better services be provided for related applications such as autonomous driving, augmented reality and digital twin. To realize geographic space intelligent perception, it is an unavoidable core link to perceive and locate dynamic targets in complex geographic space.

[0003] In the process of perceiving and locating dynamic targets, a real three-dimensional model with a geographic reference coordinate needs to be established in the target area in advance. Through accurate positioning of a sensor carrier (such as a drone or an unmanned vehicle), the perception and location of targets in the scene can be completed, and therefore accurate registration of the sensor and the three-dimensional model is required. The matching method of the sensor and the three-dimensional model mainly includes two categories: one is a geographic registration method based on a pose sensor, such as a simple registration method for small scenes by using artificial layout or based on SLAM (Simultaneous Localization and Mapping). This method is efficient and can adapt to various environments, but its registration accuracy is insufficient. If it is used as an a priori pose to locate other targets in the scene, a larger error will be accumulated. The other is to introduce some other data for matching, for example, through two-dimensional images such as RGB images or ground camera images in the target area and the obtained real three-dimensional model, the features of the two-dimensional images and the three-dimensional model are matched, the relationship between the two-dimensional images and the three-dimensional model is established through this two-dimensional-three-dimensional matching method, and the accurate positioning of the target is realized. In addition, the accuracy of the introduced other data will directly affect the matching accuracy.

[0004] Meanwhile, in the two-dimensional-three-dimensional matching process, the heterogenous images are also affected. Due to the imaging mechanism and real-time differences and other factors, certain differences will be brought to the features of the heterogenous images. In the traditional matching method, the feature-based matching method extracts the local features in a certain neighborhood as the descriptor for matching, among which the most famous is the scale invariant feature transformation (SIFT) descriptor and some improved SIFT algorithms. Most of these methods cannot provide stable features, so it is difficult to achieve matching effect in the heterogenous matching work. The template-based matching method can obtain more invariant features in a larger range, but the expansion ability and matching efficiency limit the expansibility of this method, and the generalization ability is insufficient. In recent years, with the rapid development of deep learning methods in computer vision tasks, more and more deep learning matching algorithms have been proposed. For example, an attention aggregation mechanism is introduced in SuperGlue, which can jointly infer the underlying 3D scene and features, and advanced matching results have been achieved. DFM uses existing network to learn the deepest invariance, and combines semantic features to realize image matching. On the basis of introducing D2-Net feature extraction, a heterogenous remote sensing image matching network CMM-Net using high-level semantic local features is proposed. However, this kind of deep learning-based matching method has high requirements for samples and needs a lot of learning. When facing different quality and different texture characteristics of image pairs, the matching results may have large differences, which cannot guarantee the matching accuracy of the heterogenous image pair, and then directly affects the dynamic target perception and positioning accuracy. SUMMARY

[0005] The purpose of the present application is to provide a target positioning method based on panoramic segmentation to solve the problem of low target positioning accuracy caused by the inability of the prior art to guarantee the matching accuracy of heterogenous images.

[0006] The present application provides a target positioning method based on panoramic segmentation, which comprises the following steps:

[0007] 1) acquiring a real scene image of a target area by using an image acquisition sensor, and acquiring an initial pose of the sensor; rendering a three-dimensional model of the geographical space of the target area which has been established by using the initial pose of the sensor, to obtain a rendered image of the target area under the initial pose;

[0008] 2) performing panoramic segmentation on the rendered image and the real scene image of the target area respectively to obtain the segmentation results of the rendered image and the real scene image; performing corner point detection on the segmentation results of the rendered image and the real scene image respectively to obtain candidate corner points in each image, and determining the key points of the corresponding image according to the candidate corner points;

[0009] 3) constructing a descriptor by using the key points of the two images, and determining the matching point pairs of the two images according to the matching degree of the descriptors in the two images;

[0010] 4) calculating the real pose of the image acquisition sensor according to the matching point pairs and the initial pose of the image acquisition sensor;

[0011] 5) realizing the target positioning in the real scene by using the real pose of the sensor.

[0012] According to the initial pose of the sensor, the application generates a rendered image of the target area, and matches the rendered image with the real scene image of the target area. Considering the feature difference between the two images, the application performs panoramic segmentation on the real scene image and the rendered image, detects the key points based on the segmentation results, and determines the matching point pairs of the two images according to the descriptors of the key points. Compared with the matching method in the prior art, the application does not need a large number of samples, does not need to consider the sample quality, and does not need to perform sample learning. Instead, the application directly finds the matching point pairs of the two images according to the rendered image under the sensor pose and the corresponding real scene image by using the corner detection method, improves the work efficiency, and can still realize the accurate registration of the real scene image and the rendered image when there are different quality and different textures between the matched images. In this way, the pose transformation relationship between the two images is obtained, and the real pose of the sensor is determined. The target positioning is performed by using the real pose of the sensor, and the positioning accuracy of the target is effectively improved.

[0013] Further, the determination process of the key points is as follows: traversing the pixel points on the target contour line in the image segmentation result, establishing a square frame with the pixel point as the center point, the square frame intersects the target contour line at two intersection points, calculating the included angle between the two intersection points and the center point, and taking the pixel points corresponding to the included angle within the set angle threshold value as the candidate corner points, and taking the candidate corner points as the key points.

[0014] Further, the determination process of the key points is as follows: traversing the pixel points on the target contour line in the image segmentation result, establishing a square frame with the pixel point as the center point, the square frame intersects the target contour line at two intersection points, calculating the included angle between the two intersection points and the center point, and taking the pixel points corresponding to the included angle within the set angle threshold value as the candidate corner points, and taking the candidate corner points as the key points.

[0015] The present application considers that the edge contour line after panoramic segmentation is pixel-level accuracy, and the obtained edge may have a large number of fine sawtooth segments, and is not a regular edge curve, in order to avoid the case that a large number of false corner points are detected by using a traditional corner point detection algorithm, and to avoid large errors and additional workload in subsequent descriptor establishment and matching process, the present application adopts an angle threshold-based corner point detection method, a square frame is constructed with a pixel point on the contour line as a center point, to improve the accuracy and efficiency of corner point detection; at the same time, for a pixel point, one square frame can be established, and two square frames can be established to further improve the detection accuracy, and two square frames are used to determine the candidate corner points, which can reduce the number of candidate corner points determined by one square frame, and also reduces the workload in subsequent matching.

[0016] Further, when there are multiple candidate corner points in the set corner point region, the candidate corner points with the smallest angle difference of the included angles are selected as the key points in the set corner point region.

[0017] In order to reduce the workload of subsequent descriptor construction and the workload of subsequent point matching, for multiple key points in a certain region, the key points with the smallest angle difference of the included angles are selected as the optimal key points in the region, to further improve the work efficiency.

[0018] Further, a descriptor is constructed for the key point, and the descriptor includes a first descriptor and / or a second descriptor; wherein the first descriptor is the average value of the angles of the two included angles of the key point; and the second descriptor is the label value of each pixel point in the square frame in the smaller square frame established by the key point according to a set moving order.

[0019] Through the above process, the descriptor of the determined key point is constructed, and at least one descriptor is used as the basis for subsequent key point matching, when only one descriptor is selected, the calculation amount is small and the matching efficiency is fast; when two descriptors are selected, the feature description of the key point is more accurate, and the matching precision is higher.

[0020] Further, the matching degree of the key points is determined according to the quotient of the first descriptors of the key points in the two images, and the closer the quotient value is to 1, the higher the matching degree of the key points in the two images is.

[0021] The matching degree of the key points is determined according to the quotient of the first descriptors of the key points in the two images, and the closer the quotient value is to 1, the closer the included angles of the key points in the two images are, which proves that the key points in the two images are more likely to be corner points at the same position.

[0022] Further, the matching degree of the key points is determined according to the alignment degree of the second descriptors of the key points in the two images, wherein the alignment degree refers to the ratio of the number of the same labels at the same positions in the second descriptors of the two key points to the total number of labels, and the closer the ratio is to 1, the higher the matching degree of the key points in the two images is.

[0023] The matching degree of the key points is determined according to the label degree of the second descriptors of the key points in the two images, and it is determined whether the labels at the positions in the second descriptors of the two key points (i.e. the label values of each pixel point in the square box) are consistent, and the closer the ratio of the number of the consistent labels to the total number of labels (the number of pixels in the square box) is to 1, the more likely the two key points are the corner points at the same position.

[0024] Further, the formula for determining the matching degree of the key points in the two images is:

[0025] match = ω1match1 + ω2match2

[0026] In the formula, match is the matching degree of the key points in the two images, match1 is the matching degree of the key points in the two images determined by using the first descriptor, match2 is the matching degree of the key points in the two images determined by using the second descriptor, ω1 and ω2 are weight coefficients of match1 and match2 respectively, and ω1 + ω2 = 1.

[0027] According to the above formula, the matching degree of the key points in the real scene image and the rendered image can be determined, the matching point pairs of the two images can be determined according to the matching degree, and the weight coefficients of the two descriptors can be adaptively adjusted according to the importance of the different descriptors in the actual situation.

[0028] Further, the real pose of the sensor is calculated by using the epipolar geometry constraint method in the step 4).

[0029] The motion relationship between the real scene image and the rendered image can be quickly and accurately determined by using the epipolar geometry constraint method, and the real pose of the sensor can be obtained according to the initial pose of the sensor.

[0030] Further, the panoramic segmentation of the image is performed by using the MaskFormer network in the step 2).

[0031] The semantic segmentation and instance segmentation of the image can be simultaneously realized by using the MaskFormer network. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a specific flowchart of the target positioning based on panoramic segmentation of the present application;

[0033] Figure 2 is a matching flowchart of real scene image and rendering image of the present application;

[0034] Figure 3 is a diagram of epipolar geometry constraint principle. DETAILED DESCRIPTION

[0035] The specific embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0036] The present application proposes a target positioning method based on panoramic segmentation, and the specific flow is as shown in Figure 1 Firstly, the real scene image of the target area is obtained by using the image acquisition sensor, and the initial pose of the sensor is obtained; the rendering image is generated according to the initial pose of the sensor and the three-dimensional geographical space model which has been established; then the real scene image and the rendering image are subjected to panoramic segmentation, the corner point detection and matching of the segmentation result are performed, and the matching point pairs of the two images are obtained; the real pose of the sensor is calculated by using the matching point pairs and the initial pose of the sensor; finally, the target positioning in the real scene is realized by using the real pose of the sensor.

[0037] The specific flow of matching the rendering image and the real scene image is as shown in Figure 2 Firstly, the real scene image and the model rendering image are subjected to panoramic segmentation, the corner point detection of the target contour line in the segmentation result is performed, the detected corner points are screened, and the corner points meeting the set conditions are taken as key points; a descriptor is established for each key point, and the matching point pairs which finally match successfully are obtained according to the matching degree of the descriptor.

[0038] Embodiment 1:

[0039] Step 1. Data acquisition

[0040] In order to realize the target positioning in the target area, two kinds of images are needed in the present application, one is the real scene image in the target area, and the other is the rendering image under the corresponding sensor pose during image acquisition. The present application uses the image acquisition sensor installed on a fixed carrier (such as a vehicle or a drone) to obtain the real scene image of the target area. In this embodiment, the image acquisition sensor uses a depth camera, and the real scene depth image of the target area is obtained by using the depth camera for shooting. The initial pose of the sensor is obtained according to the GPS and IMU system carried on the sensor carrier and the pre-calibration parameters; the rendering image under the initial pose, i.e. the rendering image under the corresponding view angle, is generated according to the initial pose of the sensor and the three-dimensional geographical space model which has been established. The three-dimensional geographical model can be constructed according to aerial images or radar images.

[0041] Step 2. Panoramic segmentation

[0042] The present application respectively performs panoramic segmentation on the photographed real scene image and the rendered image to obtain the panoramic segmentation result of each image, as shown in Figure 2 The contour line of the target object can be obtained from the segmentation results of the two images. The present application uses MaskFormer as the panoramic segmentation network, which can uniformly implement instance segmentation and semantic segmentation of the image. Unlike the semantic segmentation task, instance segmentation needs to further distinguish different targets, for example, two cats in the scene. The task of semantic segmentation is to classify all pixels belonging to the cat, while instance segmentation needs to further distinguish which pixels belong to the first cat and which pixels belong to the second cat. MaskFormer introduces a bidirectional matching loss function proposed in DETR, which uses a Transformer as a decoder to calculate a set of pairs of points, each pair containing a class prediction and a mask embedding vector, which uniformly solves the semantic and instance level segmentation tasks. The model used by it uses Swin-L as the network skeleton, and achieves 52.7 PQ (Panoptic quality, panoramic segmentation evaluation index) on the COCO dataset, with good segmentation effect. The COCO training set used by the model contains 80 instance labels including people, bicycles and cars, and 52 semantic labels including buildings, roads and sky, which can meet most natural scenes. At the same time, in order to meet the real-time requirement, the present application removes redundant kernels in each deconvolution layer, slightly improves the segmentation time of each image while ensuring the accuracy is not reduced. For the panoramic segmentation result, the semantic segmentation result of the static area is used to match the real scene image and the rendered image, and the target instance segmentation result in the real scene image is used for target perception and positioning in the scene.

[0043] Step 3. Key point extraction

[0044] Since the contour line of the image is the pixel-level classification result obtained by panoramic segmentation, there may be a large number of fine sawtooth line segments in these contour lines, which are not regular edge curves. Therefore, using traditional corner detection algorithms may result in a large number of false corner points being detected, which will bring large errors to the subsequent descriptor establishment and matching process, and will also bring additional workload. In order to improve efficiency and accuracy, the present application uses an angle threshold-based corner detection method to detect the corners of the rendered image and the real scene image after panoramic segmentation and extract key points. The key point extraction process of the two images is consistent, and the following will take the segmented real scene image as an example to illustrate the specific process as follows:

[0045] First, traverse the pixel points on the target contour line in the real scene image segmentation result, as shown in Figure 2As shown, two square frames with different side lengths are established with the center point as the center, the two square frames intersect with the target contour line at two intersection points, the included angle of the two intersection points and the center point in each square frame is calculated, when the included angle of the two square frames is within the set angle threshold range, and the angle difference of the two angles is less than the set difference threshold, the corresponding pixel point is taken as a candidate corner point, and the key point is determined according to the candidate corner point. In this embodiment, the side length of one of the two square frames is 11 pixels, and the side length of the other square frame is 19 pixels; the set angle threshold range is [60°, 140°], and the set difference threshold is 10°, that is, when the intersection point of the square frame and the target contour line and the center point is within the set angle threshold ([60°, 140°]), it is indicated that the pixel point (the center point) cannot be on a straight line with the intersection point of the square frame and the target contour line, that is, it can be a corner point; meanwhile, considering the problem of large calculation amount caused by too many corner points, the present application also needs to consider whether the difference between the corresponding intersection points and the center point under the two square frames is within 10°, and the center point meeting the condition is taken as a candidate corner point. As other embodiments, the side length of the square frame can be determined according to the resolution of the actual image, and when the image resolution is high, the side length of the square frame can be set to be larger.

[0046] Since there can be multiple pixel points as candidate corner points in the set corner point region, the intersection point with the smallest angle difference between the corresponding intersection points and the center point under the two square frames is selected as the key point in the region. When there is only one candidate corner point in the set corner point region, it is taken as the key point. Among them, the size of the set corner point region is determined according to the square frame with longer side length, for example, the side lengths of the two square frames are 11 pixels and 19 pixels respectively, and the size of the set region is 19 pixels x 19 pixels. As other embodiments, all candidate corner points can also be taken as key points.

[0047] Step 4. Constructing a key point descriptor

[0048] A descriptor is constructed by using each key point, and the matching of the key points is realized through the descriptor. The descriptor is divided into two types (first descriptor and second descriptor). The first descriptor is an angle descriptor, as shown in formula (1), the average of the two angles of the key point obtained in step 3 (including the angle direction) is obtained, and the average angle is the first descriptor; the second descriptor is taken as the starting point of the top left corner of the square frame with the shortest side length, and the semantic label value of each pixel point on the square frame is obtained in the clockwise direction, as shown in formula (2), n is equal to the number of pixel points of the square frame with the shortest side length.

[0049]

[0050] The semantic label represents a target classification in the target region, and different values can represent different labels (different targets), for example, label=1 represents a building, label=2 represents a road, and the like.

[0051] As other embodiments, only one descriptor can be constructed, for example, only the first descriptor is constructed, or two descriptors can be constructed simultaneously.

[0052] Step 5. Determine the matching point pair

[0053] The present application determines the matching point pair of two images according to the matching degree of the descriptors in the two images.

[0054] For the first descriptor, the matching degree of the key points is determined according to the quotient of the first descriptors of the key points in the two images, and the closer the quotient value is to 1, the higher the matching degree of the key points in the two images is. As shown in formula (3), is the descriptor with a small angle, is the descriptor with a larger angle, and the closer the value of match1 is to 1, the more matched the two descriptors are, and the higher the matching degree of the key points in the two images is.

[0055]

[0056] For the second descriptor, the matching degree is described by using the alignment degree of the two descriptors, and the alignment degree refers to the ratio of the number of the same labels at the same position in the second descriptors of the two key points to the total number of labels, and the closer the ratio is to 1, the higher the matching degree of the key points in the two images is. The calculation formula is:

[0057]

[0058] In the formula, N true is the number of the same labels in def2, N is the total number of labels, and the closer the value of match2 is to 1, the more matched the two descriptors are, and the higher the matching degree of the key points in the two images is. For example, n in formula (2) is 40, that is, the side length of the smaller square frame is 11 pixels, the second descriptor of the key points of the real scene image is [label s1 , label s2 , label s3 , …, label sn ], the second descriptor of the key points of the rendered image is [label t1 , label t2 , label t3 , …, label tn ], and label s1 and label t1 , labels2 with label t2 , …, label sn with label tn if equal, get label si = label ti the number of label true = label si = label ti , N is the number of pixels of the square frame (40).

[0059] When only one descriptor is established in step 4, the matching degree of the corresponding descriptor is used to determine the matching point pair of the two images, the calculation amount is small and the matching efficiency is fast. For example, when the first descriptor of the two key points is determined to be a matching point pair, the quotient value (match1) of the first descriptor of the two key points is within the first set threshold range; or only the second descriptor is used to determine the matching point pair, when the alignment degree (match2) of the second descriptor of the two key points is within the second set threshold range, the two key points are determined as a matching point pair. When two descriptors are constructed, the matching degree of the two descriptors can be used to determine the matching point pair by weighted calculation, and the feature description of the key point is more accurate, and the matching precision is higher, as shown in formula (5):

[0060] match = ω1match1 + ω2match2 (5)

[0061] In the formula, match is the matching degree of the key points of the two images, match1 is the matching degree of the key points of the two images determined by the first descriptor, match2 is the matching degree of the key points of the two images determined by the second descriptor, ω1 and ω2 are weight coefficients of match1 and match2 respectively, and ω1 + ω2 = 1. When match is within the third set threshold range, the two key points are considered as a matching point pair.

[0062] In this embodiment, the first set threshold, the second set threshold and the third set threshold are all [0.7, 1]. As other embodiments, the selection of the threshold can also be determined according to the image quality. If the image quality (such as definition) is not high, the size of the minimum value can be appropriately adjusted, for example, the set threshold can be adjusted to [0.6, 1].

[0063] Step 6. Inverse calculation of real pose of sensor

[0064] After obtaining the matching point pairs, the motion of the imaging centers of the real scene image and the rendered image is calculated using the matching point pairs, and then the real pose of the sensor is calculated according to the previously estimated initial pose. The present application uses the epipolar geometric constraint in the SLAM idea to restore the motion of the imaging centers between two images through the corresponding relationship between the matching points of the two images. As shown in Figure 3 , if the motion relationship between the real scene image I1 and the rendered image I2 is to be calculated, let the motion be R, t, the camera (sensor) centers be O1 (real scene image) and O2 (rendered image) respectively, and p1 and p2 be the matching point pairs, after the camera intrinsic parameters, the pixel coordinates of p1 and p2, and the real coordinates of O2 obtained through the geographic space three-dimensional model are known, the real coordinates of O1 can be calculated using the projection relationship. The epipolar constraint formula (6) can be obtained:

[0065]

[0066] where E and F are the essential matrix (Essential Matrix) and the fundamental matrix (Fundamental Matrix) respectively, K is the camera intrinsic parameter matrix, and x1 and x2 are the coordinates of two pixel points on the normalized plane. Formula (6) gives the spatial position relationship of two matching points concisely, so the camera pose estimation problem is converted into the following two steps, first, the E or F is obtained according to the pixel position of the matching point pair, and then the R, t is obtained according to the E or F. Since the essential matrix E is more concise in form, E is selected for solving. According to the definition and properties of E, it has 5 degrees of freedom, and at least 5 pairs of points can be used for solving. Considering its non-linear intrinsic property, the present application uses the eight-point algorithm (Eight-point-algorithm) for solving, the specific process is as follows: in a pair of matching points, let the normalized coordinates be x1 = [u1, v1, 1] T , x2 = [u2, v2, 1] T , according to formula (6), the following can be obtained:

[0067]

[0068] The matrix E is expanded into a vector form to obtain e = [e1, e2, e3, e4, e5, e6, e7, e8, e9] T , the epipolar constraint can be written in linear form [u2u1, u2v1, u2, v2u1, v2v1, v2, u1, v1, 1] · e = 0. Similarly, other point pairs can also be represented in this way, and all points can be put into a matrix equation to obtain:

[0069]

[0070] If the matrix satisfies the full rank condition, the essential matrix E can be solved. The last step is to recover the motion R, t of the two imaging centers from the estimated essential matrix E, which is done by singular value decomposition (SVD).

[0071] Finally, the real pose of the sensor is estimated from the initial pose of the sensor and the estimated R and t, and the purpose of geo-fine registration is achieved. In addition, the calculated real pose is relative to the coordinate system of the three-dimensional model, and its accuracy in the real world depends on the geographical accuracy of the three-dimensional model.

[0072] Step 7. Target localization

[0073] After obtaining the real pose of the sensor and the instance target detected in the real scene image in step 2, the instance target is localized using the depth information and orientation of the real scene image. In order to reduce the amount of calculation, after detecting the instance target, the pixel center of each instance target is taken as the center of mass of the instance. The localization is completed using the depth information of the image, and the specific process is as follows:

[0074] Let P = [u, v, d] represent a point in the image, where u, v represent the pixel coordinates of the image, and d represents the corresponding depth value. According to the camera (sensor) intrinsic parameters and formula (9), the camera coordinates of the point can be calculated.

[0075]

[0076] where (x c ,y c ) is the principal point of the camera, f x and f y are the focal lengths of the camera, and (X P ,Y P ,Z P ) represents the camera coordinates of point P under the current pose. Then, according to the geographical coordinates of the sensor, the coordinates of the target are converted to the model (real world) geographical coordinate system to realize the localization of the target to support subsequent higher-level tasks. For example, after detecting the target as a dynamic target, the visualization requirement of the target can be completed to realize the coherent motion visualization of the dynamic target.

[0077] Embodiment 2:

[0078] The main difference between this embodiment and embodiment 1 is the extraction of key points and the construction of descriptors.

[0079] In order to simplify the extraction process of the key points, in the extraction of the key points, the embodiment only constructs a square frame for the pixel points on the contour line, when the included angle between the two intersection points of the square frame and the center point (pixel point) and the contour line is within the set angle threshold range, the point is taken as a candidate corner point, and the candidate corner point is taken as a key point. For example, only a square frame with a side length of 11 pixels is used for key point extraction, when the side length of the square frame is smaller, the distance between the two intersection points and the center point is closer, the correlation is higher, and the center point feature can be better represented, the center point corresponding to the included angle within [60°, 140°] is taken as a candidate corner point, and the candidate corner point is taken as a key point.

[0080] In the construction of the descriptor, since the embodiment only uses a square frame to extract the key points, the included angle between the two intersection points of the square frame and the center point and the contour line is taken as the first descriptor of the key point; the label value of each pixel point on the square frame in the set moving order is taken as the second descriptor of the key point. Consistent with embodiment 1, only one descriptor can be constructed, or two descriptors can be constructed at the same time. Similarly, the matching method of the key point is consistent with embodiment 1, either the matching degree of only one descriptor can be used to determine the matching point pair, or the matching degree of two descriptors can be used to determine the matching point pair.

Claims

1. A target localization method based on panoramic segmentation, characterized in that, The method includes the following steps: 1) Use an image acquisition sensor to acquire real scene images of the target area and obtain the initial pose of the sensor; use the initial pose of the sensor to render the established geospatial 3D model of the target area to obtain the rendered image of the target area under the initial pose. 2) Perform panoramic segmentation on the rendered image and the real scene image of the target area respectively to obtain the segmentation results of the rendered image and the real scene image; perform corner detection on the segmentation results of the rendered image and the real scene image respectively to obtain candidate corner points in each image, and determine the key points of the corresponding image based on the candidate corner points; 3) Construct descriptors using key points from two images, and determine matching point pairs between the two images based on the degree of matching between the descriptors in the two images; 4) Calculate the true pose of the sensor based on the matching point pair and the initial pose of the image acquisition sensor: After acquiring the matching point pair, use the matching point pair to calculate the motion of the imaging center of the real scene image and the rendered image, and then calculate the true pose of the sensor based on the initial pose of the sensor. 5) Target localization in real scene using the sensor’s real pose: After obtaining the sensor’s real pose and the target in the real scene image, the target is localized using the depth information and orientation of the real scene image.

2. The target localization method based on panoramic segmentation according to claim 1, characterized in that, The process of determining the key points is as follows: traverse the pixels on the target contour line in the image segmentation result, establish a square frame with the pixel as the center point, the square frame intersects the target contour line at two intersection points, calculate the angle between the two intersection points and the center point, and select the pixels corresponding to the angle within the set angle threshold range as candidate corner points, and select the candidate corner points as key points.

3. The target localization method based on panoramic segmentation according to claim 1, characterized in that, The process of determining the key points is as follows: traverse the pixels on the target contour line in the image segmentation result, establish two square boxes with different side lengths with the pixel as the center point, and intersect the target contour line at two intersection points respectively. Calculate the angle between the two intersection points and the center point in each square box. When the angle between the two square boxes is within the set angle threshold range and the difference between the two angles is less than the set difference threshold, the corresponding pixel is taken as a candidate corner point, and the key point is determined based on the candidate corner point.

4. The target localization method based on panoramic segmentation according to claim 3, characterized in that, When there are multiple candidate corner points within a defined corner point area, the candidate corner point with the smallest angle difference between the two included angles is selected as the key point within that defined corner point area.

5. The target localization method based on panoramic segmentation according to claim 3, characterized in that, A descriptor is constructed for the key point, the descriptor including a first descriptor and / or a second descriptor; wherein, the first descriptor is the average angle of the two included angles obtained from the key point; the second descriptor is the label value of each pixel in the square frame with the smaller side length established by the key point according to the set movement order.

6. The target localization method based on panoramic segmentation according to claim 5, characterized in that, The degree of matching of key points in two images is determined by the quotient of the first descriptor of key points in the two images. The closer the quotient is to 1, the higher the degree of matching of key points in the two images.

7. The target localization method based on panoramic segmentation according to claim 5, characterized in that, The matching degree of key points is determined based on the alignment degree of the second descriptors of key points in two images. The alignment degree refers to the ratio of the number of identical labels at the same position in the second descriptors of two key points to the total number of labels. The closer the ratio is to 1, the higher the matching degree of key points between the two images.

8. The target localization method based on panoramic segmentation according to claim 5, characterized in that, The formula for determining the degree of matching of key points between two images is: match = ω1match1 + ω2match2 In the formula, match represents the degree of matching between key points in two images, match1 represents the degree of matching between key points in two images determined by the first descriptor, match2 represents the degree of matching between key points in two images determined by the second descriptor, ω1 and ω2 are the weight coefficients of match1 and match2 respectively, and ω1+ω2=1.

9. The target localization method based on panoramic segmentation according to claim 1, characterized in that, In step 4), the epipolar geometric constraint method is used to calculate the true pose of the sensor.

10. The target localization method based on panoramic segmentation according to claim 1, characterized in that, In step 2), the MaskFormer network is used for panoramic image segmentation.

Images