A terminal device positioning method and related device

Abstract

The application provides a terminal device positioning method and related equipment, which can improve the accuracy of the positioning result of the terminal device. The method of the application comprises: obtaining, from a vector map, a first map point matched with a first feature point of a current image frame photographed by a terminal device; obtaining, from the vector map, a second map point matched with a second feature point of another image frame before the current image frame; and adjusting a pose of the terminal device when the current image frame is photographed to obtain a pose of the terminal device when the current image frame is photographed after a current adjustment, as a positioning result of the terminal device.

Description

[0001] This application is a divisional application, the original application number is 202110460636.4, the original application date is April 27, 2021, and the entire contents of the original application are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of artificial intelligence, and in particular to a terminal device positioning method and related equipment thereof. BACKGROUND

[0003] At present, intelligent terminal devices such as autonomous vehicles, drones and robots have been widely used in daily life. For this part of mobile terminal devices, in order to accurately obtain the real-time position of the terminal device, high-precision positioning technology has emerged.

[0004] In the process of positioning the terminal device, the current image frame captured by the terminal device can be obtained, and the map points matched with the feature points in the current image frame for presenting the objects in the traffic environment are obtained from the preset vector map (in the map, the objects in the traffic environment can be represented by map points, for example, a lamp pole is represented by a straight line formed by map points, a signboard is represented by a rectangular frame formed by map points, etc.). Finally, the positioning result of the terminal device in the vector map is determined according to the matching result between the feature points and the map points.

[0005] However, the factors considered in the above positioning process of the terminal device are relatively single, resulting in low accuracy of the positioning result of the terminal device. SUMMARY

[0006] The embodiments of the present application provide a terminal device positioning method and related equipment, which can improve the accuracy of the positioning result of the terminal device.

[0007] The first aspect of the embodiments of the present application provides a terminal device positioning method, which comprises:

[0008] In the process of moving, the terminal device can capture the traffic environment through the camera at the current time to obtain the current image frame. Further, the terminal device can also obtain other image frames before the current image frame. Then, the terminal device can position itself according to the current image frame and the other image frames.

[0009] Specifically, the terminal device first acquires, from the vector map, first map points matched with first feature points of the current image frame. For example, a feature point in the current image frame for presenting a traffic light and a map point in the vector map for representing the traffic light are matched points, a feature point in the current image frame for presenting a lane line and a map point in the vector map for representing the lane line are matched points, and the like. Similarly, the terminal device can also acquire, from the vector map, second map points matched with second feature points of other image frames before the current image frame.

[0010] Since there is a certain matching error between the first feature points and the first map points, and there is a certain matching error between the second feature points and the second map points, it is necessary to make the two matching errors as small as possible to improve the accuracy of the positioning result of the terminal device.

[0011] Based on this, the terminal device can construct a target function according to the first matching error between the first feature points and the first map points, and the second matching error between the second feature points and the second map points, and adjust the pose of the terminal device when shooting the current image frame according to the target function, that is, optimize the pose of the terminal device when shooting the current image frame according to the target function, until the target function converges, thereby obtaining the pose of the terminal device when shooting the current image frame after the current adjustment (optimization), and taking it as the positioning result of the terminal device in the vector map. Wherein, the pose of the terminal device when shooting the current image frame usually refers to the pose of the terminal device in the three-dimensional coordinate system corresponding to the vector map when shooting the current image frame.

[0012] As can be seen from the above method, after the current image frame and the other image frames before the current image frame are obtained, the first map point matched with the first feature point of the current image frame and the second map point matched with the second feature point of the other image frames before the current image frame can be obtained from the vector map. Then, the pose of the terminal device when the current image frame is captured can be adjusted according to the target function constructed according to the first matching error between the first feature point and the first map point and the second matching error between the second feature point and the second map point, to obtain the pose of the terminal device when the current image frame is captured after the current adjustment. In the foregoing process, since the target function contains the matching error between the feature points of the current image frame and the map points of the vector map and the matching error between the feature points of the other image frames and the map points of the vector map, the pose of the terminal device when the current image frame is captured is adjusted through the target function, which not only considers the influence of the optimization process of the current image frame on the pose of the terminal device when the current image frame is captured, but also considers the influence of the optimization process of the other image frames on the pose of the terminal device when the current image frame is captured, that is, the correlation between the current image frame and the other image frames is considered, the considered factors are more comprehensive, and therefore the positioning result of the terminal device obtained based on this method has higher accuracy.

[0013] In a possible implementation, the method further includes obtaining the pose of the terminal device when the current image frame is captured and the pose of the terminal device when the other image frames are captured after the last adjustment, and performing semantic detection on the current image frame and the other image frames before the current image frame, to obtain the first feature point of the current image frame and the second feature point of the other image frames before the current image frame. Then, the first map point matched with the first feature point can be obtained from the vector map according to the pose of the terminal device when the current image frame is captured, and the second map point matched with the second feature point can be obtained from the vector map according to the pose of the terminal device when the other image frames are captured after the last adjustment. In this way, the correlation matching between the feature points and the map points can be completed.

[0014] In a possible implementation, the adjusting the pose of the terminal device when capturing the current image frame according to the target function to obtain the pose of the terminal device when capturing the current image frame after the current adjustment comprises: after obtaining the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system, the initial value of the first matching error can be obtained by calculating the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system. Then, after obtaining the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system, the initial value of the second matching error can be obtained by calculating the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system. Finally, the target function is iteratively solved according to the initial value of the first matching error and the initial value of the second matching error until the preset iteration condition is met, and the pose of the terminal device when capturing the current image frame after the current adjustment is obtained. In the foregoing implementation, after the matching between the first feature point and the first map point and the matching between the second feature point and the second map point are completed, the initial value of the first matching error between the first feature point and the first map point and the initial value of the second matching error between the second feature point and the second map point can be calculated, so as to iteratively solve the target function in combination with the two initial values, which is equivalent to adjusting the pose of the terminal device when capturing the current image frame according to the current image frame and other image frames, and the considered factors are more comprehensive, so that the positioning result of the terminal device is accurately obtained.

[0015] In a possible implementation, the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system comprises at least one of the following: (1) the distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame, where the position of the first map point in the current image frame is obtained according to the position of the first map point in the three-dimensional coordinate system corresponding to the vector map and the pose of the terminal device when the current image frame is captured; (2) the distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the first map point in the three-dimensional coordinate system corresponding to the vector map, where the position of the first feature point in the three-dimensional coordinate system corresponding to the vector map is obtained according to the position of the first feature point in the current image frame and the pose of the terminal device when the current image frame is captured; and (3) the distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device, where the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device is obtained according to the position of the first feature point in the current image frame and the pose of the terminal device when the current image frame is captured, and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device is obtained according to the position of the first map point in the three-dimensional coordinate system corresponding to the vector map and the pose of the terminal device when the current image frame is captured.

[0016] In a possible implementation, the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system comprises at least one of the following: (1) the distance between the position of the second feature point in the other image frame and the position of the second map point in the other image frame, where the position of the second map point in the other image frame is obtained according to the position of the second map point in the three-dimensional coordinate system corresponding to the vector map and the pose of the terminal device when the other image frame is captured after the last adjustment; (2) the distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the second map point in the three-dimensional coordinate system corresponding to the vector map, where the position of the second feature point in the three-dimensional coordinate system corresponding to the vector map is obtained according to the position of the second feature point in the other image frame and the pose of the terminal device when the other image frame is captured after the last adjustment; and (3) the distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the second map point in the three-dimensional coordinate system corresponding to the terminal device, where the position of the second feature point in the three-dimensional coordinate system corresponding to the terminal device is obtained according to the position of the second feature point in the other image frame and the pose of the terminal device when the other image frame is captured after the last adjustment, and the position of the second map point in the three-dimensional coordinate system corresponding to the terminal device is obtained according to the position of the second map point in the three-dimensional coordinate system corresponding to the vector map and the pose of the terminal device when the other image frame is captured after the last adjustment.

[0017] In a possible implementation, the iteration condition is: if a difference between the inter-frame pose difference obtained in the iteration and the inter-frame pose difference calculated by the terminal device is less than a preset threshold, the iteration is stopped, the inter-frame pose difference is determined according to a pose of the terminal device when the current image frame is captured and a pose of the terminal device when the other image frames are captured, and the inter-frame pose difference is a pose difference between two adjacent image frames captured by the terminal device; if the difference is greater than or equal to the threshold, the next iteration is performed, and the iterations are performed until the number of iterations is equal to a preset number of times.

[0018] In a possible implementation, the number of the other image frames can change with the motion state of the terminal device. Specifically, the number of the other image frames can be determined according to the speed of the terminal device.

[0019] In a possible implementation, obtaining the pose of the terminal device when the current image frame is captured includes: calculating a predicted pose of the terminal device when the current image frame is captured according to the pose of the terminal device when the other image frames are captured after the last adjustment and the inter-frame pose difference calculated by the terminal device; and performing hierarchical sampling on the predicted pose of the terminal device when the current image frame is captured to obtain the pose of the terminal device when the current image frame is captured. In the foregoing implementation, the pose of the terminal device when the current image frame is captured obtained through hierarchical sampling can be used as the initial pose of the current adjustment, thereby improving the convergence speed and robustness of the current adjustment.

[0020] In a possible implementation, if the pose of the terminal device when capturing the current image frame includes a horizontal coordinate, a vertical coordinate and a heading angle, the hierarchical sampling of the predicted pose of the terminal device when capturing the current image frame includes: obtaining a position of a third map point in a three-dimensional coordinate system corresponding to a vector map and a position of a first feature point in the current image frame; keeping the heading angle of the predicted pose of the terminal device when capturing the current image frame unchanged, and changing the horizontal coordinate and the vertical coordinate of the predicted pose of the terminal device when capturing the current image frame to obtain a first candidate pose; transforming the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the first candidate pose to obtain a position of the third map point in a preset image coordinate system; keeping the horizontal coordinate and the vertical coordinate of the predicted pose of the terminal device when capturing the current image frame unchanged, and changing the heading angle of the predicted pose of the terminal device when capturing the current image frame to obtain a second candidate pose; transforming the position of the first feature point in the current image frame according to the second candidate pose to obtain a position of the first feature point in the image coordinate system; and determining the pose of the terminal device when capturing the current image frame from combinations of the first candidate pose and the second candidate pose according to a distance between the position of the third map point in the image coordinate system and the position of the first feature point in the image coordinate system. Through the foregoing pose sampling manner, the amount of calculation required in the pose sampling process can be effectively reduced.

[0021] In a possible implementation, if the pose of the terminal device when the current image frame is captured includes a horizontal coordinate, a vertical coordinate, a vertical coordinate, a heading angle, a roll angle and a pitch angle, the predicted pose of the terminal device when the current image frame is captured is sampled in layers to obtain the pose of the terminal device when the current image frame is captured, including: obtaining the position of the third map point in the three-dimensional coordinate system corresponding to the vector map and the position of the first feature point in the current image frame; keeping the heading angle, the roll angle, the pitch angle and the vertical coordinate of the predicted pose of the terminal device when the current image frame is captured unchanged, changing the horizontal coordinate and the vertical coordinate of the predicted pose of the terminal device when the current image frame is captured, to obtain a first candidate pose; transforming the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the first candidate pose to obtain the position of the third map point in the preset image coordinate system; keeping the horizontal coordinate, the vertical coordinate, the vertical coordinate, the roll angle and the pitch angle of the predicted pose of the terminal device when the current image frame is captured unchanged, changing the heading angle of the predicted pose of the terminal device when the current image frame is captured, to obtain a second candidate pose; transforming the position of the first feature point in the current image frame according to the second candidate pose to obtain the position of the first feature point in the image coordinate system; determining the third candidate pose from the combination of the first candidate pose and the second candidate pose according to the distance between the position of the third map point in the image coordinate system and the position of the first feature point in the image coordinate system; keeping the horizontal coordinate, the vertical coordinate, the heading angle and the roll angle of the predicted pose of the third candidate pose unchanged, changing the pitch angle and the vertical coordinate of the third candidate pose to obtain a fourth candidate pose; transforming the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the fourth candidate pose to obtain the position of the third map point in the current image frame; and determining the pose of the terminal device when the current image frame is captured from the fourth candidate pose according to the distance between the position of the first feature point in the current image frame and the position of the third map point in the current image frame. Through the foregoing pose sampling manner, the amount of calculation required in the pose sampling process can be effectively reduced.

[0022] A second aspect of the embodiment of the present application provides a terminal device positioning apparatus, which includes: a first matching module, configured to obtain, from a vector map, a first map point matched with a first feature point of a current image frame; a second matching module, configured to obtain, from the vector map, a second map point matched with a second feature point of another image frame before the current image frame; and an optimization module, configured to adjust a pose of the terminal device when the current image frame is captured according to a target function to obtain a current-time adjusted pose of the terminal device when the current image frame is captured as a positioning result of the terminal device, the target function including a first matching error between the first feature point and the first map point and a second matching error between the second feature point and the second map point.

[0023] It can be seen from the above device that after the current image frame and the other image frames before the current image frame are obtained, the first map point matched with the first feature point of the current image frame and the second map point matched with the second feature point of the other image frames before the current image frame can be obtained from the vector map. Then, the target function constructed according to the first matching error between the first feature point and the first map point and the second matching error between the second feature point and the second map point is used to adjust the pose of the terminal device when the current image frame is captured, so as to obtain the pose of the terminal device when the current image frame is captured after the current adjustment. In the foregoing process, since the target function contains the matching error between the feature points of the current image frame and the map points of the vector map and the matching error between the feature points of the other image frames and the map points of the vector map, the adjustment of the pose of the terminal device when the current image frame is captured by using the target function not only considers the influence of the optimization process of the current image frame on the pose of the terminal device when the current image frame is captured, but also considers the influence of the optimization process of the other image frames on the pose of the terminal device when the current image frame is captured, that is, the correlation between the current image frame and the other image frames is considered, the considered factors are more comprehensive, and therefore the positioning result of the terminal device obtained based on this method has higher accuracy.

[0024] In a possible implementation, the device further includes an obtaining module configured to obtain the first feature point of the current image frame, the second feature point of the other image frames before the current image frame, the pose of the terminal device when the current image frame is captured, and the pose of the terminal device when the other image frames are captured after the last adjustment; a first matching module configured to obtain, from the vector map, the first map point matched with the first feature point according to the pose of the terminal device when the current image frame is captured; and a second matching module configured to obtain, from the vector map, the second map point matched with the second feature point according to the pose of the terminal device when the other image frames are captured after the last adjustment.

[0025] In a possible implementation, the optimization module is configured to: obtain the initial value of the first matching error by calculating the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system; obtain the initial value of the second matching error by calculating the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system; and iteratively solve the target function according to the initial value of the first matching error and the initial value of the second matching error until a preset iteration condition is met, so as to obtain the pose of the terminal device when the current image frame is captured after the current adjustment.

[0026] In a possible implementation, the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system comprises at least one of the following: a distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame; a distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the first map point in the three-dimensional coordinate system corresponding to the vector map; and a distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device.

[0027] In a possible implementation, the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system comprises at least one of the following: a distance between the position of the second feature point in the other image frame and the position of the second map point in the other image frame; a distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the second map point in the three-dimensional coordinate system corresponding to the vector map; and a distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the second map point in the three-dimensional coordinate system corresponding to the terminal device.

[0028] In a possible implementation, the iteration condition is that, for any iteration, if a difference between the inter-frame pose difference obtained in the iteration and the inter-frame pose difference calculated by the terminal device is less than a preset threshold, the iteration is stopped, the inter-frame pose difference is determined according to the pose of the terminal device when the current image frame is captured in the iteration and the pose of the terminal device when the other image frame is captured in the iteration, and the inter-frame pose difference is a pose difference between two adjacent image frames captured by the terminal device in the current image frame and the other image frame; if the difference is greater than or equal to the threshold, the next iteration is performed until the number of iterations is equal to a preset number.

[0029] In a possible implementation, the number of the other image frames is determined according to the speed of the terminal device.

[0030] In a possible implementation, the obtaining module is configured to: calculate the predicted pose of the terminal device when the current image frame is captured according to the pose of the terminal device when the other image frame is captured after the last adjustment and the inter-frame pose difference calculated by the terminal device; and perform hierarchical sampling on the predicted pose of the terminal device when the current image frame is captured to obtain the pose of the terminal device when the current image frame is captured.

[0031] In a possible implementation, if the pose of the terminal device when capturing the current image frame comprises a horizontal coordinate, a vertical coordinate and a heading angle, the obtaining module is configured to: obtain a position of the third map point in a three-dimensional coordinate system corresponding to the vector map and a position of the first feature point in the current image frame; keep the heading angle of the predicted pose of the terminal device when capturing the current image frame unchanged, change the horizontal coordinate and the vertical coordinate of the predicted pose of the terminal device when capturing the current image frame, and obtain a first candidate pose; transform the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the first candidate pose, and obtain a position of the third map point in a preset image coordinate system; keep the horizontal coordinate and the vertical coordinate of the predicted pose of the terminal device when capturing the current image frame unchanged, change the heading angle of the predicted pose of the terminal device when capturing the current image frame, and obtain a second candidate pose; transform the position of the first feature point in the current image frame according to the second candidate pose, and obtain a position of the first feature point in the image coordinate system; and determine the pose of the terminal device when capturing the current image frame from combinations of the first candidate pose and the second candidate pose according to a distance between the position of the third map point in the image coordinate system and the position of the first feature point in the image coordinate system. Through the foregoing pose sampling manner, the amount of calculation required in the pose sampling process can be effectively reduced.

[0032] In a possible implementation, if the pose of the terminal device when the current image frame is captured comprises a horizontal coordinate, a vertical coordinate, a vertical coordinate, a heading angle, a roll angle and a pitch angle, the obtaining module is configured to: obtain a position of the third map point in a three-dimensional coordinate system corresponding to the vector map and a position of the first feature point in the current image frame; keep the heading angle, the roll angle, the pitch angle and the vertical coordinate of the predicted pose of the terminal device when the current image frame is captured unchanged, change the horizontal coordinate and the vertical coordinate of the predicted pose of the terminal device when the current image frame is captured, and obtain a first candidate pose; transform the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the first candidate pose, and obtain a position of the third map point in a preset image coordinate system; keep the horizontal coordinate, the vertical coordinate, the vertical coordinate, the roll angle and the pitch angle of the predicted pose of the terminal device when the current image frame is captured unchanged, change the heading angle of the predicted pose of the terminal device when the current image frame is captured, and obtain a second candidate pose; transform the position of the first feature point in the current image frame according to the second candidate pose, and obtain a position of the first feature point in the image coordinate system; determine the third candidate pose from combinations of the first candidate pose and the second candidate pose according to a distance between the position of the third map point in the image coordinate system and the position of the first feature point in the image coordinate system; keep the horizontal coordinate, the vertical coordinate, the heading angle and the roll angle of the predicted pose of the third candidate pose unchanged, change the pitch angle and the vertical coordinate of the third candidate pose, and obtain a fourth candidate pose; transform the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the fourth candidate pose, and obtain a position of the third map point in the current image frame; and determine the pose of the terminal device when the current image frame is captured according to a distance between the position of the first feature point in the current image frame and the position of the third map point in the current image frame. Through the foregoing pose sampling manner, the amount of calculation required in the pose sampling process can be effectively reduced.

[0033] The third aspect of the embodiments of the present application provides a terminal device positioning apparatus, which comprises a memory and a processor; the memory stores code, and the processor is configured to execute the code, and when the code is executed, the terminal device positioning apparatus performs the method according to the first aspect or any possible implementation manner of the first aspect.

[0034] The fourth aspect of the embodiments of the present application provides a vehicle, which comprises the terminal device positioning apparatus according to the third aspect.

[0035] The fifth aspect of the embodiments of the present application provides a computer storage medium, which stores a computer program, and when the computer program is executed by a computer, the computer program causes the computer to implement the method according to the first aspect or any possible implementation manner of the first aspect.

[0036] The sixth aspect of the embodiments of the present application provides a computer program product, the computer program product stores instructions, when the instructions are executed by a computer, the computer implements the method in the first aspect or any possible implementation manner of the first aspect.

[0037] In the embodiments of the present application, after obtaining the current image frame and the other image frame before the current image frame, the first map point matched with the first feature point of the current image frame and the second map point matched with the second feature point of the other image frame before the current image frame can be obtained from the vector map. Then, the target function constructed according to the first matching error between the first feature point and the first map point and the second matching error between the second feature point and the second map point is used to adjust the pose of the terminal device when the terminal device shoots the current image frame, so as to obtain the pose of the terminal device when the terminal device shoots the current image frame after the current adjustment. In the foregoing process, since the target function contains the matching error between the feature point of the current image frame and the map point of the vector map and the matching error between the feature point of the other image frame and the map point of the vector map, the adjustment of the pose of the terminal device when the terminal device shoots the current image frame through the target function not only considers the influence caused by the optimization process of the current image frame on the pose of the terminal device when the terminal device shoots the current image frame, but also considers the influence caused by the optimization process of the other image frame on the pose of the terminal device when the terminal device shoots the current image frame (that is, the correlation between the current image frame and the other image frame is considered), the considered factors are more comprehensive, and therefore the positioning result of the terminal device obtained based on this kind of method has higher accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 An illustration of a vector map is shown in FIG. 1;

[0039] Figure 2 An illustration of a flow of the positioning method of the terminal device provided by the embodiments of the present application is shown in FIG. 2;

[0040] Figure 3 An illustration of a three-dimensional coordinate system corresponding to the terminal device provided by the embodiments of the present application is shown in FIG. 3;

[0041] Figure 4 An illustration of the inter-frame pose difference provided by the embodiments of the present application is shown in FIG. 4;

[0042] Figure 5 An illustration of the first feature point of the current image frame provided by the embodiments of the present application is shown in FIG. 5;

[0043] Figure 6 An illustration of the calculation of the overlapping degree provided by the embodiments of the present application is shown in FIG. 6;

[0044] Figure 7 An illustration of the structure of the terminal device positioning apparatus provided by the embodiments of the present application is shown in FIG. 7;

[0045] Figure 8 Another structural schematic diagram of the terminal device positioning device provided in the embodiments of this application. Detailed Implementation

[0046] This application provides a terminal device positioning method and related equipment, which can improve the accuracy of the positioning results of the terminal device.

[0047] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0048] The embodiments of this application can be implemented through terminal devices, such as in-vehicle devices in automobiles, drones, robots, etc. For ease of explanation, the in-vehicle devices in automobiles will be referred to as automobiles below, and an example of a moving automobile will be used for the description.

[0049] When a car is in motion, high-precision positioning is required for the user to determine its location. In related technologies, cars typically have pre-installed complete vector maps. Figure 1 A schematic diagram of a vector map, such as Figure 1 As shown, a vector map displays the virtual traffic environment in which a car is currently located. This environment includes various objects around the car, such as traffic lights, light poles, signs, lane lines, etc. These objects are represented by pixels on the vector map, that is, by map points. For example, a light pole can be represented by a straight line formed by multiple map points, and a sign can be represented by a rectangle formed by multiple map points, and so on. It is worth noting that the virtual traffic environment displayed by the vector map is drawn based on the real-world traffic environment, while the car's pose displayed on the vector map is generally calculated by the car and may differ from the car's actual pose in the real world. Therefore, it is necessary to correct and optimize the car's pose in the vector map to improve the accuracy of the car's localization results. Understandably, the car's pose usually includes the car's position and orientation, which will not be elaborated further.

[0050] Specifically, during driving, the vehicle can capture a current image frame for presenting a real traffic environment at a current moment. Then, the vehicle can match feature points of the current image frame with map points of the vector map, which is equivalent to matching the real traffic environment where the vehicle is located with a virtual traffic environment where the vehicle is located. Finally, according to a matching result between the feature points of the current image frame and the map points of the vector map, such as a matching error therebetween, and the like, a pose of the vehicle in the vector map is adjusted, and the pose of the vehicle after optimization is taken as a positioning result of the vehicle.

[0051] However, only by using the current image frame to determine the positioning result of the vehicle, the considered factors are relatively single, resulting in low accuracy of the positioning result of the vehicle.

[0052] Therefore, in order to improve the accuracy of the positioning result of the terminal device, an embodiment of the present application provides a positioning method of a terminal device. For the convenience of description, hereinafter, a pose of the terminal device when capturing an arbitrary image frame is referred to as a pose of the image frame, for example, a pose of the terminal device when capturing a current image frame is referred to as a pose of the current image frame, and a pose of the terminal device when capturing other image frames before the current image frame is referred to as a pose of the other image frames, and the like. For example, after a current optimization (adjustment) is performed on the pose of the terminal device when capturing the current image frame, a pose of the terminal device when capturing the current image frame after the current optimization (adjustment) is performed is obtained, which is referred to as a pose of the current image frame after the current optimization, and the like. Details are not described herein. Figure 2 A flowchart of a positioning method of a terminal device provided by an embodiment of the present application is shown in FIG. 1. The method includes the following steps. Figure 2

[0053] 201. Obtain a first feature point of a current image frame, a second feature point of other image frames before the current image frame, a pose of the current image frame, and a pose of the other image frames after a last optimization.

[0054] In the embodiment, the terminal device has a camera. During movement, the terminal device can capture a current traffic environment through the camera to obtain a current image frame. Further, the terminal device can also obtain other image frames before the current image frame. The number of the other image frames can be determined according to a speed of the terminal device, as shown in formula (1).

[0055]

[0056] ​In the above formula, t is the number of the current image frame and other image frames, t-1 is the number of the other image frames, t0 is a preset threshold, a is a preset adjustment coefficient, and v is the speed of the terminal device at the current moment. In this way, after the current image frame and the other image frames are obtained, the terminal device can perform positioning on itself according to the current image frame and the other image frames.

[0057] After the terminal device obtains the current image frame and the other image frames, the terminal device can obtain the pose of the current image frame and the pose of the other image frames after the last optimization. It should be noted that, for the pose of the current image frame, the pose of the current image frame can be optimized according to the current image frame and the other image frames to obtain the pose of the current image frame after the current optimization and the pose of the other image frames after the current optimization. As can be seen, the pose of the other image frames after the last optimization is the result obtained after the last optimization of the pose of the other image frames according to the other image frames.

[0058] The pose of the current image frame can be obtained in the following manner: first, the predicted pose of the current image frame is calculated according to the pose of the other image frames after the last optimization and the inter-frame pose difference calculated by the terminal device. Then, the predicted pose of the current image frame is sampled in layers to obtain the pose of the current image frame.

[0059] Specifically, the terminal device can also have an odometer, which can construct a three-dimensional coordinate system corresponding to the terminal device, such as a vehicle body coordinate system and the like. Figure 3 A schematic diagram of a three-dimensional coordinate system corresponding to the terminal device provided by an embodiment of the present application is shown in Figure 3 As shown in the figure, in the three-dimensional coordinate system, the origin is the starting point of the motion of the terminal device, the X-axis points to the front of the terminal device at the starting point of the motion, the Y-axis points to the left side of the terminal device at the starting point of the motion, and the Z-axis can be set as zero by default. Then, when the terminal device starts to move from the origin, its position and orientation change constantly (i.e., rotation and translation). The odometer can calculate the pose difference of the terminal device when capturing two adjacent image frames during the motion of the terminal device. The pose difference is also called inter-frame pose difference.

[0060] ΔT = {ΔR, Δt} (2)

[0061] In the above formula, ΔT is the inter-frame pose difference, ΔR is the rotation between the two adjacent image frames, and Δt is the translation between the two adjacent image frames.

[0062] In order to further understand the inter-frame pose difference, the following Figure 4 will be further introduced. Figure 4A schematic diagram of the inter-frame pose difference provided in the embodiments of this application is shown below. Figure 4 As shown, let there be a total of t frames, including the current image frame and all other image frames preceding it. Here, F1 represents the first image frame among the other image frames, F2 represents the second image frame among the other image frames, ..., F... t-1 F represents the last image frame among other image frames (i.e., the image frame preceding the current image frame). t This represents the current image frame. The odometry can calculate the pose difference ΔT between F1 and F2. t-1 ..., to obtain F t-1 and F t The pose difference between them is ΔT t-1 Therefore, the predicted pose of the current image frame can be calculated using formula (3):

[0063] P t =P t-1 *ΔT t-1 (3)

[0064] In the above formula, P t For the predicted pose of the current image frame, P t-1 This is the pose of the previous image frame after the last optimization. Based on formula (3), the predicted pose of the current image frame can also be calculated using formula (4):

[0065] P t =(ΔT) t-1 *ΔT t-2 *…*ΔT t-m )*P t-m (4)

[0066] In the above formula, P t-m This refers to the pose of the tm-th image frame among the other image frames from the previous optimization.

[0067] After obtaining the predicted pose of the current image frame, the predicted pose can be sampled hierarchically to obtain the pose of the current image frame, which is the initial pose value used for the current optimization. Specifically, the predicted pose of the current image frame can be sampled hierarchically in several ways, which will be introduced below:

[0068] In one possible implementation, if the pose of the current image frame is a three-degree-of-freedom quantity, i.e., including the horizontal coordinate, vertical coordinate, and heading angle, the layered sampling process includes: (i) arbitrarily selecting a portion of map points within a pre-defined range in the vector map as third map points, and obtaining the position of the third map points in the three-dimensional coordinate system corresponding to the vector map and the position of the first feature point in the current image frame. It can be understood that the position of the third map points in the three-dimensional coordinate system corresponding to the vector map is a three-dimensional coordinate, and the position of the first feature point in the current image frame is a two-dimensional coordinate. (ii) keeping the heading angle of the predicted pose of the current image frame unchanged, changing the horizontal and vertical coordinates of the predicted pose of the current image frame to obtain the first candidate pose. (iii) transforming the position of the third map points in the three-dimensional coordinate system corresponding to the vector map based on the first candidate pose to obtain the position of the third map points in a preset image coordinate system. This process is equivalent to projecting the third map points into the image coordinate system. (iv) Keeping the x and y coordinates of the predicted pose of the current image frame unchanged, change the heading angle of the predicted pose of the current image frame to obtain the second candidate pose. (v) Transform the position of the first feature point in the current image frame according to the second candidate pose to obtain the position of the first feature point in the image coordinate system. This process is equivalent to projecting the first feature point into the image coordinate system. (vi) Determine the pose of the current image frame from the combination of the first candidate pose and the second candidate pose based on the distance between the position of the third map point in the image coordinate system and the position of the first feature point in the image coordinate system. The aforementioned pose sampling method can effectively reduce the amount of computation required in the pose sampling process.

[0069] To further understand the above sampling process, an example is provided below, which includes: (i) determining the first feature point of the current image frame and the third map point in the vector map used for hierarchical sampling. (ii) keeping the heading angle of the current image frame unchanged, sampling N1 times based on the original value of the horizontal coordinate and N2 times based on the original value of the vertical coordinate, obtaining N1×N2 first candidate poses. (iii) based on each first candidate pose, projecting the third map point in the vector map onto a preset image coordinate system, obtaining N1×N2 new sets of third map points. (iv) keeping the horizontal and vertical coordinates of the predicted pose of the current image frame unchanged, sampling N3 times based on the original value of the heading angle, obtaining N3 second candidate poses. (v) based on each second candidate pose, projecting the first feature point of the current image frame onto a preset image coordinate system, obtaining N3 new sets of first feature points. (vi) In the preset image coordinate system, based on the N1×N2 new third map points and the N3 new first feature points, N1×N2×N3 new pose combinations are formed. The distance between the third map point and the first feature point in each combination is calculated to obtain N1×N2×N3 distances. The smallest distance is selected from these distances, and the pose of the current image frame is formed by the x-coordinate and y-coordinate of the first candidate pose corresponding to this distance, and the heading angle of the second candidate pose corresponding to this distance.

[0070] In another possible implementation, if the pose of the current image frame is a six-degree-of-freedom quantity, i.e., including the x-coordinate, y-coordinate, vertical coordinate, heading angle, roll angle, and pitch angle, then the process of layered sampling includes: (i) obtaining the position of the third map point in the three-dimensional coordinate system corresponding to the vector map and the position of the first feature point in the current image frame. (ii) keeping the heading angle, roll angle, pitch angle, and vertical coordinate of the predicted pose of the current image frame unchanged, and changing the x-coordinate and y-coordinate of the predicted pose of the current image frame to obtain the first candidate pose. (iii) transforming the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the first candidate pose to obtain the position of the third map point in the preset image coordinate system. (iv) keeping the x-coordinate, y-coordinate, vertical coordinate, roll angle, and pitch angle of the predicted pose of the current image frame unchanged, and changing the heading angle of the predicted pose of the current image frame to obtain the second candidate pose. (v) transforming the position of the first feature point in the current image frame according to the second candidate pose to obtain the position of the first feature point in the image coordinate system. (vi) Based on the distance between the position of the third map point in the image coordinate system and the position of the first feature point in the image coordinate system, determine the third candidate pose from the combination of the first and second candidate poses. (vii) Keeping the x-coordinate, y-coordinate, heading angle, and roll angle of the predicted pose of the third candidate pose unchanged, change the pitch angle and vertical coordinate of the third candidate pose to obtain the fourth candidate pose. (viii) Transform the position of the third map point in the three-dimensional coordinate system corresponding to the vector map based on the fourth candidate pose to obtain the position of the third map point in the current image frame. (ix) Based on the distance between the position of the first feature point in the current image frame and the position of the third map point in the current image frame, determine the pose of the current image frame from the fourth candidate pose. Through the aforementioned pose sampling method, the amount of computation required in the pose sampling process can be effectively reduced.

[0071] To further understand the above sampling process, an example is provided below, which includes steps (i) to (ix). Steps (i) to (v) can be referred to in the previous example, and will not be repeated here. (vi) In the preset image coordinate system, N1×N2 new third map points and N3 new first feature points are used to form N1×N2×N3 new combinations. The distance between the third map point and the first feature point in each combination is calculated to obtain N1×N2×N3 distances. The smallest distance is selected, and the abscissa and ordinate of the first candidate pose corresponding to the distance, and the heading angle of the second candidate pose corresponding to the distance are used to form the third candidate pose. (vii) Keeping the abscissa, ordinate, heading angle and roll angle of the predicted pose of the third candidate pose unchanged, N4 times are sampled based on the original value of the pitch angle, and N5 times are sampled based on the original value of the ordinate, to obtain N4×N5 fourth candidate poses. (viii) Based on each fourth candidate pose, project the third map point in the vector map onto the current image frame to obtain N4×N5 new third map points. (ix) In the current image frame, based on the N4×N5 new third map points and the first feature point in the current image frame, construct N4×N5 new combinations, calculate the distance between the third map point and the first feature point in each combination, and obtain N4×N5 distances. Select the smallest distance from these distances, and use the pitch angle and vertical coordinate of the fourth candidate pose corresponding to this distance, as well as the horizontal coordinate, vertical coordinate, heading angle, and roll angle of the third candidate pose, to form the pose of the current image frame.

[0072] After obtaining the pose of the current image frame and the poses of other image frames after the last optimization, semantic detection can be performed on the current image frame and other image frames to obtain the first feature point of the current image frame and the second feature point of other image frames. Specifically, semantic detection processing, i.e., feature extraction, can be performed on the current image frame and other image frames separately using a neural network to obtain the first feature point of the current image frame and the second feature point of other image frames. The first and second feature points can be understood as semantic markers on the image. It should be noted that the first feature point of the current image frame includes feature points of various objects in the traffic environment. Figure 5 A schematic diagram of the first feature point of the current image frame provided in the embodiments of this application, as shown below. Figure 5 As shown, the feature points of a lamppost and lane lines can be pixels at their two ends, while the feature points of a traffic light and a sign can be rectangular boxes (i.e., bounding boxes) formed by multiple pixels, and so on. Similarly, the second feature points of other image frames are also like this, which will not be elaborated here.

[0073] It should be understood that the aforementioned neural network is a trained neural network model. The training process of this neural network will be briefly described below:

[0074] Before training the model, a batch of training image frames is acquired, and the true feature points in each training image frame are pre-determined. After training begins, each training image frame is input into the model. Then, the model obtains the feature points of each training image frame; these are the predicted feature points. Finally, the difference between the feature points of each training image frame and the corresponding true feature points is calculated using the objective loss function. If the difference between the two sets of feature points for a given training image frame is within a acceptable range, it is considered a qualified training image frame; otherwise, it is considered an unqualified training image frame. If only a small number of qualified training image frames exist in the batch, the parameters of the training model are adjusted, and training is repeated with another batch of training image frames until a large number of qualified training image frames are obtained, thus producing the neural network used for semantic detection.

[0075] It should also be understood that, in this embodiment, the pose of the current image frame usually refers to the pose of the terminal device in the three-dimensional coordinate system corresponding to the vector map when the current image frame is captured. Similarly, the pose of other image frames refers to the pose of the terminal device in the three-dimensional coordinate system corresponding to the vector map when other image frames are captured, and so on.

[0076] It should also be understood that the process of the previous optimization can be referenced to the process of the current optimization, and similarly, the process of the next optimization can also be referenced to the process of the current optimization, and so on.

[0077] It should also be understood that, among all the image frames captured by the terminal device, the pose of the first image frame can be obtained through the terminal device's global positioning system (GPS) and used as the object for the first optimization.

[0078] 202. Based on the pose of the current image frame, obtain the first map point that matches the first feature point from the vector map.

[0079] After obtaining the pose of the current image frame in step 201, the initial pose value of the current image frame for the current sub-optimization is obtained. Based on this pose, a first map point matching the first feature point can be obtained from the vector map preset inside the terminal device. Specifically, the first map point matching the first feature point can be obtained in various ways, which will be described below:

[0080] In one possible implementation, a region containing the terminal device can be defined in the vector map, for example, a 150m × 150m area. Based on the pose of the current image frame, the positions of multiple map points within this region in the corresponding 3D coordinate system of the vector map are calculated using coordinate transformation. This yields the positions of these map points in the current image frame. This process is equivalent to projecting multiple map points within the region onto the current image frame based on its pose. Since the first feature point of the current image frame includes feature points of various objects, and the multiple map points within this region also include map points of various objects, a nearest neighbor algorithm can be used to calculate the positions of the first feature point and these map points in the current image frame. This allows for matching the first feature point with these map points within the same object category, thereby identifying the first map point that matches the first feature point. For example, in a vector map, an object like a lamppost can be represented by a straight line formed by multiple map points. This straight line, projected onto the current image frame, remains a straight line, referred to hereafter as the projected line. Objects like light poles are represented in the current image frame by feature points at their two ends, which will be referred to as endpoints. Therefore, when light poles A, B, and C are projected onto the current image frame from a vector map, to determine which light pole matches light pole D in the current image frame, we can calculate the average distances from the two endpoints of light pole D to the projected lines of light pole A, B, and C. The light pole with the smallest average distance is identified as the matching light pole D, and its map point is then matched with the feature point of light pole D. Similarly, in a vector map, objects like signs (or traffic lights) can be represented by a rectangular frame formed by multiple map points. This rectangular frame is also projected onto the current image frame as a rectangle, and similarly, signs are represented in the current image frame by a rectangular frame formed by multiple feature points. So, when signs X and Y are projected onto the current image frame from a vector map, to determine which sign matches sign Z in the current image frame, we can calculate the average distances from the four vertices of sign Z's rectangle to the projected straight lines of the two parallel sides of sign X's rectangle, and the average distances from the four vertices of sign Z's rectangle to the projected straight lines of the two parallel sides of sign Y's rectangle. The sign with the smallest average distance is identified as the sign that matches sign Z, and its map points are then matched with the feature points of sign Z. Similarly, in a vector map, lane lines can be represented by a straight line formed by multiple map points. This straight line, projected onto the current image frame, remains a straight line. Lane lines are represented in the current image frame by feature points at their two ends.When lane lines E and F are projected onto the current image frame from the vector map, to determine which lane line matches lane line G in the current image frame, the average distance from the two endpoints of lane line G to the projected lines of lane line E, and the degree of overlap between the projected lines of lane line G and lane line E, are calculated. Similarly, the average distance from the two endpoints of lane line G to the projected lines of lane line F, and the degree of overlap between the projected lines of lane line G and lane line F, are calculated. The lane line with the smallest overall distance (for example, if the overlap of lane line E and lane line F is the same, then the lane line with the smaller overall distance is the lane line with the smaller overall distance, etc.) is determined as the lane line that matches lane line G. The map points of this lane line are then matched with the feature points of lane line G.

[0081] Specifically, the calculation process for the degree of overlap is as follows: Figure 6 As shown, Figure 6 This is a schematic diagram of calculating the degree of overlap provided in an embodiment of this application. Let there be a lane line JK in the current image frame, and a projection line PQ of the lane line in the vector map. The foot of the perpendicular from endpoint J to the projection line PQ is U, and the foot of the perpendicular from endpoint K to the projection line PQ is V. Therefore, the degree of overlap between lane line JK and projection line PQ is as shown in formula (5):

[0082]

[0083] In the above formula, l overlap d represents the degree of overlap. UV Let a be the length of line segment UV. UV∩PQ Let be the length of the overlapping portion between line segment UV and line segment PQ. Based on formula (5), it can be seen that... Figure 6 The degree of overlap from left to right is 1, a, etc. PV / a UV a PQ / a UV And 0.

[0084] In another possible implementation, a region containing the terminal device can be defined in the vector map. Then, based on the pose of the current image frame, the position of the first feature point in the current image frame is calculated using coordinate transformation. This yields the position of the first feature point in the corresponding 3D coordinate system of the vector map. This process is equivalent to projecting the first feature point of the current image frame onto the corresponding 3D coordinate system of the vector map based on the pose of the current image frame. Since the first feature point of the current image frame includes feature points of various objects, and multiple map points within this region of the vector map also contain map points of various objects, a nearest neighbor algorithm can be used to calculate the positions of the first feature point and these map points in the corresponding 3D coordinate system of the vector map. This allows for matching the first feature point with these map points of the same type of object, thereby identifying the first map point that matches the first feature point within this subset of map points.

[0085] In another possible implementation, a region containing the terminal device can be defined in the vector map. Based on the pose of the current image frame, coordinate transformation calculations are performed on the positions of multiple map points within this region in the corresponding 3D coordinate system of the vector map, thus obtaining the positions of these map points in the 3D coordinate system corresponding to the terminal device. Simultaneously, based on the pose of the current image frame, coordinate transformation calculations are also performed on the position of the first feature point in the current image frame, thus obtaining the position of the first feature point in the corresponding 3D coordinate system of the terminal device. Since the first feature point of the current image frame includes feature points of various objects, and the multiple map points within this region of the vector map also include map points of various objects, a nearest neighbor algorithm can be used to calculate the positions of the first feature point and these map points in the corresponding 3D coordinate system of the terminal device. This allows for matching the first feature point with these map points of the same type of object, thereby identifying the first map point that matches the first feature point within this subset of map points.

[0086] All three implementation methods described above involve setting the feature points of all objects in the current image frame and the map points of all objects in the divided regions of the vector map into a single coordinate system to complete the matching between feature points and map points. Furthermore, feature points and map points of certain categories of objects (e.g., traffic lights, lampposts, signs, etc.) can be matched within a single coordinate system (e.g., the current image frame), while feature points and map points of other categories of objects (e.g., lane lines) can be matched within a different coordinate system (e.g., the 3D coordinate system corresponding to the terminal device).

[0087] It is worth noting that after obtaining the first map point matching the first feature point, it is equivalent to obtaining the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system. This distance includes at least one of the following: 1. The distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame. 2. The distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the first map point in the three-dimensional coordinate system corresponding to the vector map. 3. The distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device.

[0088] To further understand the above explanation, an example will be used below. Suppose the current image frame contains a lamppost W1, a sign W2, lane lines W3 and W4. In the vector map, lamppost W5 matches lamppost W1, sign W6 matches sign W2, lane lines W7 match lane lines W3, and lane lines W8 match lane lines W4. When the meaning of the first coordinate system is different, the following situations will exist:

[0089] Case 1: When the first coordinate system is the current image frame, the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system is the distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame. This includes: the average distance between the two endpoints of the lamp post W1 and the projected straight line of the lamp post W5 after being projected onto the current image frame; the average distance between the four vertices of the rectangular frame of the sign W2 and the projected straight lines of the two parallel sides of the rectangular frame of the sign W6; the combined distance between lane line W3 and lane line W7; and the combined distance between lane line W4 and lane line W8.

[0090] Scenario 2: When the first coordinate system includes the current image frame and the three-dimensional coordinate system corresponding to the terminal device, the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system includes: the distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame, and the distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device. Specifically, the distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame includes: the average distance from the two endpoints of the lamp post W1 to the projected straight line of the lamp post W5 after projection onto the current image frame, and the average distance from the four vertices of the rectangular frame of the signpost W2 to the projected straight lines of the two parallel sides of the rectangular frame of the signpost W6. The distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device includes: the combined distance between lane line W3 and lane line W7 after projection onto the three-dimensional coordinate system corresponding to the terminal device, and the combined distance between lane line W4 and lane line W8.

[0091] Similarly, there are also cases three (when the first coordinate system is the three-dimensional coordinate system corresponding to the vector map), four (when the first coordinate system is the three-dimensional coordinate system corresponding to the terminal device), five (when the first coordinate system includes the current image frame and the three-dimensional coordinate system corresponding to the vector map), six (when the first coordinate system includes the three-dimensional coordinate system corresponding to the terminal device and the three-dimensional coordinate system corresponding to the vector map), and seven (when the first coordinate system includes the current image frame, the three-dimensional coordinate system corresponding to the terminal device, and the three-dimensional coordinate system corresponding to the vector map). For the explanations of cases one and two, please refer to the relevant explanations, which will not be repeated here.

[0092] 203. Based on the poses of other image frames after the previous optimization, obtain the second map point that matches the second feature point from the vector map.

[0093] After obtaining the poses of other image frames after the previous optimization in step 201, the initial pose values ​​of other image frames used for the current optimization are obtained. Based on these poses, a second map point matching the second feature point can be obtained from the vector map inside the terminal device.

[0094] After obtaining the second map point that matches the second feature point, it is equivalent to obtaining the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system. This distance includes at least one of the following: 1. The distance between the position of the second feature point in other image frames and the position of the second map point in other image frames. 2. The distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the second map point in the three-dimensional coordinate system corresponding to the vector map. 3. The distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the second map point in the three-dimensional coordinate system corresponding to the terminal device.

[0095] It should be noted that the process of obtaining the second map point can be found in the relevant explanation section of step 202 regarding the process of obtaining the first map point, and will not be repeated here. Furthermore, the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system can also be found in the relevant explanation section of step 202 regarding the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system, and will not be repeated here.

[0096] 204. Adjust the pose of the current image frame according to the objective function to obtain the pose of the current image frame after the current optimization, which is used as the positioning result of the terminal device. The objective function includes the first matching error between the first feature point and the first map point, and the second matching error between the second feature point and the second map point.

[0097] After obtaining the first map point that matches the first feature point and the second map point that matches the second feature point, the pose of the current image frame can be adjusted based on the objective function constructed according to the first matching error between the first feature point and the first map point, and the second matching error between the second feature point and the second map point. That is, the pose of the current image frame is optimized to obtain the pose of the current image frame after the current optimization, which is used as the positioning result of the terminal device.

[0098] Specifically, the initial value of the first matching error can be obtained by first determining the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system. Using the example above, the initial value of the first matching error can be obtained through formula (6):

[0099] Huber ε1 (d pp +d pl )+β*Huber ε2 (d pH )

[0100]

[0101] In the above formula, the first matching error is obtained through Huber. ε1 and Huber ε2 Confirmed, Huber ε1 Let ε be the Huber loss function with parameter ε1. ε2 Let d be the Huber loss function with parameter ε², where β is a preset parameter. pp This represents the distance to objects like lampposts in the current image frame. Let be the distance between the i-th lamp post in the current image frame and its matching lamp post. Let d be the distance from the two endpoints of the i-th lamp post to the projected straight lines of the matching lamp post. pl This refers to the distances between the two types of objects (traffic lights or signs) in the current image frame. Let be the distance between the i-th traffic light (or sign) in the current image frame and its matching traffic light (or sign). and Let d be the distance from the four vertices of the i-th traffic light (or sign) to the projected straight lines of the two parallel sides within the rectangle of the matching traffic light. pH This is the combined distance to objects such as lane lines in the current image frame. This is the combined distance between the i-th lane line and its matching lane line in the current image frame. Let be the distance between the i-th lane line and its matching lane line. Let represent the degree of overlap between the i-th lane line and its matching lane line. Let be the distances from the two endpoints of the i-th lane line to the projected straight lines of the matching lane lines.

[0102] Furthermore, the initial value of the second matching error can be calculated based on the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system. As in the example above, the initial value of the second matching error can also be obtained using formula (6), which will not be elaborated here.

[0103] After obtaining the initial values ​​of the first and second matching errors, these initial values ​​can be input into the objective function, and the objective function can be iteratively solved until the preset iteration conditions are met, thus obtaining the pose of the current image frame after the current optimization. Based on formula (6), the objective function can be expressed by formula (7):

[0104]

[0105] In the above formula, in the current image frame and other image frames, Let be the distance corresponding to objects like lampposts in the i-th image frame. Let be the distance between the two types of objects (traffic lights or signs) in the i-th image frame. Let be the distance to objects like lane lines in the i-th image frame.

[0106] It should be understood that this embodiment is only illustrated by formulas (6) and (7) and does not limit the calculation method of the matching error or the expression method of the objective function.

[0107] During the iterative solution of the objective function, after the first iteration, the initial values ​​of the first and second matching errors are input into the objective function for solution. This yields the pose of the current image frame and the poses of other image frames obtained in the first iteration. Then, based on the poses of the current and other image frames obtained in the first iteration, the inter-frame pose difference is calculated. If the difference between this inter-frame pose difference and the inter-frame pose difference calculated by the terminal device's odometer is less than a preset threshold, it indicates that the objective function has converged, and the iteration stops. The pose of the current image frame obtained in the first iteration is then used as the pose of the current optimized image frame. If the difference between this inter-frame pose difference and the inter-frame pose difference calculated by the terminal device's odometer is greater than or equal to the preset threshold, a second iteration is performed.

[0108] In the second iteration, the first map point matching the first feature point can be re-determined based on the pose of the current image frame obtained in the first iteration (i.e., step 202 is re-executed), and the second map point matching the second feature point can be re-determined based on the poses of other image frames obtained in the first iteration (i.e., step 203 is re-executed). Then, the first iteration value of the first matching error between the first feature point and the first map point, and the first iteration value of the second matching error between the second feature point and the second map point are calculated. Subsequently, the first iteration values ​​of the first matching error and the first iteration values ​​of the second matching error are output to the objective function for solving, thereby obtaining the pose of the current image frame obtained in the second iteration and the poses of other image frames obtained in the second iteration. Next, based on the pose of the current image frame obtained in the second iteration and the poses of other image frames obtained in the second iteration, the inter-frame pose difference obtained in the second iteration is calculated. If the difference between the inter-frame pose difference and the inter-frame pose difference calculated by the odometer of the terminal device is less than a preset threshold, the iteration stops, and the pose of the current image frame obtained in the second iteration is used as the pose of the current image frame after the current optimization. If the difference between the inter-frame pose difference and the inter-frame pose difference calculated by the odometer of the terminal device is greater than or equal to the preset threshold, the third iteration is performed until the number of iterations is equal to the preset number. At this point, the objective function is considered to have converged, and the pose of the current image frame obtained in the last iteration is used as the pose of the current image frame after the current optimization.

[0109] In this embodiment, after obtaining the current image frame and other image frames preceding it, a first map point matching the first feature point of the current image frame and a second map point matching the second feature point of other image frames preceding it can be obtained from the vector map. Then, the pose of the current image frame can be adjusted based on a target function constructed according to the first matching error between the first feature point and the first map point, and the second matching error between the second feature point and the second map point, to obtain the pose of the current image frame after the current optimization. In the aforementioned process, since the target function includes both the matching error between the feature point of the current image frame and the map point of the vector map, and the matching error between the feature point of other image frames and the map point of the vector map, adjusting the pose of the current image frame through this target function not only considers the impact of the current image frame on the optimization process of the current image frame's pose, but also considers the impact of other image frames on the optimization process of the current image frame's pose. That is, it considers the correlation between the current image frame and other image frames, and the factors considered are more comprehensive. Therefore, the positioning result of the terminal device obtained based on this method has higher accuracy.

[0110] Furthermore, in related technologies, the objective function is constructed solely based on the matching error between feature points of the current image frame and map points of the vector map. Since the content presented in the current image frame is limited, when selecting map points to match the feature points of the current image frame, the map points are often sparse and overlapping. Therefore, when iteratively solving the objective function, the matching error between feature points and map points cannot be minimized sufficiently, thus affecting the accuracy of the localization result. In this embodiment, the objective function is constructed using the first matching error between the first feature point of the current image frame and the first map point of the vector map, and the second matching error between the second feature points of other image frames and the second map points of the vector map. Since the content presented by multiple image frames usually has a high degree of differentiation, the situation of sparse and overlapping map points can be avoided. Therefore, when iteratively solving the objective function (jointly optimizing the poses of multiple image frames), the first and second matching errors can be minimized sufficiently, thereby improving the accuracy of the localization result.

[0111] Furthermore, the pose of the current image frame obtained through hierarchical sampling can be used as the initial pose value of the current image frame for the current sub-optimization, thereby improving the convergence speed and robustness of the current sub-optimization.

[0112] The above is a detailed description of the terminal device positioning method provided in the embodiments of this application. The following will introduce the terminal device positioning device provided in the embodiments of this application. Figure 7A schematic diagram of the terminal device positioning device provided in the embodiments of this application is shown below. Figure 7 As shown, the device includes:

[0113] The first matching module 701 is used to obtain a first map point from the vector map that matches the first feature point of the current image frame captured by the terminal device;

[0114] The second matching module 702 is used to obtain a second map point from the vector map that matches the second feature points of other image frames preceding the current image frame;

[0115] The adjustment module 703 is used to adjust the pose of the terminal device when capturing the current image frame according to the objective function, so as to obtain the pose of the terminal device when capturing the current image frame after the current adjustment, which is used as the positioning result of the terminal device. The objective function includes the first matching error between the first feature point and the first map point, and the second matching error between the second feature point and the second map point.

[0116] In one possible implementation, the device further includes: an acquisition module 700, configured to acquire a first feature point of the current image frame, second feature points of other image frames preceding the current image frame, the pose of the terminal device when capturing the current image frame, and the pose of the terminal device when capturing other image frames after the last adjustment; a first matching module 701, configured to acquire a first map point matching the first feature point from the vector map based on the pose of the terminal device when capturing the current image frame; and a second matching module 702, configured to acquire a second map point matching the second feature point from the vector map based on the pose of the terminal device when capturing other image frames after the last adjustment.

[0117] In one possible implementation, the adjustment module 703 is used to: calculate an initial value of a first matching error based on the distance between the position of a first feature point in the first coordinate system and the position of a first map point in the first coordinate system; calculate an initial value of a second matching error based on the distance between the position of a second feature point in the second coordinate system and the position of a second map point in the second coordinate system; and iteratively solve the objective function based on the initial values ​​of the first and second matching errors until a preset iteration condition is met, thereby obtaining the pose of the terminal device when capturing the current image frame after the current adjustment.

[0118] In one possible implementation, the distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system includes at least one of the following: the distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame; the distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the first map point in the three-dimensional coordinate system corresponding to the vector map; or the distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device.

[0119] In one possible implementation, the distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system includes at least one of the following: the distance between the position of the second feature point in other image frames and the position of the second map point in other image frames; the distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the second map point in the three-dimensional coordinate system corresponding to the vector map; or the distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the second map point in the three-dimensional coordinate system corresponding to the terminal device.

[0120] In one possible implementation, the iteration condition is as follows: for any iteration, if the difference between the inter-frame pose difference obtained in this iteration and the inter-frame pose difference calculated by the terminal device is less than a preset threshold, then the iteration stops. The inter-frame pose difference obtained in this iteration is determined based on the pose of the terminal device when capturing the current image frame and the pose of the terminal device when capturing other image frames. If the difference is greater than or equal to the threshold, then the next iteration is executed until the number of iterations equals the preset number.

[0121] In one possible implementation, the number of other image frames is determined based on the speed of the terminal device.

[0122] In one possible implementation, the acquisition module 700 is used to: calculate the predicted pose of the terminal device when capturing the current image frame based on the pose of the terminal device when capturing other image frames after the last adjustment and the inter-frame pose difference calculated by the terminal device; and perform layered sampling on the predicted pose of the terminal device when capturing the current image frame to obtain the pose of the terminal device when capturing the current image frame.

[0123] In one possible implementation, if the pose of the terminal device when capturing the current image frame includes abscissa, ordinate, and heading angle, then the acquisition module 700 is used to: acquire the position of the third map point in the three-dimensional coordinate system corresponding to the vector map and the position of the first feature point in the current image frame; keep the heading angle of the predicted pose of the terminal device when capturing the current image frame unchanged, change the abscissa and ordinate of the predicted pose of the terminal device when capturing the current image frame to obtain a first candidate pose; transform the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the first candidate pose to obtain the third map point. The position of the first feature point in the current image frame is determined by: keeping the x and y coordinates of the predicted pose of the terminal device unchanged while changing the heading angle of the predicted pose; transforming the position of the first feature point in the current image frame based on the second candidate pose to obtain the position of the first feature point in the image coordinate system; and determining the pose of the terminal device when capturing the current image frame from the combination of the first and second candidate poses based on the distance between the position of the third map point in the image coordinate system and the position of the first feature point in the image coordinate system. This pose sampling method effectively reduces the computational load required during pose sampling.

[0124] In one possible implementation, if the pose of the terminal device when capturing the current image frame includes x-coordinate, y-coordinate, vertical coordinate, heading angle, roll angle, and pitch angle, then the acquisition module 700 is used to: acquire the position of the third map point in the three-dimensional coordinate system corresponding to the vector map and the position of the first feature point in the current image frame; keep the heading angle, roll angle, pitch angle, and vertical coordinate of the predicted pose of the terminal device when capturing the current image frame unchanged, and change the x-coordinate and y-coordinate of the predicted pose of the terminal device when capturing the current image frame to obtain a first candidate pose; transform the position of the third map point in the three-dimensional coordinate system corresponding to the vector map according to the first candidate pose to obtain the position of the third map point in a preset image coordinate system; keep the x-coordinate, y-coordinate, vertical coordinate, roll angle, and pitch angle of the predicted pose of the terminal device when capturing the current image frame unchanged, and change the heading angle of the predicted pose of the terminal device when capturing the current image frame. The process involves several steps: First, a second candidate pose is obtained. Then, the position of the first feature point in the current image frame is transformed based on the second candidate pose to obtain its position in the image coordinate system. Next, a third candidate pose is determined from the combination of the first and second candidate poses based on the distance between the position of the third map point in the image coordinate system and the position of the first feature point. While keeping the x-coordinate, y-coordinate, heading angle, and roll angle of the predicted pose of the third candidate pose constant, the pitch angle and vertical coordinate of the third candidate pose are changed to obtain a fourth candidate pose. Then, the position of the third map point in the corresponding 3D coordinate system of the vector map is transformed based on the fourth candidate pose to obtain its position in the current image frame. Finally, the pose of the terminal device when capturing the current image frame is determined from the fourth candidate pose based on the distance between the position of the first feature point in the current image frame and the position of the third map point in the current image frame. This pose sampling method effectively reduces the computational load required during the pose sampling process.

[0125] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0126] Figure 8 Another structural schematic diagram of the terminal device positioning device provided in the embodiments of this application is shown. Figure 8 As shown, one embodiment of the computer in this application may include one or more central processing units 801, memory 802, input / output interfaces 803, wired or wireless network interfaces 804, and power supplies 805.

[0127] The memory 802 can be temporary or persistent storage. Furthermore, the central processing unit 801 can be configured to communicate with the memory 802 and execute a series of instructions stored in the memory 802 on the computer.

[0128] In this embodiment, the central processing unit 801 can execute the aforementioned... Figure 2 The method steps in the illustrated embodiments will not be described in detail here.

[0129] In this embodiment, the specific functional module division in the central processing unit 801 can be the same as described above. Figure 7 The division of modules such as the acquisition module, the first matching module, the second matching module, and the optimization module described in the text is similar, and will not be repeated here.

[0130] This application also relates to a computer storage medium, including computer-readable instructions, which, when executed, implement as follows: Figure 2 The method described.

[0131] This application also relates to a computer program product containing instructions that, when run on a computer, cause the computer to perform actions such as... Figure 2 The method described.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A terminal device positioning method, characterized by, The method comprises: obtaining, from a vector map, a first map point matched with a first feature point of a current image frame photographed by a terminal device; obtaining, from the vector map, a second map point matched with a second feature point of another image frame before the current image frame; adjusting a pose of the terminal device when photographing the current image frame to obtain a current time-adjusted pose of the terminal device when photographing the current image frame as a positioning result of the terminal device; wherein the current time-adjusted pose of the terminal device when photographing the current image frame is obtained by iteratively solving a target function by using an initial value of a first matching error and an initial value of a second matching error; the initial value of the first matching error is obtained according to a distance between a position of the first feature point in a first coordinate system and a position of the first map point in the first coordinate system, and the initial value of the second matching error is obtained according to a distance between a position of the second feature point in a second coordinate system and a position of the second map point in the second coordinate system; the target function comprises the first matching error between the first feature point and the first map point and the second matching error between the second feature point and the second map point, the initial value of the first matching error is an initial value of the first matching error included in the target function, and the initial value of the second matching error is an initial value of the second matching error included in the target function.

2. The method of claim 1, wherein: adjusting the pose of the terminal device when photographing the current image frame according to the target function.

3. The method of claim 1, wherein, The method further comprises: obtaining the first feature point of the current image frame, the second feature point of the another image frame before the current image frame, the pose of the terminal device when photographing the current image frame, and a last time-adjusted pose of the terminal device when photographing the another image frame; the obtaining, from the vector map, the first map point matched with the first feature point comprises: obtaining, from the vector map, the first map point matched with the first feature point according to the pose of the terminal device when photographing the current image frame; the obtaining, from the vector map, the second map point matched with the second feature point comprises: obtaining, from the vector map, the second map point matched with the second feature point according to the last time-adjusted pose of the terminal device when photographing the another image frame.

4. The method according to claim 2 or 3, characterized in that, adjusting the pose of the terminal device when photographing the current image frame according to the target function to obtain the current time-adjusted pose of the terminal device when photographing the current image frame comprises: calculating the initial value of the first matching error according to a distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system; calculating the initial value of the second matching error according to a distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system; The target function is iteratively solved according to the initial value of the first matching error and the initial value of the second matching error until a preset iteration condition is met, to obtain a pose of the terminal device when shooting the current image frame after current adjustment.

5. The method of claim 4, wherein, The distance between the position of the first feature point in the first coordinate system and the position of the first map point in the first coordinate system includes at least one of the following: The distance between the position of the first feature point in the current image frame and the position of the first map point in the current image frame; The distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the first map point in the three-dimensional coordinate system corresponding to the vector map; Or The distance between the position of the first feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the first map point in the three-dimensional coordinate system corresponding to the terminal device.

6. The method of claim 4, wherein, The distance between the position of the second feature point in the second coordinate system and the position of the second map point in the second coordinate system includes at least one of the following: The distance between the position of the second feature point in the other image frame and the position of the second map point in the other image frame; The distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the vector map and the position of the second map point in the three-dimensional coordinate system corresponding to the vector map; Or The distance between the position of the second feature point in the three-dimensional coordinate system corresponding to the terminal device and the position of the second map point in the three-dimensional coordinate system corresponding to the terminal device.

7. The method of claim 4, wherein, The iteration condition is that: for any iteration, if the difference between the frame pose difference obtained by the iteration and the frame pose difference calculated by the terminal device is less than a preset threshold value, the iteration is stopped, the frame pose difference is determined according to the pose of the terminal device when shooting the current image frame obtained by the iteration and the pose of the terminal device when shooting the other image frame obtained by the iteration, and the frame pose difference is the pose difference between two adjacent image frames shot by the terminal device in the current image frame and the other image frame; If the difference is greater than or equal to the threshold value, the next iteration is performed until the number of iterations is equal to a preset number.

8. The method according to any one of claims 1 to 3, characterized in that, The number of the other image frames is determined according to the speed of the terminal device.

9. The method of claim 3, wherein, The pose of the terminal device when shooting the current image frame includes: According to the pose of the terminal device when shooting the other image frame after the last adjustment and the frame pose difference calculated by the terminal device, a predicted pose of the terminal device when shooting the current image frame is calculated; The predicted pose of the terminal device when shooting the current image frame is sampled in layers to obtain the pose of the terminal device when shooting the current image frame.

10. A terminal device positioning apparatus characterized by comprising: The device includes: A first matching module configured to acquire, from a vector map, a first map point matched with a first feature point of a current image frame shot by a terminal device; A second matching module configured to acquire, from the vector map, a second map point matched with a second feature point of an other image frame before the current image frame; an adjusting module configured to adjust a pose of the terminal device when the terminal device captures the current image frame to obtain a pose of the terminal device when the terminal device captures the current image frame after a current adjustment, as a positioning result of the terminal device; wherein the pose of the terminal device when the terminal device captures the current image frame after the current adjustment is obtained by iteratively solving a target function using an initial value of a first matching error and an initial value of a second matching error; the initial value of the first matching error is obtained according to a distance between a position of the first feature point in a first coordinate system and a position of the first map point in the first coordinate system, and the initial value of the second matching error is obtained according to a distance between a position of the second feature point in a second coordinate system and a position of the second map point in the second coordinate system; the target function includes a first matching error between the first feature point and the first map point, and a second matching error between the second feature point and the second map point, the initial value of the first matching error is an initial value of the first matching error included in the target function, and the initial value of the second matching error is an initial value of the second matching error included in the target function.

11. A terminal device positioning apparatus characterized by comprising: a terminal device positioning apparatus comprising a memory and a processor; the memory stores a code, and the processor is configured to execute the code; when the code is executed, the terminal device positioning apparatus performs the method according to any one of claims 1 to 9.

12. A computer storage medium, characterized in that The computer storage medium stores a computer program, which is executed by a computer to enable the computer to implement the method according to any one of claims 1 to 9.

13. A computer program product, characterised in that, The computer program product stores instructions, which are executed by a computer to enable the computer to implement the method according to any one of claims 1 to 9.

14. A vehicle comprising the terminal device positioning apparatus according to claim 11.

Images