Positioning method and system and electronic equipment

Abstract

The invention discloses a positioning method and system and electronic equipment, and the method comprises the steps: obtaining the inertial information of a to-be-positioned object, determining the first pose transformation information between every two frames of inertial data based on the inertial information, obtaining a frame image of the to-be-positioned object, determining the position information of a visual feature point based on the frame image, and obtaining the radar point cloud information of the to-be-positioned object. And determining distortion-removed first point cloud information and angular point and plane point feature information based on the radar point cloud information, and determining first pose information of the to-be-positioned object based on the position information of the visual feature points, the first point cloud information, the angular point and plane point feature information and the first pose transformation information. According to the scheme, the inertial information, the frame image and the radar point cloud information of the to-be-positioned object are fused, so that the situation that the frame image and the radar point cloud information are shielded to cause lack of pose features is overcome, and the situation that the trajectory jumps due to lack of the pose features is avoided.

Description

Technical Field

[0001] This application relates to the field of vehicle positioning, and particularly to a positioning method, system and electronic device. Background Art

[0002] In open urban roads, to achieve autonomous driving of a vehicle, it is necessary to determine the position and attitude of the vehicle in the world coordinate system.

[0003] Usually, RTK (RealTime Kinematic) installed on the vehicle or a pre-established point cloud map is used to determine the position and attitude of the vehicle in the world coordinate system. Among them, RTK is a differential measurement technology that realizes fast and high-precision positioning functions by synchronous observations of a reference station and a rover station and using carrier phase observation values.

[0004] However, whether the position and attitude of the vehicle are determined by RTK or by the point cloud map, there may be cases of trajectory jumps, which are difficult to meet the smoothness requirements of the autonomous driving trajectory and reduce the user experience. Summary of the Invention

[0005] In view of this, this application provides a positioning method, system and electronic device, and its specific solutions are as follows:

[0006] A positioning method includes:

[0007] Obtain the inertial information of the object to be positioned, and determine the first pose transformation information between every two frames of inertial data based on the inertial information;

[0008] Obtain the frame image of the object to be positioned, and determine the position information of the visual feature points based on the frame image;

[0009] Obtain the radar point cloud information of the object to be positioned, and determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information;

[0010] Determine the first pose information of the object to be positioned based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

[0011] Further, the determining the first pose information of the object to be positioned based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information includes:

[0012] Determine the first depth data of the visual feature points based on the first point cloud information;

[0013] Determine the second pose transformation information between two consecutive frames of radar data based on the corner point and plane point feature information and the first pose transformation information between every two frames of inertial data;

[0014] Determine the third pose transformation information between two consecutive frames of images based on the visual feature points including the position information and the first depth data and the inertial information;

[0015] Determine the first pose information of the object to be located based on the second pose transformation information and the third pose transformation information.

[0016] Further, the determining the first pose information of the object to be located based on the second pose transformation information and the third pose transformation information includes:

[0017] Determine the first pose information of the object to be located based on the weight of the second pose transformation information and the weight of the third pose transformation information.

[0018] Further, the determining the first pose information of the object to be located based on the second pose transformation information and the third pose transformation information includes:

[0019] Based on the frame image of the object to be located, determine the frame image of the object to be located in the historical data that matches the frame image;

[0020] Determine the fourth pose transformation information of the object to be located based on the comparison between the current frame image of the object to be located and the frame image of the object to be located in the historical data;

[0021] Determine the first pose information of the object to be located according to the factor graph based on the fourth pose transformation information, the second pose transformation information and the third pose transformation information of the object to be located.

[0022] Further, it further includes:

[0023] Adjust the current offset information of the inertial measurement unit based on the pose information of the object to be located within a preset time period;

[0024] Obtain the inertial information of the object to be located based on the real-time inertial measurement unit after adjusting the current offset information.

[0025] Further, the determining the first depth data of the visual feature points based on the first point cloud information includes:

[0026] Determine the second depth data in the position information of the visual feature points based on the inertial information of the object to be located and the frame image of the object to be located;

[0027] Adjust the second depth data to the first depth data of the visual feature points based on the first point cloud information.

[0028] Further, it further includes:

[0029] If it is determined that the number of the visual feature points is less than a first preset value, and / or it is determined that the current offset information is greater than a second preset value, obtain the frame image of the object to be located again.

[0030] A positioning system, including:

[0031] A first obtaining unit, configured to obtain the inertial information of the object to be located, and determine the first pose transformation information between every two frames of inertial data based on the inertial information;

[0032] A second obtaining unit, configured to obtain the frame image of the object to be located, and determine the position information of the visual feature points based on the frame image;

[0033] A third obtaining unit, configured to obtain the radar point cloud information of the object to be located, and determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information;

[0034] A determining unit, configured to determine the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

[0035] An electronic device, including:

[0036] An inertial measurement unit, configured to obtain the inertial information of the object to be located;

[0037] An image acquisition device, configured to obtain the frame image of the object to be located;

[0038] A radar, configured to obtain the radar point cloud information of the object to be located;

[0039] A processor, configured to determine the first pose transformation information between every two frames of inertial data based on the inertial information; determine the position information of the visual feature points based on the frame image; determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information; determine the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

[0040] A readable storage medium, configured to store at least one set of instruction sets;

[0041] The instruction sets are used to be called and at least execute the positioning method as described in any one of the above.

[0042] As can be seen from the above technical solution, the positioning method, system, and electronic device disclosed in this application obtain the inertial information of the object to be positioned, determine the first pose transformation information between every two frames of inertial data based on the inertial information, obtain the frame images of the object to be positioned, determine the position information of the visual feature points based on the frame images, obtain the radar point cloud information of the object to be positioned, determine the first point cloud information after removing distortion and the corner and plane point feature information based on the radar point cloud information, and determine the first pose information of the object to be positioned based on the position information of the visual feature points, the first point cloud information, the corner and plane point feature information, and the first pose transformation information. This solution overcomes the situation where pose features are missing due to occlusion in the frame images and radar point cloud information by fusing the inertial information, frame images, and radar point cloud information of the object to be positioned, and avoids the situation of trajectory jumps caused by the lack of pose features. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a flowchart of a positioning method disclosed in an embodiment of the present application;

[0045] Figure 2 It is a flowchart of a positioning method disclosed in an embodiment of the present application;

[0046] Figure 3 It is a flowchart of a positioning method disclosed in an embodiment of the present application;

[0047] Figure 4 It is a schematic structural diagram of a positioning system disclosed in an embodiment of the present application;

[0048] Figure 5 It is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0050] The present application discloses a positioning method, and its flowchart is as Figure 1 shown, including:

[0051] Step S11: Obtain the inertial information of the object to be located, and determine the first pose transformation information between every two frames of inertial data based on the inertial information.

[0052] Step S12: Obtain the frame image of the object to be located, and determine the position information of the visual feature points based on the frame image.

[0053] Step S13: Obtain the radar point cloud information of the object to be located, and determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information.

[0054] Step S14: Determine the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

[0055] On urban open roads, there are both scenes with rich texture features and scenes with serious perspective occlusion and difficult-to-detect feature points. Among them, the camera is more likely to detect the positioning of vehicles or other objects to be located in scenes with rich features and simple structures. The detection accuracy of the radar is higher. However, the inertial measurement unit can remove the motion distortion of the point cloud, and in an environment lacking features, it can output the pose of the object to be located based on inertial data to help the camera and radar restore the detection of position information.

[0056] Specifically, the positioning method disclosed in this embodiment at least includes: an inertial measurement unit for obtaining the inertial information of the object to be located. The inertial measurement unit (Inertial measurement unit, IMU) is used to measure the object's three-axis attitude angle (or angular rate) and acceleration; an image acquisition device, such as a camera, for obtaining the frame image of the object to be located; a radar for measuring parameters such as the distance of the target by irradiating pulsed laser on the object to be located.

[0057] After the image acquisition device obtains the frame image of the object to be located, the processor analyzes the frame image to determine the position information of the visual feature points in the frame image. The position information of the visual feature points can be the position information of the object to be located, and the visual feature points are the feature points of the object to be located in the frame image. Specifically, the frame image can be input into the visual feature tracking module to obtain the position information of the visual feature points.

[0058] After the radar obtains the radar point cloud information of the object to be located, the processor performs operations to remove distortion on the radar point cloud information and detect the corner and plane point features in the environment. Specifically, the radar point cloud information is input into the point cloud feature detection module to directly output the first point cloud information with distortion removed and the corner and plane point feature information. Among them, the corner points are the feature points corresponding to the edges and corners of the objects in the environment detected by the radar, and the plane points are the feature points on the planes of the objects in the environment detected by the radar.

[0059] Among them, it takes a certain amount of time for the radar to obtain a certain frame of point cloud, and the total timestamp of the point cloud is set with the first point. If the radar is moving during the scanning of a certain frame of the point cloud, distortion will occur. And removing distortion is usually based on inertial data removal. That is, the frequency of the inertial measurement unit is relatively high, usually exceeding 100HZ, while the frequency of the radar is 10HZ. By using the IMU pre-integration module, high-frequency pose transformations can be obtained. Through interpolation, the pose change of each point of the radar compared to the first point at each moment can be obtained. Through this change, the distortion of the point cloud can be removed.

[0060] The inertial measurement unit IMU obtains the inertial information of the object to be located. This inertial information can be acceleration and angular velocity information. The measurement frequency of the inertial measurement unit is higher than the acquisition frequency of the image acquisition device and the detection frequency of the radar. After the inertial measurement unit obtains the inertial information of the object to be located, the processor determines the first pose transformation information between every two frames of inertial data based on the inertial information, that is, inputs the inertial information into the IMU pre-integration module to obtain the pose transformation between two frames of inertial data.

[0061] After obtaining the position information of the visual feature points related to the image acquisition device, the first point cloud information and the corner and plane point feature information related to the radar, and the first pose transformation information related to the inertial measurement unit IMU, the information related to the above-mentioned image acquisition device, radar and IMU is fused to obtain more accurate positioning data based on the above three positioning devices. The first pose information obtained based on the fusion of the information related to the above three positioning devices is the data of the position and pose of the object to be located obtained after combining the respective advantages of the image acquisition device, radar and inertial measurement unit.

[0062] The positioning method disclosed in this embodiment obtains the inertial information of the object to be positioned, determines the first pose transformation information between every two frames of inertial data based on the inertial information, obtains the frame image of the object to be positioned, determines the position information of the visual feature points based on the frame image, obtains the radar point cloud information of the object to be positioned, determines the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information, and determines the first pose information of the object to be positioned based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information. This solution fuses the inertial information, frame image, and radar point cloud information of the object to be positioned to overcome the situation where pose features are missing due to occlusion in the frame image and radar point cloud information, and avoid the situation of trajectory jumps caused by the lack of pose features.

[0063] This embodiment discloses a positioning method, and its flowchart is as Figure 2 shown, including:

[0064] Step S21: Obtain the inertial information of the object to be positioned, and determine the first pose transformation information between every two frames of inertial data based on the inertial information;

[0065] Step S22: Obtain the frame image of the object to be positioned, and determine the position information of the visual feature points based on the frame image;

[0066] Step S23: Obtain the radar point cloud information of the object to be positioned, and determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information;

[0067] Step S24: Determine the first depth information of the visual feature points based on the first point cloud information;

[0068] Step S25: Determine the second pose transformation information between two consecutive frames of radar data based on the corner point and plane point feature information and the first pose transformation information between every two frames of inertial data;

[0069] Step S26: Determine the third pose transformation information between two consecutive frames of images based on the visual feature points including the position information and the first depth information and the inertial information;

[0070] Step S27: Determine the first pose information of the object to be positioned based on the second pose transformation information and the third pose transformation information.

[0071] When determining the first pose information of the object to be located based on the position information of visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information, it is first necessary to obtain the second pose transformation information between two consecutive frames of radar data based on the above data, and also obtain the third pose transformation information between two consecutive frames of images. Only after that can the first pose information of the object to be located be determined based on the second pose transformation information and the third pose transformation information, that is, the final position and pose of the object to be located.

[0072] Among them, the second pose transformation information between two consecutive frames of radar data, that is, Lidar Inertial Odometry (LIO), and the third pose transformation information between two consecutive frames of images, that is, Visual Inertial Odometry (VIO). Among them, odometry is a standard for measuring the pose from the initial pose to the end pose, and the pose does not need to be transformed into the world coordinate system, and is usually used to express the pose transformation between the front and back frames during positioning.

[0073] Lidar Inertial Odometry is the pose transformation information between two frames of radar data obtained based on laser information and inertial information; Visual Inertial Odometry is the pose transformation information between two frames of images obtained based on image information and inertial information. Thus, the pose characterized by the point cloud detected by the radar is adjusted based on inertial information, and the pose characterized by the feature points in the image collected by the image acquisition device is adjusted based on inertial information, and on this basis, the two are combined to obtain more accurate pose transformation information, that is, the first pose information.

[0074] Specifically, the first depth data of the visual feature points is determined based on the first point cloud data after removing distortion. The frame image obtained by the image acquisition device may only include visual feature point data representing the position, without depth data. It is necessary to extract the depth data from the first point cloud data obtained based on the radar and use it as the first depth data of the visual feature points;

[0075] Alternatively, when the image acquisition device obtains a frame image, the second depth data in the position information of the visual feature points is jointly determined based on the inertial information of the object to be located and the frame image of the object to be located. Since the second depth data is determined based on inertial information, it is not accurate. Therefore, after obtaining the first point cloud data based on the radar, the second depth data needs to be optimized and adjusted based on the first point cloud data to obtain more accurate first depth data compared to the second depth data.

[0076] Specifically, align the image frame with the radar frame. Since the radar scans sparse points, combine multiple frames of radar data to obtain a dense depth map. Project the visual feature points and the radar point cloud onto a unit circle centered on the camera, downsample the depth points and save them using polar coordinates to ensure that the density of the points is constant. Then use a two-dimensional polar coordinate KD-tree to determine the three nearest points around the visual feature points. Finally, the depth of the feature points is calculated as the line connecting the center of the image acquisition device and the feature points. This line intersects the plane formed by the three depth points at a point in the Cartesian coordinate system.

[0077] After determining the first depth data of the visual feature points, determine the pose transformation information between two consecutive frames of images based on the visual feature points including position information and the first depth data and the inertial information, that is, Visual Inertial Odometry (VIO). Obtain more accurate pose transformation information than that detected solely by the image acquisition device using the visual feature points including depth data and the acceleration and angular velocity information detected by the Inertial Measurement Unit (IMU).

[0078] In addition, it is also necessary to determine the second pose transformation information between two consecutive frames of radar data based on the corner point and plane point feature information and the first pose transformation information between every two frames of inertial data. That is, input the corner point and plane point features into the point cloud feature matching module, fuse the results of the IMU pre-integration module, and obtain the pose transformation information between the front and rear frames of radar data, that is, Laser Inertial Odometry (LIO).

[0079] Among them, the frequencies of the image acquisition device for collecting images, the radar for obtaining point cloud information, and the Inertial Measurement Unit for obtaining inertial information are different. Therefore, the moments when the image acquisition device collects frame images, the radar obtains point cloud information, and the Inertial Measurement Unit obtains inertial information are different. Among them, since the Inertial Measurement Unit has the highest frequency of obtaining inertial information, when determining the pose information of the object to be located, align with the time stamp of the Inertial Measurement Unit to ensure the accuracy of the finally determined first pose information.

[0080] After obtaining the second pose transformation information and the third pose transformation information, determine the first pose information of the object to be located based on the second pose transformation information and the third pose transformation information, which can be determined based on the weights of the second pose transformation information and the weights of the third pose transformation information.

[0081] The weights corresponding to the laser inertial odometer (LIO) and the visual inertial odometer (VIO) are different when determining the first pose information of the object to be located. If the LIO has higher accuracy, the weight of the LIO is greater; if the VIO has higher accuracy, the weight of the VIO is greater. The determination of the weight value can be based on the current environment where the object to be located is located, or it can be based on the accuracies of the LIO and the VIO when determining the first pose information in historical data. The final first pose information is determined based on different weights to improve the accuracy of positioning.

[0082] The positioning method disclosed in this embodiment obtains the inertial information of the object to be located, determines the first pose transformation information between every two frames of inertial data based on the inertial information, obtains the frame image of the object to be located, determines the position information of the visual feature points based on the frame image, obtains the radar point cloud information of the object to be located, and determines the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information. The first pose information of the object to be located is determined based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information. This solution fuses the inertial information, frame image, and radar point cloud information of the object to be located to overcome the situation where pose features are missing due to occlusion in the frame image and radar point cloud information, and avoid the situation of trajectory jumps caused by the lack of pose features.

[0083] This embodiment discloses a positioning method, and its flowchart is as Figure 3 shown, including:

[0084] Step S31: Obtain the inertial information of the object to be located, and determine the first pose transformation information between every two frames of inertial data based on the inertial information;

[0085] Step S32: Obtain the frame image of the object to be located, and determine the position information of the visual feature points based on the frame image;

[0086] Step S33: Obtain the radar point cloud information of the object to be located, and determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information;

[0087] Step S34: Determine the first depth data of the visual feature points based on the first point cloud;

[0088] Step S35: Determine the second pose transformation information between two consecutive frames of radar data based on the corner point and plane point feature information and the first pose transformation information between every two frames of inertial data;

[0089] Step S36: Determine the third pose transformation information between two consecutive frames of images based on the visual feature points including the position information and the first depth data and the inertial information;

[0090] Step S37: Based on the frame image of the object to be located, determine the frame image of the object to be located in the historical data that matches the frame image.

[0091] Step S38: Based on the comparison between the current frame image of the object to be located and the frame image of the object to be located in the historical data, determine the fourth pose transformation information of the object to be located.

[0092] Step S39: Based on the fourth pose transformation information, the second pose transformation information, and the third pose transformation information of the object to be located, determine the first pose information of the object to be located according to the factor graph.

[0093] The frame image obtained by the image acquisition device is not only input to the feature detection module to obtain the position information of the visual feature points, but also input to the visual loop detection module to determine whether there is a frame image in the historical data that matches the current frame image.

[0094] Specifically, if there is a frame image in the historical data that matches the current frame image, that is, determine the position reached by the object to be located in the current frame image, and determine whether there is a frame image in the historical data corresponding to this position. If it exists, it indicates that the object to be located has reached this position in the historical record, and the frame image in the historical data corresponding to this position is determined as the historical frame image.

[0095] At this time, determine the frame image of the object to be located in the historical data that matches the frame image, and determine the pose of the object to be located in the current frame image, and also, the pose of the object to be located in the determined historical frame image, and determine the fourth pose transformation information between the pose of the object to be located in the current frame image and the pose in the historical frame image. For example: the position coordinates of the object to be located in the historical frame image are (0, 0, 0), while the position coordinates in the current frame image are (1, 1, 1), then there is a change in pose between the current frame image and the historical frame image, and thus the fourth pose transformation information is determined.

[0096] The visual loop detection module can be implemented using a traditional bag-of-words (DBow) model or a visual CNN network. Taking the DBow for loop detection as an example, for each new frame image, detect the BRIEF descriptor and match the new frame image with the descriptors detected in the historical data, and send the timestamp of the closed-loop candidate frame detected by the DBow to the radar-side system for further verification.

[0097] After obtaining the fourth pose transformation information, input the fourth pose transformation information, the second pose transformation information, and the third pose transformation information into the factor graph optimization framework, and then a more accurate positioning result optimized by the factor graph can be output based on the inertial measurement frequency.

[0098] After processing the frame image, the VIO pose is obtained. After processing the radar data, the LIO pose is obtained. The pose change between two frames of inertial data obtained after IMU pre-integration of inertial data. These poses are not exactly the same, or do not exactly match the actual situation. Therefore, it is necessary to add the above data to the factor graph for optimization to obtain accurate pose information.

[0099] Among them, the factor graph is a common representation method in the SLAM system, and its essence is to solve the nonlinear least squares of the model. The factor graph is an undirected graph, which is divided into variable nodes (i.e., observations) representing optimization variables and factor nodes (the error between the constructed observation and prediction, i.e., constraints) representing factors. If you want to obtain a more accurate positioning structure, it is to require its maximum posterior probability. The maximum posterior probability is the multiplication of multiple factors. The optimization of the factor graph is to adjust the values of each variable to maximize the factor product.

[0100] In this embodiment, in the process of obtaining pose information using the factor graph, its constraints include not only the constraints of visual-inertial odometry and laser-inertial odometry, but also the constraints of loop closure, that is, the obtained fourth pose transformation information is also used as a reference value to determine the first pose information of the object to be located.

[0101] In addition, it can also include the constraints of IMU pre-integration, and the constraints of IMU pre-integration are only used in the process of determining the second pose transformation information.

[0102] In this solution, the visual-inertial navigation system performs the tracking of visual features for restoring depth detected by the radar. The visual odometry that optimizes the visual reprojection error and IMU measurement error provides an initial value for the scan-matching of the lidar, and adds the constraints to the factor graph. After removing the distortion of the point cloud using the IMU, the laser-inertial navigation system monitors the edge and plane features of the point cloud, that is, corner point and plane point features, and aligns them with the feature map saved in the sliding window. The constraints from visual-inertial odometry, laser-inertial odometry, IMU pre-integration, and loop closure are all input into the factor graph, and finally the bias information of the IMU is optimized.

[0103] Furthermore, the positioning method disclosed in this embodiment may further include:

[0104] Adjust the current offset information of the inertial measurement unit based on the pose information of the object to be located within a preset time period, and obtain the inertial information of the object to be located based on the real-time inertial measurement unit after adjusting the current offset information.

[0105] The offset information bias of the inertial measurement unit is an inherent property of the inertial measurement unit (IMU). It needs to be optimally estimated at any time based on the pose information to obtain the optimal current offset information that conforms to the current state. For example, if a pose information at a certain moment is obtained based on the inertial measurement unit and a pose information at that moment is obtained based on the radar point cloud information, and if the two pose information are different, since the accuracy of the radar point cloud information is higher, therefore, the final pose information is based on the pose information obtained from the radar point cloud information. Then, the offset information of the IMU needs to be adjusted to ensure that the inertial data obtained based on the inertial measurement unit (IMU) can better conform to the current pose state.

[0106] Among them, if it is determined that the number of visual feature points is less than the first preset value, and / or, it is determined that the current offset information is greater than the second preset value, the frame image of the object to be located is obtained again.

[0107] If the motion changes violently, the lighting changes, or the environment lacks texture, it will cause the image acquisition device to fail to recognize. This is because the number of feature points tracked in the above scenarios will decrease significantly, and fewer feature points will lead to optimization failure. When the image acquisition device fails, it will cause the offset information of the IMU to increase. Therefore, when the number of visual feature points is less than the first preset value, or, when the determined current offset information is greater than the second preset value, it is considered that the image acquisition device fails. At this time, it is necessary to re-obtain the frame image and re-identify the image acquisition device.

[0108] In addition, when the system is applied, first the image acquisition device is started, and then the radar is started. And since the frequency of obtaining the frame image by the image acquisition device is greater than the frequency of obtaining the radar point cloud information by the radar, therefore, the estimated value of the frame image is used as the initial value of the radar for frame matching.

[0109] The initial value plays an important role in continuous scan-matching, especially in the case of violent motion. The source of the initial value is different before and after the initialization of the image acquisition device.

[0110] Before the image acquisition device is initialized, if the object to be located is stationary at the initial position, assuming that the bias and noise of the IMU are both 0, the position and pose of two radar frame data are obtained by integrating the original IMU values and used as the initial values for scan - match; after the image acquisition device is initialized, the bias of the IMU, the pose and velocity of the object to be located in the factor graph are estimated, and then these data are transmitted to the processing unit for processing frame images in the processor to complete its initialization. After the radar is initialized, the initial values for scan - match can be obtained through two channels: the pre - integration of the IMU and the system for processing frame images. If the visual odometry can output the pose, the pose output by the visual odometry is used as the initial value; if the visual odometry cannot output the pose, the pre - integration of the IMU is used as the initial value.

[0111] The positioning method disclosed in this embodiment obtains the inertial information of the object to be located, determines the first pose transformation information between every two frames of inertial data based on the inertial information, obtains the frame image of the object to be located, determines the position information of the visual feature points based on the frame image, obtains the radar point cloud information of the object to be located, determines the first point cloud information after removing distortion and the corner and plane point feature information based on the radar point cloud information, and determines the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner and plane point feature information, and the first pose transformation information. This solution overcomes the situation where the pose features are missing due to occlusion in the frame image and the radar point cloud information by fusing the inertial information, frame image, and radar point cloud information of the object to be located, and avoids the situation of trajectory jumps caused by the lack of pose features.

[0112] This embodiment discloses a positioning system, and its structural schematic diagram is as Figure 4 shown, including:

[0113] The first acquisition unit 41, the second acquisition unit 42, the third acquisition unit 43 and the determination unit 44.

[0114] Among them, the first acquisition unit 41 is used to obtain the inertial information of the object to be located and determine the first pose transformation information between every two frames of inertial data based on the inertial information;

[0115] The second acquisition unit 42 is used to obtain the frame image of the object to be located and determine the position information of the visual feature points based on the frame image;

[0116] The third acquisition unit 43 is used to obtain the radar point cloud information of the object to be located and determine the first point cloud information after removing distortion and the corner and plane point feature information based on the radar point cloud information;

[0117] The determination unit 44 is configured to determine the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

[0118] Further, the determination unit is configured to:

[0119] Determine the first depth data of the visual feature points based on the first point cloud information; determine the second pose transformation information between two consecutive frames of radar data based on the corner point and plane point feature information and the first pose transformation information between every two frames of inertial data; determine the third pose transformation information between two consecutive frames of images based on the visual feature points including the position information and the first depth data and the inertial information; determine the first pose information of the object to be located based on the second pose transformation information and the third pose transformation information.

[0120] Further, the determination unit determines the first pose information of the object to be located based on the second pose transformation information and the third pose transformation information, including:

[0121] The determination unit determines the first pose information of the object to be located based on the weight of the second pose transformation information and the weight of the third pose transformation information.

[0122] Further, the determination unit determines the first pose information of the object to be located based on the second pose transformation information and the third pose transformation information, including:

[0123] Based on the frame image of the object to be located, determine the frame image of the object to be located in the historical data that matches the frame image; determine the fourth pose transformation information of the object to be located based on the comparison between the current frame image of the object to be located and the frame image of the object to be located in the historical data; determine the first pose information of the object to be located according to the factor graph based on the fourth pose transformation information, the second pose transformation information, and the third pose transformation information of the object to be located.

[0124] Further, the positioning system disclosed in this embodiment may further include: an adjustment unit,

[0125] configured to adjust the current offset information of the inertial measurement unit based on the pose information of the object to be located within a preset time period; obtain the inertial information of the object to be located based on the real-time inertial measurement unit after adjusting the current offset information.

[0126] Further, the determination unit determines the first depth data of the visual feature points based on the first point cloud information, including:

[0127] Determine the second depth data in the position information of the visual feature points based on the inertial information of the object to be located and the frame image of the object to be located; adjust the second depth data to the first depth data of the visual feature points based on the first point cloud information.

[0128] Further, the second acquisition unit is further configured to:

[0129] When it is determined that the number of visual feature points is less than a first preset value, and / or when it is determined that the current offset information is greater than a second preset value, re-acquire the frame image of the object to be located.

[0130] The positioning system disclosed in this embodiment is implemented based on the positioning method disclosed in the above embodiment, and details are not described herein again.

[0131] The positioning system disclosed in this embodiment acquires the inertial information of the object to be located, determines the first pose transformation information between every two frames of inertial data based on the inertial information, acquires the frame image of the object to be located, determines the position information of the visual feature points based on the frame image, acquires the radar point cloud information of the object to be located, determines the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information, and determines the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information. This solution overcomes the situation where the pose features are missing due to occlusion in the frame image and the radar point cloud information by fusing the inertial information, frame image, and radar point cloud information of the object to be located, and avoids the situation of trajectory jumps due to the lack of pose features.

[0132] This embodiment discloses an electronic device, and its structural schematic diagram is as Figure 5 shown, including:

[0133] An inertial measurement unit 51, an image acquisition device 52, a radar 53, and a processor 54.

[0134] Among them, the inertial measurement unit 51 is configured to acquire the inertial information of the object to be located;

[0135] The image acquisition device 52 is configured to acquire the frame image of the object to be located;

[0136] The radar 53 is configured to acquire the radar point cloud information of the object to be located;

[0137] The processor 54 is configured to determine the first pose transformation information between every two frames of inertial data based on the inertial information; determine the position information of the visual feature points based on the frame image; determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information; and determine the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

[0138] The electronic device disclosed in this embodiment is implemented based on the positioning method disclosed in the above embodiment, and details are not described herein again.

[0139] The electronic device disclosed in this embodiment obtains the inertial information of the object to be located, determines the first pose transformation information between every two frames of inertial data based on the inertial information, obtains the frame image of the object to be located, determines the position information of the visual feature points based on the frame image, obtains the radar point cloud information of the object to be located, determines the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information, and determines the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information. This solution overcomes the situation where the pose features are missing due to occlusion in the frame image and the radar point cloud information by fusing the inertial information, the frame image, and the radar point cloud information of the object to be located, and avoids the situation of trajectory jumps caused by the lack of pose features.

[0140] An embodiment of the present application further provides a readable storage medium, on which a computer program is stored. The computer program is loaded and executed by a processor to implement the steps of the above positioning method. The specific implementation process can refer to the description of the corresponding part of the above embodiment, and this embodiment will not be elaborated.

[0141] The present application also proposes a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the methods provided in various optional implementation manners of the above positioning method aspect or positioning system aspect. The specific implementation process can refer to the description of the corresponding embodiment above and will not be elaborated.

[0142] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0143] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0144] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination thereof. The software modules may be located in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium well known in the art.

[0145] The foregoing description of the disclosed embodiments enables those skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A positioning method, characterized in that, it includes: Obtain the inertial information of the object to be positioned, and determine the first pose transformation information between every two frames of inertial data based on the inertial information; Obtain the frame image of the object to be positioned, and determine the position information of the visual feature points based on the frame image; Obtain the radar point cloud information of the object to be positioned, and determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information; Determine the first pose information of the object to be positioned based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

2. The method according to claim 1, characterized in that, The determining the first pose information of the object to be positioned based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information includes: Determine the first depth data of the visual feature points based on the first point cloud information; Determine the second pose transformation information between two consecutive frames of radar data based on the corner point and plane point feature information and the first pose transformation information between every two frames of inertial data; Determine the third pose transformation information between two consecutive frames of images based on the visual feature points including the position information and the first depth data and the inertial information; Determine the first pose information of the object to be positioned based on the second pose transformation information and the third pose transformation information.

3. The method according to claim 2, characterized in that, The determining the first pose information of the object to be positioned based on the second pose transformation information and the third pose transformation information includes: Determine the first pose information of the object to be positioned based on the weight of the second pose transformation information and the weight of the third pose transformation information.

4. The method according to claim 2, characterized in that, The determining the first pose information of the object to be positioned based on the second pose transformation information and the third pose transformation information includes: Based on the frame image of the object to be positioned, determine the frame image of the object to be positioned in the historical data that matches the frame image; Determine the fourth pose transformation information of the object to be positioned based on the comparison between the current frame image of the object to be positioned and the frame image of the object to be positioned in the historical data; Determine the first pose information of the object to be positioned according to the factor graph based on the fourth pose transformation information, the second pose transformation information, and the third pose transformation information of the object to be positioned.

5. The method according to claim 1, characterized in that, it further includes: Adjust the current offset information of the inertial measurement unit based on the pose information of the object to be positioned within a preset time period; Obtain the inertial information of the object to be positioned based on the real-time inertial measurement unit after adjusting the current offset information.

6. The method according to claim 2, characterized in that, The determining the first depth data of the visual feature points based on the first point cloud information includes: Determine the second depth data in the position information of the visual feature points based on the inertial information of the object to be positioned and the frame image of the object to be positioned; Adjust the second depth data to the first depth data of the visual feature points based on the first point cloud information.

7. The method according to claim 1 or 5, wherein, further comprising: if it is determined that the number of the visual feature points is less than a first preset value, and / or it is determined that the current offset information is greater than a second preset value, re-obtain the frame image of the object to be located.

8. A positioning system, wherein, comprising: a first acquisition unit, configured to acquire the inertial information of the object to be located, and determine the first pose transformation information between every two frames of inertial data based on the inertial information; a second acquisition unit, configured to acquire the frame image of the object to be located, and determine the position information of the visual feature points based on the frame image; a third acquisition unit, configured to acquire the radar point cloud information of the object to be located, and determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information; a determination unit, configured to determine the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

9. An electronic device, wherein, comprising: an inertial measurement unit, configured to acquire the inertial information of the object to be located; an image acquisition device, configured to acquire the frame image of the object to be located; a radar, configured to acquire the radar point cloud information of the object to be located; a processor, configured to determine the first pose transformation information between every two frames of inertial data based on the inertial information; determine the position information of the visual feature points based on the frame image; determine the first point cloud information after removing distortion and the corner point and plane point feature information based on the radar point cloud information; determine the first pose information of the object to be located based on the position information of the visual feature points, the first point cloud information, the corner point and plane point feature information, and the first pose transformation information.

10. A readable storage medium for storing at least a set of instruction sets; the instruction sets are used to be called and at least execute the positioning method as described in any one of the above.

Images