Pose estimation method and device

Abstract

The invention provides a pose estimation method and device, and the method comprises the steps: obtaining a first aerial view and a second aerial view which are two adjacent frames of images, inputting the first aerial view and the second aerial view into a deep learning model, obtaining first feature information corresponding to the first aerial view, and obtaining a second feature information corresponding to the second aerial view; determining N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information from a plurality of preset feature categories based on a preset feature information classification rule according to the first feature information corresponding to the first aerial view and the second feature information corresponding to the second aerial view, N and M being integers greater than or equal to 0, and based on the N first feature categories and the M second feature categories, estimating a relative pose transformation result between the first aerial view and the second aerial view.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and in particular, relates to a posture estimation method and device thereof. Background Art

[0002] Autonomous parking is an important part of existing autonomous driving, and the reliability of pose estimation results is crucial for the real-time and accuracy requirements of automatic parking technology. Current pose estimation methods generally use the geometric or optical feature points between two adjacent frames as registration input to estimate the relative pose transformation between the two frames. However, geometric or optical feature points have the disadvantage of scene robustness, which may cause large registration errors in complex scenes, resulting in poor reliability of pose estimation results. Summary of the invention

[0003] The embodiments of the present application provide a posture estimation method and a device thereof to improve the reliability of posture estimation results.

[0004] In a first aspect, an embodiment of the present application provides a posture estimation method, the method comprising:

[0005] Acquire a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images,

[0006] The first bird's-eye view image and the second bird's-eye view image are input into the deep learning model to obtain first feature information corresponding to the first bird's-eye view image and second feature information corresponding to the second bird's-eye view image.

[0007] Based on a preset feature information classification rule, N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information are determined from a plurality of preset feature categories, where N and M are integers greater than or equal to 0.

[0008] Based on the N first feature categories and the M second feature categories, a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image is estimated.

[0009] In a second aspect, an embodiment of the present application provides a posture estimation device, the device comprising:

[0010] An acquisition module is used to acquire a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images.

[0011] An input module is used to input the first bird's-eye view and the second bird's-eye view into the deep learning model to obtain first feature information corresponding to the first bird's-eye view and second feature information corresponding to the second bird's-eye view,

[0012] a determination module, configured to determine, based on a preset feature information classification rule, N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information from a plurality of preset feature categories, where N and M are integers greater than or equal to 0,

[0013] The estimation module is used to estimate the relative posture transformation result between the first bird's-eye view image and the second bird's-eye view image based on the N first feature categories and the M second feature categories.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising:

[0015] A processor and a memory storing programs or instructions;

[0016] The above method is implemented when the processor executes the program or instruction.

[0017] In a fourth aspect, an embodiment of the present application provides a machine-readable storage medium, on which a program or instruction is stored, and the program or instruction implements the above method when executed by a processor.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the above method.

[0019] The pose estimation method and device of the embodiment of the present application can obtain a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images, input the first bird's-eye view and the second bird's-eye view into a deep learning model, obtain first feature information corresponding to the first bird's-eye view, and second feature information corresponding to the second bird's-eye view, and based on a preset feature information classification rule, determine N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information from multiple preset feature categories, where N and M are integers greater than or equal to 0, and estimate the relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the N first feature categories and the M second feature categories.

[0020] In this way, after classification based on the feature information in two adjacent frames of images, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be estimated according to the feature categories in the two frames of images. By utilizing the point cloud feature information, the shortcomings of scene robustness can be overcome and the registration accuracy can be guaranteed, thereby improving the reliability of the pose estimation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 It is a flowchart of a posture estimation method provided by an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a specific scenario embodiment of the posture estimation method provided in an embodiment of the present application;

[0024] Figure 3 is a structural schematic diagram of a posture estimation device provided by another embodiment of the present application;

[0025] Figure 4 It is a structural diagram of an electronic device provided by yet another embodiment of the present application. DETAILED DESCRIPTION

[0026] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0028] In order to solve the problems in the prior art, the present invention provides a method and device for posture estimation. The posture estimation method provided by the present invention is first introduced below.

[0029] Figure 1FIG. 1 is a flow chart of a method for estimating a posture provided by an embodiment of the present application. Figure 1 As shown, the pose estimation method may include the following steps:

[0030] Step 101: Acquire a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images.

[0031] In step 101, the first bird's-eye view and the second bird's-eye view may be obtained based on an image acquisition device installed on a vehicle. The first bird's-eye view and the second bird's-eye view may be two adjacent frames of images. For example, the first bird's-eye view t can be the image at the current moment, the second bird's-eye view f t-1 It can be the image of the previous moment.

[0032] Step 102: input the first bird's-eye view and the second bird's-eye view into a deep learning model to obtain first feature information corresponding to the first bird's-eye view and second feature information corresponding to the second bird's-eye view.

[0033] In step 102, the first bird's-eye view and the second bird's-eye view may be input into a deep learning model. The deep learning model may be trained based on multiple images and feature information annotated in each image. The deep learning model may be used to identify feature information in the image.

[0034] Based on this, the first bird's-eye view and the second bird's-eye view are input into a deep learning model, and the deep learning model can output first feature information corresponding to the first bird's-eye view and second feature information corresponding to the second bird's-eye view.

[0035] It can be understood that both the first characteristic information and the second characteristic information can include one or more of the pre-set different characteristic information. For example, the characteristic information may include background, lane lines, grid lines, arrows, sidewalks, manhole covers, speed bumps, parking lines, wheel chocks, curbs, etc.

[0036] In other words, the first bird's-eye view and the second bird's-eye view may contain the above-mentioned background, lane lines, grid lines, arrows, sidewalks, manhole covers, speed bumps, parking lines, wheel chocks, curbs and other objects. Based on the objects contained in the first bird's-eye view and the second bird's-eye view, the deep learning model can identify the corresponding feature information.

[0037] Step 103: Based on a preset feature information classification rule, determine N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information from a plurality of preset feature categories, where N and M are integers greater than or equal to 0.

[0038] In step 103, the multiple preset feature categories may refer to feature categories pre-set for the above different feature information, for example, the multiple preset feature categories may include background features, driving features, obstacle features, etc., and the multiple preset feature categories may also include surface features, line features and other features. The specific classification can be set according to actual needs and is not specifically limited here.

[0039] The preset feature information classification rules may refer to the feature categories corresponding to each feature information. For example, the background may be classified as background features, lane lines, grid lines, arrows, and parking lines may be classified as driving features, and sidewalks, manhole covers, speed bumps, and curbs may be classified as obstacle features. For another example, lane lines and parking lines may be classified as line features, grid lines, arrows, manhole covers, speed bumps, wheel chocks, and curbs may be classified as surface features, and sidewalks and background may not be included in the classification criteria. Specific feature information classification rules may be set according to actual needs and are not specifically limited here.

[0040] Based on the preset feature information classification rules, N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information can be determined from multiple preset feature categories. For example, if the first feature information includes lane lines, arrows, and background, it can be considered that the N first feature categories include line features and surface features, and if the second feature information includes lane lines and background, it can be considered that the M second feature categories only include line features.

[0041] Step 104 : estimating a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image based on the N first feature categories and the M second feature categories.

[0042] In step 104, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be estimated based on the N first feature categories and the M second feature categories. For example, if the first feature category and the second feature category both include surface features, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be estimated by aligning the point cloud of the surface features. If the first feature category and the second feature category both include line features, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be estimated by aligning the point cloud of the line features. If the first feature category and the second feature category do not include line features and surface features, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be estimated by aligning geometric or optical feature points using existing pose estimation methods.

[0043] The pose estimation method of the embodiment of the present application can obtain a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images, and the first bird's-eye view and the second bird's-eye view are input into a deep learning model to obtain first feature information corresponding to the first bird's-eye view and second feature information corresponding to the second bird's-eye view. Based on a preset feature information classification rule, N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information are determined from multiple preset feature categories, where N and M are integers greater than or equal to 0. Based on the N first feature categories and the M second feature categories, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view is estimated.

[0044] In this way, after classification based on the feature information in two adjacent frames of images, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be estimated according to the feature categories in the two frames of images. By utilizing the point cloud feature information, the shortcomings of scene robustness can be overcome and the registration accuracy can be guaranteed, thereby improving the reliability of the pose estimation results.

[0045] In some embodiments, the first feature information and / or the second feature information includes at least one of a lane line, a parking space line, a grid line, an arrow, a manhole cover, a speed bump, a wheel chock, a curb, a sidewalk, and a background.

[0046] Among them, the preset feature information classification rules are:

[0047] The preset feature category corresponding to lane lines and parking space lines is line features.

[0048] The preset feature category corresponding to grid lines, arrows, manhole covers, speed bumps, wheel chocks, and curbs is surface features.

[0049] In this embodiment, the first feature information and / or the second feature information may include at least one of lane lines, parking space lines, grid lines, arrows, manhole covers, speed bumps, wheel chocks, curbs, sidewalks, and backgrounds. Among them, based on the feature information output by the deep learning model and manual experience, the above feature information can be classified into two categories: line features and surface features. The preset feature categories corresponding to lane lines and parking space lines can be set to line features, that is: M line ={lane line, parking space line}.

[0050] The preset feature categories corresponding to grid lines, arrows, manhole covers, speed bumps, wheel blocks, and curbs can be set as surface features, that is: M plane ={grid lines, arrows, manhole covers, speed bumps, wheel chocks, curbs}.

[0051] The sidewalk and the background are not included in the classification criteria, that is, the sidewalk and the background are neither line features nor surface features.

[0052] In this way, the first feature information and the second feature information can be classified based on the above-mentioned feature information classification rules, so that the relative posture transformation result between the first bird's-eye view and the second bird's-eye view can be calculated based on the point clouds of each feature category in the first bird's-eye view and the second bird's-eye view.

[0053] In some embodiments, the above step 104 may include the following steps:

[0054] The point cloud corresponding to each first feature category in the N first feature categories is stored as a first point cloud frame,

[0055] The point cloud corresponding to each second feature category in the M second feature categories is stored as a second point cloud frame,

[0056] When the same feature category exists in the N first feature categories and the M second feature categories, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view is estimated based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame.

[0057] In this embodiment, after the first feature information and the second feature information in the first bird's-eye view and the second bird's-eye view are respectively divided into N first feature categories and M second feature categories according to the preset feature information classification rule, the final classification results can be stored in point cloud frame format, that is, the point cloud corresponding to each first feature category in the N first feature categories can be stored as a first point cloud frame, and the point cloud corresponding to each second feature category in the M second feature categories can be stored as a second point cloud frame.

[0058] After obtaining the first point cloud frame and the second point cloud frame of the classification result, different feature registration methods can be used for the point clouds corresponding to different feature categories in the first point cloud frame and the second point cloud frame to estimate the pose estimation results corresponding to the feature categories. For example, it can be determined whether there are the same feature categories in the N first feature categories and the M second feature categories. If there are the same feature categories, the pose estimation results corresponding to the feature categories can be estimated based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame using the registration method corresponding to the same feature category.

[0059] For example, if the same feature category is surface feature, the point-to-point pose estimation algorithm can be used to calculate the surface feature pose estimation result; if the same feature category is line feature, the point-to-line pose estimation algorithm can be used to calculate the line feature pose estimation result, so that the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be determined based on the surface feature pose estimation result and / or the line feature pose estimation result.

[0060] In this way, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view can be estimated based on the point clouds corresponding to the same feature categories in the first point cloud frame and the second point cloud frame, thereby improving the accuracy of point cloud registration and thus improving the reliability of the pose estimation result.

[0061] In some embodiments, when the same feature category is a surface feature, estimating the relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame may include the following steps:

[0062] According to the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame, the surface feature pose estimation result is obtained.

[0063] The face feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view and the second bird's-eye view.

[0064] In an embodiment of the present application, if the same feature category is surface features, in other words, both the first bird's-eye view and the second bird's-eye view include feature information corresponding to the surface features, and at least one of the first bird's-eye view and the second bird's-eye view lacks feature information corresponding to the line features, then the surface feature pose estimation result can be obtained based on the point cloud corresponding to the surface features in the first point cloud frame and the point cloud corresponding to the surface features in the second point cloud frame using a registration method corresponding to the surface features (such as a point-to-point pose estimation algorithm).

[0065] For example, only the point clouds corresponding to the surface features in the first point cloud frame and the second point cloud frame can be used as input, and the surface feature pose estimation can be calculated through a point-to-point pose estimation algorithm (Point-to-Point Matcher). The surface feature pose estimation result can be determined as the relative pose transformation result between the first bird's-eye view and the second bird's-eye view and output.

[0066] In some embodiments, when the same feature category is a line feature, estimating the relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame may include the following steps:

[0067] According to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame, the line feature pose estimation result is obtained.

[0068] The line feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image.

[0069] In an embodiment of the present application, if the same feature category is line features, in other words, both the first bird's-eye view and the second bird's-eye view include feature information corresponding to the line features, and at least one of the first bird's-eye view and the second bird's-eye view lacks feature information corresponding to the surface features, then the line feature pose estimation result can be obtained based on the point cloud corresponding to the line features in the first point cloud frame and the point cloud corresponding to the line features in the second point cloud frame by adopting a registration method corresponding to the line features (such as a point-to-line pose estimation algorithm).

[0070] For example, only the point clouds corresponding to the line features in the first point cloud frame and the second point cloud frame can be used as input, and the line feature pose estimation can be calculated by a point-to-line pose estimation algorithm (Point-to-Line Matcher). The line feature pose estimation result can be determined as the relative pose transformation result between the first bird's-eye view and the second bird's-eye view and output.

[0071] In some embodiments, when the same feature category is a line feature and a surface feature, estimating the relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame may include the following steps:

[0072] According to the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame, the surface feature pose estimation result is obtained.

[0073] The surface feature pose estimation result is used as the initial value for line feature point cloud registration.

[0074] Based on the initial value of line feature point cloud registration, the line feature pose estimation result is obtained according to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame.

[0075] The line feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image.

[0076] In the embodiment of the present application, if the same feature category is line feature and surface feature, in other words, the first bird's-eye view and the second bird's-eye view both include feature information corresponding to the line feature and feature information corresponding to the surface feature. At this time, the surface feature pose estimation result can be obtained by using the registration method corresponding to the surface feature (such as a point-to-point pose estimation algorithm) based on the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame.

[0077] For example, the point clouds corresponding to the surface features in the first point cloud frame and the second point cloud frame can be used as input to a point-to-point pose estimation algorithm (Point-to-Point Matcher), and the time-synchronized inertial navigation data is used as the initial alignment value to calculate the surface feature pose estimate.

[0078] Then, the surface feature pose estimation can be used as the initial value of the line feature point cloud registration. On this basis, according to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame, the line feature pose estimation result is obtained by using the line feature corresponding registration method (such as the point-to-line pose estimation algorithm). For example, the point cloud corresponding to the line feature in the first point cloud frame and the second point cloud frame can be used as input, and the line feature pose estimation can be calculated by the point-to-line pose estimation algorithm (Point-to-LineMatcher), and the line feature pose estimation result can be determined as the relative pose transformation result between the first bird's-eye view and the second bird's-eye view for output.

[0079] In some embodiments, obtaining a surface feature pose estimation result according to a point cloud corresponding to the surface feature in the first point cloud frame and a point cloud corresponding to the surface feature in the second point cloud frame may include the following steps:

[0080] The point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame are used as input values ​​of the point-to-point pose estimation algorithm to obtain the surface feature pose estimation result. The optimization equation of the cost function of the point-to-point pose estimation algorithm is:

[0081]

[0082] Among them, e is the cost function, p i,src is any point in the first point cloud frame, p j,tar is the distance p in the second point cloud frame i,src The nearest point, is the value of the face feature pose estimation result, and T is the numerical solution of the inter-frame pose transformation of the face feature.

[0083] In this embodiment, since the feature information corresponding to the surface feature generally has no obvious and reliable azimuth information, the surface feature pose estimation result can use the traditional integrated point-to-point pose estimation algorithm (Point-to-PointMatcher). For any point p in the first point cloud frame, i,src , which uses the nearest neighbor search to convert the distance p in the second point cloud frame i,src The nearest point is set to p i,tar . Its optimized cost function can be written as:

[0084] e=||p j,tar -Tp i,src ||

[0085] Then the optimization equation can be written as:

[0086]

[0087] Among them, e is the cost function, p i,src is any point in the first point cloud frame, p j,tar is the distance p in the second point cloud frame i,src The nearest point, is the value of the face feature pose estimation result, and T is the numerical solution of the inter-frame pose transformation of the face feature.

[0088] The above optimization equations can be solved by using Newton iteration method or Levenberg-Marquardt numerical optimization method to obtain the numerical solution of the surface feature pose estimation result.

[0089] In this way, the corresponding point-to-point pose estimation algorithm is used for face features to perform pose estimation, so as to avoid mismatching between different feature information, improve the accuracy of pose estimation, and reduce the time used for estimation accordingly, thereby improving the efficiency of pose estimation.

[0090] In some embodiments, obtaining a line feature pose estimation result according to a point cloud corresponding to the line feature in the first point cloud frame and a point cloud corresponding to the line feature in the second point cloud frame may include the following steps:

[0091] The point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame are used as input values ​​of the point-to-line pose estimation algorithm to obtain the line feature pose estimation result. The optimization equation of the cost function of the point-to-line pose estimation algorithm is:

[0092]

[0093] Among them, e is the cost function, p i,src is any point in the first point cloud frame, To project the point cloud in the first point cloud frame to the isometric transformation in the point cloud coordinate system corresponding to the second point cloud frame, p j,tar and p k,tar is the distance p in the second point cloud frame i,src The two endpoints on the nearest line segment, is the value of the line feature pose estimation result, and T is the numerical solution of the inter-frame pose transformation of the line feature.

[0094] In this embodiment, due to its aspect ratio, the line feature can be included in the cost function of its corresponding point-to-line pose estimation algorithm (Point-to-Line Matcher) to take into account the overall azimuth constraint of the line feature. i,src , the distance p in the second point cloud frame can be obtained by nearest neighbor search i,src The two endpoints p on the nearest line segment j,tar and pk,tar . Its optimized cost function can be written as:

[0095]

[0096] Then the optimization equation can be written as:

[0097]

[0098] Among them, e is the cost function, p i,src is any point in the first point cloud frame, To project the point cloud in the first point cloud frame to the isometric transformation in the point cloud coordinate system corresponding to the second point cloud frame, p j,tar and p k,tar is the distance p in the second point cloud frame i,src The two endpoints on the nearest line segment, is the value of the line feature pose estimation result, and T is the numerical solution of the inter-frame pose transformation of the line feature.

[0099] The above optimization equation can be solved by using the Newton iteration method or the Levenberg-Marquardt numerical optimization method to obtain the numerical solution T of the line feature pose estimation result. * line .

[0100] In this way, the corresponding point-to-line pose estimation algorithm is used for line features to perform pose estimation, so as to avoid mismatching between different feature information, improve the accuracy of pose estimation, and reduce the time used for estimation accordingly, thereby improving the efficiency of pose estimation.

[0101] In some embodiments, the pose estimation method may further include the following steps:

[0102] In the case that the same feature category does not exist in the N first feature categories and the M second feature categories, a unit matrix is ​​determined based on the first point cloud frame and the second point cloud frame.

[0103] The identity matrix is ​​determined as a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image.

[0104] In this embodiment, if the same feature category does not exist in the N first feature categories and the M second feature categories, in actual working conditions, there is usually no feature information corresponding to the line feature in the first bird's-eye view and the second bird's-eye view, nor is there feature information corresponding to the surface feature. In other words, there is no valid line feature point cloud in the first point cloud frame and no valid surface feature point cloud in the second point cloud frame. At this time, it can be said that there is not enough point cloud information in the first point cloud frame and the second point cloud frame.

[0105] Based on this, a unit matrix can be determined based on the first point cloud frame and the second point cloud frame, and the unit matrix is ​​determined as a relative posture transformation result between the first bird's-eye view and the second bird's-eye view for output.

[0106] Based on the posture estimation method provided in the above embodiment, the present application also provides a specific scenario embodiment of the posture estimation method. Figure 2 As shown, the scenario embodiment includes the following steps:

[0107] Step 201, start.

[0108] Step 202: input two adjacent frames of images (the first bird's-eye view and the second bird's-eye view) into a deep learning model to extract feature information.

[0109] Step 203: classify the above feature information.

[0110] Step 204 , determining whether there are enough surface features in two adjacent image frames, if yes, executing step 205 , if not, executing step 210 .

[0111] Step 205: Obtain the point cloud corresponding to the surface feature.

[0112] Step 206, using the point cloud corresponding to the surface feature as the input of a point-to-point pose estimation algorithm (Point-to-PointMatcher) to calculate the surface feature pose estimation result.

[0113] Step 207 , determining whether there are enough line features in two adjacent frames of images, if yes, executing step 208 , if not, executing step 213 .

[0114] Step 208: Obtain the point cloud corresponding to the line feature.

[0115] Step 209, using the point cloud corresponding to the line feature as the input of the point-to-point pose estimation algorithm (Point-to-LineMatcher) to calculate the line feature pose estimation result.

[0116] Step 210 , determining whether there are enough line features in two adjacent frames of images, if yes, executing step 208 , if not, executing step 211 .

[0117] Step 211, obtaining a downsampled point cloud.

[0118] Step 212: Use the downsampled point cloud as input to a point-to-point pose estimation algorithm (Point-to-Point Matcher) to obtain a unit matrix.

[0119] Step 213: output the final pose estimation result, that is, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view.

[0120] Step 214, end.

[0121] In this way, line features and surface features are classified and constructed according to different feature information, and corresponding pose estimation methods are used for these two types of features to avoid mismatching between different feature information, improve the accuracy of pose estimation, and reduce the time used for estimation accordingly, thereby improving the efficiency of pose estimation.

[0122] Based on the posture estimation method provided in the above embodiment, the present application also provides an embodiment of a posture estimation device.

[0123] Figure 3 A structural schematic diagram of a posture estimation device provided in another embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0124] Reference Figure 3 , the posture estimation device 300 includes:

[0125] The acquisition module 301 is used to acquire a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images.

[0126] An input module 302 is used to input the first bird's-eye view and the second bird's-eye view into a deep learning model to obtain first feature information corresponding to the first bird's-eye view and second feature information corresponding to the second bird's-eye view.

[0127] The determination module 303 is used to determine, based on a preset feature information classification rule, N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information from a plurality of preset feature categories, where N and M are integers greater than or equal to 0.

[0128] The estimation module 304 is used to estimate the relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image based on the N first feature categories and the M second feature categories.

[0129] In some embodiments, the first feature information and / or the second feature information includes at least one of a lane line, a parking space line, a grid line, an arrow, a manhole cover, a speed bump, a wheel chock, a curb, a sidewalk, and a background.

[0130] Among them, the preset feature information classification rules are:

[0131] The preset feature category corresponding to lane lines and parking space lines is line features.

[0132] The preset feature category corresponding to grid lines, arrows, manhole covers, speed bumps, wheel chocks, and curbs is surface features.

[0133] In some embodiments, the estimation module 304 may also be used to:

[0134] The point cloud corresponding to each first feature category in the N first feature categories is stored as a first point cloud frame,

[0135] The point cloud corresponding to each second feature category in the M second feature categories is stored as a second point cloud frame,

[0136] When the same feature category exists in the N first feature categories and the M second feature categories, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view is estimated based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame.

[0137] In some embodiments, when the same feature category is a face feature, the estimation module 304 may also be used to:

[0138] According to the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame, the surface feature pose estimation result is obtained.

[0139] The face feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view and the second bird's-eye view.

[0140] In some embodiments, when the same feature category is a line feature, the estimation module 304 may also be used to:

[0141] According to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame, the line feature pose estimation result is obtained.

[0142] The line feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image.

[0143] In some embodiments, when the same feature category is a line feature and a surface feature, the estimation module 304 may also be used to:

[0144] According to the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame, the surface feature pose estimation result is obtained.

[0145] The surface feature pose estimation result is used as the initial value for line feature point cloud registration.

[0146] Based on the initial value of line feature point cloud registration, the line feature pose estimation result is obtained according to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame.

[0147] The line feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image.

[0148] In some embodiments, the estimation module 304 may also be used to:

[0149] The point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame are used as input values ​​of the point-to-point pose estimation algorithm to obtain the surface feature pose estimation result. The optimization equation of the cost function of the point-to-point pose estimation algorithm is:

[0150]

[0151] Among them, e is the cost function, p i,src is any point in the first point cloud frame, p j,tar is the distance p in the second point cloud frame i,src The nearest point, It is the value of the face feature pose estimation result.

[0152] In some embodiments, the estimation module 304 may also be used to:

[0153] The point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame are used as input values ​​of the point-to-line pose estimation algorithm to obtain the line feature pose estimation result. The optimization equation of the cost function of the point-to-line pose estimation algorithm is:

[0154]

[0155] Among them, e is the cost function, p i,src is any point in the first point cloud frame, To project the point cloud in the first point cloud frame to the isometric transformation in the point cloud coordinate system corresponding to the second point cloud frame, p j,tar and p k,tar is the distance p in the second point cloud frame i,src The two endpoints on the nearest line segment, is the value of the line feature pose estimation result.

[0156] In some embodiments, the estimation module 304 may also be used to:

[0157] In the case that the same feature category does not exist in the N first feature categories and the M second feature categories, a unit matrix is ​​determined based on the first point cloud frame and the second point cloud frame.

[0158] The identity matrix is ​​determined as a relative pose transformation result between the first bird's-eye view image and the second bird's-eye view image.

[0159] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application, and are devices corresponding to the above-mentioned posture estimation method. All implementation methods in the above-mentioned method embodiment are applicable to the embodiments of the device. Its specific functions and technical effects can be found in the method embodiment part, which will not be repeated here.

[0160] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0161] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in yet another embodiment of the present application is shown.

[0162] The device may include a processor 401 and a memory 402 storing programs or instructions.

[0163] When the processor 401 executes the program, the steps in any of the above method embodiments are implemented.

[0164] Exemplarily, the program may be divided into one or more modules / units, one or more modules / units are stored in the memory 402, and executed by the processor 401 to complete the present application. One or more modules / units may be a series of program instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the program in the device.

[0165] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0166] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0167] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) machine-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0168] The processor 401 implements any one of the methods in the above embodiments by reading and executing the program or instruction stored in the memory 402 .

[0169] In one example, the electronic device may further include a communication interface 403 and a bus 404. The processor 401, the memory 402, and the communication interface 403 are connected via the bus 404 and communicate with each other.

[0170] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0171] Bus 404 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 404 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0172] In addition, in combination with the method in the above embodiment, the embodiment of the present application may provide a machine-readable storage medium for implementation. The machine-readable storage medium stores a program or instruction; when the program or instruction is executed by a processor, any one of the methods in the above embodiment is implemented. The machine-readable storage medium can be read by a machine such as a computer.

[0173] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0174] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0175] An embodiment of the present application provides a computer program product, which is stored in a machine-readable storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0176] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0177] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer grids such as the Internet, intranets, etc.

[0178] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0179] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by a computer program or instruction. These programs or instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0180] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A pose estimation method, characterized in that: include: Acquire a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images, Inputting the first bird's-eye view and the second bird's-eye view into a deep learning model to obtain first feature information corresponding to the first bird's-eye view and second feature information corresponding to the second bird's-eye view, Based on a preset feature information classification rule, N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information are determined from a plurality of preset feature categories, where N and M are integers greater than or equal to 0. Based on the N first feature categories and the M second feature categories, a relative pose transformation result between the first bird's-eye view and the second bird's-eye view is estimated.

2. The method according to claim 1, characterized in that The estimating a relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the N first feature categories and the M second feature categories includes: The point cloud corresponding to each first feature category in the N first feature categories is stored as a first point cloud frame, The point cloud corresponding to each second feature category in the M second feature categories is stored as a second point cloud frame, In the case where the same feature categories exist in the N first feature categories and the M second feature categories, the relative pose transformation result between the first bird's-eye view and the second bird's-eye view is estimated based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame.

3. The method according to claim 2, characterized in that In the case where the same feature category is a surface feature, estimating a relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame includes: According to the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame, a surface feature pose estimation result is obtained, The face feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view and the second bird's-eye view.

4. The method according to claim 2, characterized in that: In the case where the same feature category is a line feature, estimating a relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame includes: Obtaining a line feature pose estimation result according to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame, The line feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view and the second bird's-eye view.

5. The method according to claim 2, characterized in that: In a case where the same feature category is a line feature and a surface feature, estimating a relative pose transformation result between the first bird's-eye view and the second bird's-eye view based on the point cloud corresponding to the same feature category in the first point cloud frame and the point cloud corresponding to the same feature category in the second point cloud frame includes: According to the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame, a surface feature pose estimation result is obtained, The surface feature pose estimation result is used as the initial value for line feature point cloud registration. Based on the line feature point cloud registration initial value, a line feature pose estimation result is obtained according to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame. The line feature pose estimation result is determined as a relative pose transformation result between the first bird's-eye view and the second bird's-eye view.

6. The method according to claim 3 or 5, characterized in that: The step of obtaining a surface feature pose estimation result according to the point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame includes: The point cloud corresponding to the surface feature in the first point cloud frame and the point cloud corresponding to the surface feature in the second point cloud frame are used as input values ​​of the point-to-point pose estimation algorithm to obtain the surface feature pose estimation result. The formula of the point-to-point pose estimation algorithm is: Among them, e is the cost function, p i,src is any point in the first point cloud frame, p j,tar is the distance p in the second point cloud frame i,src The nearest point, It is the value of the face feature pose estimation result.

7. The method according to claim 4 or 5, characterized in that: The obtaining a line feature pose estimation result according to the point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame includes: The point cloud corresponding to the line feature in the first point cloud frame and the point cloud corresponding to the line feature in the second point cloud frame are used as input values ​​of the point-to-line pose estimation algorithm to obtain a line feature pose estimation result. The formula of the point-to-line pose estimation algorithm is: Among them, e is the cost function, p i,src is any point in the first point cloud frame, To project the point cloud in the first point cloud frame to the isometric transformation in the point cloud coordinate system corresponding to the second point cloud frame, p j,tar and p k,tar is the distance p in the second point cloud frame i,src The two endpoints on the nearest line segment, is the value of the line feature pose estimation result.

8. The method according to claim 2, characterized in that: The method further comprises: In the case that the N first feature categories and the M second feature categories do not have the same feature category, a unit matrix is ​​determined based on the first point cloud frame and the second point cloud frame, The unit matrix is ​​determined as a relative pose transformation result between the first bird's-eye view and the second bird's-eye view.

9. The method according to claim 1, characterized in that: The first feature information and / or the second feature information includes at least one of a lane line, a parking space line, a grid line, an arrow, a manhole cover, a speed bump, a wheel chock, a curb, a sidewalk, and a background, The preset feature information classification rule is: The preset feature categories corresponding to the lane line and the parking space line are line features. The preset feature categories corresponding to the grid lines, the arrows, the manhole covers, the speed bumps, the wheel chocks and the curbs are surface features.

10. A posture estimation device, characterized in that: include: An acquisition module is used to acquire a first bird's-eye view and a second bird's-eye view, wherein the first bird's-eye view and the second bird's-eye view are two adjacent frames of images. an input module, configured to input the first bird's-eye view and the second bird's-eye view into a deep learning model to obtain first feature information corresponding to the first bird's-eye view and second feature information corresponding to the second bird's-eye view, a determination module, configured to determine, based on a preset feature information classification rule, N first feature categories corresponding to the first feature information and M second feature categories corresponding to the second feature information from a plurality of preset feature categories, where N and M are integers greater than or equal to 0, An estimation module is used to estimate a relative posture transformation result between the first bird's-eye view and the second bird's-eye view based on the N first feature categories and the M second feature categories.

Images