A method, apparatus, medium, and electronic device for converting a planar image

Abstract

The application relates to the technical field of intelligent transportation, in particular to a planar image conversion method and device, a medium and an electronic device. The method comprises the following steps: acquiring planar image data and corresponding point cloud image data; determining a fitting curved surface model based on the ground form of a current scene, the planar image data and the point cloud image data, wherein the fitting curved surface model comprises at least two sub-curved surfaces with different curvatures; determining a first feature point set from the planar image data; determining a second feature point set based on the point cloud image data and the fitting curved surface model; determining a coordinate mapping relationship based on the feature point coordinate data in the first feature point set and the second feature point set; and determining the three-dimensional coordinates of each pixel point of the planar image data according to the coordinate mapping relationship, so as to complete the conversion of the planar image data. The technical scheme provided by the application can convert the pixel point coordinate values in the planar image into the three-dimensional coordinate values in the three-dimensional point cloud image when the internal and external parameter data cannot be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a planar image conversion method and device, a medium and an electronic device. BACKGROUND

[0002] The conversion relationship between the image captured by the camera and the point cloud image measured by the laser radar is generally obtained by using a calibration board and other devices to obtain the internal and external parameters of the device through nine-point calibration and other methods, and then determining the mutual relationship between the three-dimensional geometric position of a point on the surface of a space object and the corresponding point in the image through the internal and external parameters. However, in some special cases, internal and external parameter data cannot be obtained, so the corresponding relationship of the coordinates cannot be found, and ultimately the captured planar image cannot be converted into a three-dimensional image.

[0003] Therefore, the technical personnel in the field urgently need a planar image conversion method to convert the pixel point coordinate value in the planar image into a three-dimensional coordinate value in the three-dimensional point cloud image when internal and external parameter data cannot be obtained. SUMMARY

[0004] Embodiments of the present application provide a planar image conversion method, device, medium and electronic device, which can at least partially convert the pixel point coordinate value in the planar image into a three-dimensional coordinate value in the three-dimensional point cloud image when internal and external parameter data cannot be obtained.

[0005] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0006] According to an aspect of an embodiment of the present application, a planar image conversion method is provided, the method comprising: obtaining planar image data and corresponding point cloud image data, determining a fitted curved surface model based on the ground form of a current scene, the planar image data and the point cloud image data, the fitted curved surface model comprising at least two sub-curved surfaces with different curvatures; determining a first feature point set from the planar image data, determining a second feature point set based on the point cloud image data and the fitted curved surface model; determining a coordinate mapping relationship based on the feature point coordinate data in the first feature point set and the second feature point set, the coordinate mapping relationship being used to represent the corresponding relationship between the two-dimensional coordinates of the feature points in the planar image data and the three-dimensional coordinates in the world coordinate system; and determining the three-dimensional coordinates of each pixel point of the planar image data according to the coordinate mapping relationship, to complete the conversion of the planar image data.

[0007] In some embodiments of the present application, the determining the first set of feature points from the planar image data, determining the second set of feature points based on the point cloud image data and the fitted surface model, comprises: selecting feature points from the planar image data, determining two-dimensional coordinate data of each feature point, and constructing the first set of feature points; based on the point cloud image data, reading three-dimensional coordinate data of at least one feature point; based on the two-dimensional coordinate data of the target feature point and the fitted surface model, calculating the three-dimensional coordinate data of the target feature point, the target feature point being a feature point that cannot read the three-dimensional coordinate data from the point cloud image data; and constructing the second set of feature points according to the three-dimensional coordinate data of each feature point.

[0008] In some embodiments of the present application, based on the foregoing scheme, the calculating the three-dimensional coordinate data of the target feature point based on the two-dimensional coordinate data of the target feature point and the fitted surface model comprises: determining remote sensing coordinate data matched by the target feature point in the remote sensing satellite map according to the two-dimensional coordinate data of the target feature point; and calculating the three-dimensional coordinate data of the target feature point based on the remote sensing coordinate data and the fitted surface model.

[0009] In some embodiments of the present application, the determining the fitted surface model based on the ground form of the current scene, the planar image data and the point cloud image data comprises: selecting feature points based on the planar image data, determining a third set of feature points corresponding to the planar image data; determining an initial fitted surface model according to the ground form of the current scene, the initial fitted surface model comprising at least one fitting parameter; and fitting and calculating each fitting parameter according to the third set of feature points, a set of point cloud points in the point cloud image data matched with the third set of feature points and the initial fitted surface model, to determine the fitted surface model.

[0010] In some embodiments of the present application, based on the foregoing scheme, the selecting feature points based on the planar image data and determining a third set of feature points corresponding to the planar image data comprises: selecting feature points based on the planar image data; determining three-dimensional coordinate data corresponding to each feature point from a satellite map, and constructing a fourth set of feature points; determining coordinates of each feature point of the fourth set of feature points in a world coordinate system through offsetting and longitude and latitude conversion according to the feature point coordinate data of the fourth set of feature points; and constructing the third set of feature points corresponding to the planar image data according to the coordinates of each feature point of the fourth set of feature points in the world coordinate system.

[0011] In some embodiments of the present application, the determining of the coordinate mapping relationship based on the feature point coordinate data in the first feature point set and the second feature point set comprises: determining feature point coordinate data corresponding to each sub-surface based on the feature point coordinate data in the first feature point set and the second feature point set; and determining a coordinate mapping relationship corresponding to each sub-surface respectively according to the feature point coordinate data corresponding to each sub-surface.

[0012] In some embodiments of the present application, based on the foregoing scheme, the coordinate mapping relationship is a conversion matrix, and the determining of the coordinate mapping relationship corresponding to each sub-surface respectively according to the feature point coordinate data corresponding to each sub-surface comprises: for a target sub-surface, which is any one of the sub-surfaces; S1: generating a random conversion matrix, and performing S2; S2: converting the feature point coordinate data of the target sub-surface in the second feature point set into two-dimensional coordinate data through the random conversion matrix to obtain a predicted feature point set, and performing S3; S3: calculating an average Euclidean distance value between corresponding feature points in the predicted feature point set and the first feature point set, and performing S4; S4: if the average Euclidean distance value is greater than or equal to a preset distance value, performing S1, and if the average Euclidean distance value is less than the preset distance value, performing S5; S5: outputting an inverse matrix of the random conversion matrix to determine the coordinate mapping relationship of the target sub-surface.

[0013] According to an aspect of an embodiment of the present application, a plane image conversion device is provided, the device comprising: an acquisition unit configured to acquire plane image data and corresponding point cloud image data, determine a fitted surface model based on a ground form of a current scene, the plane image data and the point cloud image data, the fitted surface model comprising at least two sub-surfaces with different curvatures; a first determination unit configured to determine a first feature point set from the plane image data, and determine a second feature point set based on the point cloud image data and the fitted surface model; a second determination unit configured to determine a coordinate mapping relationship based on feature point coordinate data in the first feature point set and the second feature point set, the coordinate mapping relationship being used to represent a corresponding relationship between two-dimensional coordinates of a feature point in the plane image data and three-dimensional coordinates of the feature point in a world coordinate system; and a third determination unit configured to determine three-dimensional coordinates of each pixel point of the plane image data according to the coordinate mapping relationship, so as to complete conversion of the plane image data.

[0014] According to an aspect of an embodiment of the present application, a computer readable storage medium is provided, the computer readable storage medium storing at least one program code, the at least one program code being loaded and executed by a processor to implement operations performed by the conversion method of the plane image as described.

[0015] According to an aspect of some embodiments of the present application, an electronic device is provided, which includes one or more processors and one or more memories having at least one program code stored therein, the at least one program code being loaded and executed by the one or more processors to implement operations performed by the method for converting a planar image as described.

[0016] Based on the above scheme, the present application has at least the following advantages or progresses:

[0017] In the technical scheme provided in some embodiments of the present application, by acquiring planar image data and corresponding point cloud image data, and determining a fitted curved surface model based on the ground form of the current scene, the planar image data and the point cloud image data, then determining a first feature point set and a second feature point set from the planar image data and the fitted curved surface model, and further determining the coordinate mapping relationship between the two-dimensional coordinates and the three-dimensional coordinates, the planar image data is converted into a three-dimensional image. The conversion process does not require the internal and external parameter data of the device, and the coordinate mapping relationship can replace the internal and external parameter data to convert the planar image data into a three-dimensional image.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0020] Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0021] In the drawings:

[0022] Figure 1 A flowchart of a method for converting a planar image according to an embodiment of the present application is shown;

[0023] Figure 2 A flowchart of a method for converting a planar image according to an embodiment of the present application is shown;

[0024] Figure 3 A frame of image data in video data for a preset location in an embodiment of the present application is shown;

[0025] Figure 4 Point cloud image data for a preset location in an embodiment of the present application is shown;

[0026] Figure 5 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0027] Figure 6 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0028] Figure 7 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0029] Figure 8 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0030] Figure 9 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0031] Figure 10 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0032] Figure 11 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0033] Figure 12 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application;

[0034] Figure 13 FIG. 8 shows a flowchart of a method for converting a planar image according to an embodiment of the present application; DETAILED DESCRIPTION

[0035] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0036] Moreover, described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, methods, and so forth have not been shown or described in detail to avoid obscuring aspects of the application.

[0037] The block diagrams shown in the drawings are merely functional entities and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] The flowcharts shown in the drawings are merely exemplary illustrations and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0039] The implementation details of the technical solutions of the embodiments of the present application are described in detail as follows:

[0040] Please refer to Figure 1 .

[0041] Figure 1 A flowchart of a conversion method of a planar image according to an embodiment of the present application is shown, as shown in Figure 1 , the method can include steps S101-S104:

[0042] Step S101, obtain planar image data and corresponding point cloud image data, determine a fitted curved surface model based on the ground form of the current scene, the planar image data and the point cloud image data, the fitted curved surface model includes at least two sub-curved surfaces with different curvatures.

[0043] Step S102, determine a first feature point set from the planar image data, and determine a second feature point set based on the point cloud image data and the fitted curved surface model.

[0044] Step S103, determine a coordinate mapping relationship based on the feature point coordinate data in the first feature point set and the second feature point set, the coordinate mapping relationship is used to represent the corresponding relationship between the two-dimensional coordinates of the feature points in the planar image data and the three-dimensional coordinates in the world coordinate system.

[0045] Step S104, determine the three-dimensional coordinates of each pixel point of the planar image data according to the coordinate mapping relationship, to complete the conversion of the planar image data.

[0046] In this application, a fitted surface model is determined by acquiring planar image data and corresponding point cloud image data; a first feature point set and a second feature point set are determined from the planar image data and the fitted surface model, and then the coordinate mapping relationship between two-dimensional coordinates and three-dimensional coordinates is determined, so as to convert the planar image data into a three-dimensional image. The conversion process does not require the intrinsic and extrinsic parameter data of the device, and the intrinsic and extrinsic parameter data can be replaced by the coordinate mapping relationship to convert the planar image data into a three-dimensional image.

[0047] Please see Figure 2 .

[0048] Figure 2 A flowchart of a planar image conversion method according to an embodiment of this application is shown, as follows: Figure 2 As shown, the method for acquiring planar image data and corresponding point cloud image data may include steps S201-S202:

[0049] Step S201: Obtain at least one frame of image data from the video data for the preset location as planar image data.

[0050] Step S202: Obtain lidar scanning data for the preset location, parse the lidar scanning data, and obtain point cloud image data.

[0051] In this application, a camera device can be installed at a preset location to take a video or photograph of the preset location. At the same time, a point cloud scan can be performed on the preset location using a LiDAR. The point cloud scan can represent the elevation and undulation of the preset location through point cloud images, but the captured image data cannot reflect the elevation and undulation situation.

[0052] For example, Figure 3 This application illustrates a frame of image data from video data for a preset location in one embodiment, such as... Figure 3 As shown, the current frame of image data can be used as the planar image data described in this application, while... Figure 3 It is not difficult to see that planar image data cannot reflect the undulations of the road surface at an intersection.

[0053] For example, Figure 4 Point cloud image data for a preset location is shown in one embodiment of this application, such as... Figure 4 As shown, the road surface of the intersection at the preset location is not a standard plane but a curved surface with three slopes, and the point cloud data of the concave middle part of the intersection is extremely scarce.

[0054] Please see Figure 5 .

[0055] Figure 5A flow chart of a conversion method of a planar image according to an embodiment of the present application is shown in FIG. 1. Figure 5 As shown in FIG. 1, the method of determining a first feature point set from the planar image data and determining a second feature point set based on the point cloud image data and the fitted surface model can include steps S501-S504:

[0056] Step S501, selecting feature points from the planar image data, determining two-dimensional coordinate data of each feature point, and constructing a first feature point set.

[0057] Step S502, based on the point cloud image data, reading three-dimensional coordinate data of at least one feature point.

[0058] Step S503, based on the two-dimensional coordinate data of a target feature point and the fitted surface model, calculating three-dimensional coordinate data of the target feature point, the target feature point being a feature point that cannot read three-dimensional coordinate data from the point cloud image data.

[0059] Step S504, constructing a second feature point set according to three-dimensional coordinate data of each feature point.

[0060] Please refer to Figure 6 .

[0061] Figure 6 A flow chart of a conversion method of a planar image according to an embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the method of determining a first feature point set from the planar image data and determining a second feature point set based on the point cloud image data and the fitted surface model can include steps S501-S504:

[0062] Step S601, determining remote sensing coordinate data of the target feature point matched in the remote sensing satellite map according to the two-dimensional coordinate data of the target feature point.

[0063] Step S602, based on the remote sensing coordinate data and the fitted surface model, calculating three-dimensional coordinate data of the target feature point.

[0064] In the present application, feature points can be found in planar image data and their positions in point cloud image data can be determined. Due to complex terrain, point cloud image data cannot cover all feature points, at which time the coordinates of missing feature points on point cloud image data can be determined through a fitted surface model.

[0065] For example, for Figure 3 planar image data shown in FIG. 2 and Figure 4 point cloud image data shown in FIG. 3, it can be known from Figure 4 that there are more feature points close by and it is easier to match Figure 3Feature point matching in the point cloud involves matching near-end points with the bottom center points of the 3D anchor frames from the 3D detection results of vehicles and pedestrians in the point cloud, obtaining several pairs of one-to-one corresponding curves. The point sets of these curves are the found 2D and 3D calibration point sets, denoted as {α1} and {β1}, respectively. However, distant points are sparse or missing and difficult to match... Figure 3 Matching and locating feature points can lead to problems such as a lack of or inability to find distant feature points. In such cases, a more sophisticated approach can be adopted. Figure 3 Feature points {α2} on streetlights, ground markings, and zebra crossings in the mid-to-far distance are matched with these feature points {γ} in the remote sensing satellite map. Since {γ} is the projection of the point set {β2} corresponding to {α2} in the world coordinate system onto the xy plane, the x and y values ​​of {γ} can be substituted into Formula 1 of the fitted surface model to solve for the value of z, thus obtaining {β2}. {α1}, {α2} and {β1}, {β2} are then merged to obtain the first feature point set {α} and the second feature point set {β}, respectively. The more points in {α} and {β}, the wider the coverage, and the better the performance of subsequent calculations. Please refer to [link to relevant documentation]. Figure 7 .

[0066] Figure 7 A flowchart of a planar image conversion method according to an embodiment of this application is shown, as follows: Figure 7 As shown, the method for determining the fitted surface model based on the planar image data and the point cloud image data may include steps S701-S703:

[0067] Step S701: Based on the planar image data, select feature points to determine the third feature point set corresponding to the planar image data.

[0068] Step S702: Determine an initial fitted surface model based on the ground morphology of the current scene. The initial fitted surface model includes at least one fitting parameter.

[0069] Step S703: Based on the third feature point set, the point cloud point set in the point cloud image data that matches the third feature point set, and the initial fitted surface model, calculate each fitting parameter to determine the fitted surface model. (See also...) Figure 8 .

[0070] Figure 8 A flowchart of a planar image conversion method according to an embodiment of this application is shown, as follows: Figure 8 As shown, in step S701, the method for selecting feature points and determining the third feature point set corresponding to the planar image data based on the planar image data may include steps S801-S804:

[0071] Step S801: Select feature points based on the planar image data.

[0072] Step S802, determine the three-dimensional coordinate data corresponding to each feature point in the satellite map, and construct a fourth feature point set.

[0073] Step S803, for the feature point coordinate data of the fourth feature point set, determine the coordinates of each feature point of the fourth feature point set in the world coordinate system through offset and latitude and longitude conversion.

[0074] Step S804, according to the coordinates of each feature point of the fourth feature point set in the world coordinate system, construct a third feature point set corresponding to the planar image data.

[0075] In this application, feature points need to be selected on the planar image, and in the subsequent process, the three-dimensional coordinate data of each feature point needs to be determined on the open source satellite map, so as to select as many stationary feature points as possible on the planar image, so as to Figure 3 For example, feature points on the ground lane direction markers, zebra crossings, lane lines and other targets can be selected.

[0076] In this application, since the actual ground form corresponding to the planar image is usually not a plane, there will be ups and downs, so the selection of feature points needs to cover different curved surfaces as much as possible, and the construction of the fitting curved surface model also needs to consider the actual ground form. For example, as shown in the point cloud image data, Figure 4 It can be known that the road surface of the intersection can be roughly divided into three sections, so a sufficient number of feature points can be selected on the three curved surfaces respectively to approach the actual ground form as much as possible.

[0077] In this application, according to the selected feature points on the planar image data, the three-dimensional coordinate data corresponding to each feature point can be determined from the open source satellite map, that is, the fourth feature point set {k} of the planar image data can be constructed, and the coordinates data of each feature point in the world coordinate system can be easily obtained through offset and latitude and longitude conversion, so as to construct the third feature point set {s}, whose x direction of the world coordinate system H is north, y direction is west, z direction is vertical to the ground upward, and the unit is m.

[0078] In this application, an initial fitting curved surface model can be determined according to the point cloud image data and / or the ground form of the current scene, for example, as shown in the point cloud image data, Figure 4 It can be known that the road surface of the intersection can be roughly divided into three sections, which can be fitted by a spatial quadratic surface, and its expression can be:

[0079] x 2 +ay 2 +bz 2 +cx+dy+ez+f=0 (1)

[0080] In the present application, the three-dimensional coordinate data of each feature point can be extracted from the third feature point set {s}, and then the values of the fitting parameters a, b, c, d, e, and f in the formula can be solved by the inverse solution of x i , y i , and z i . The approximate matrix equation can be obtained by the least square method as follows:

[0081]

[0082] where x i , y i , and z i are the values of a single point in the world coordinate system H in the x, y, and z directions of the calibration point set {s}, respectively. After determining the fitting parameters, the fitting surface model can be generated. For example, please refer to Figure 9 , Figure 9 which shows the fitting surface model in an embodiment of the present application.

[0083] Please refer to Figure 10 .

[0084] Figure 10 which shows the flowchart of the conversion method of the planar image according to an embodiment of the present application. As shown in Figure 10 , the method of determining the coordinate mapping relationship based on the feature point coordinate data in the first feature point set and the second feature point set can include steps S1001-S1002:

[0085] Step S1001, determining the feature point coordinate data corresponding to each sub-surface based on the feature point coordinate data in the first feature point set and the second feature point set.

[0086] Step S1002, determining the coordinate mapping relationship corresponding to each sub-surface respectively according to the feature point coordinate data corresponding to each sub-surface.

[0087] In the present application, since the actual ground form corresponding to the planar image is usually not a plane, there will be ups and downs, and therefore the fitting surface model obtained by fitting usually has multiple sub-surfaces. In order to improve the coordinate mapping accuracy, the calculation of the corresponding coordinate mapping relationship can be performed for each sub-surface.

[0088] In the present application, for any one sub-surface, the pixel coordinate system and camera coordinate system conversion relationship formula can be obtained as follows:

[0089]

[0090] wherein, can represent the intrinsic parameters of the camera, The extrinsic parameters of the camera can be characterized, and since both the intrinsic and extrinsic parameters are unknown, the coordinate system mapping relationship can be abstracted as a matrix M.

[0091] wherein the extrinsic parameters φ, θ, t are constant values, and f x , f y , u0, v0 are also constant values, although z c is not constant, multiplying z c on both sides of equation (3) gives

[0092] Therefore, the coordinates of the projection in the undistorted image can be directly calculated, and this coordinate can be used to restore the undistorted image to the original image by using three radial distortion coefficients k1, k2, k3 and two tangential distortion coefficients p1, p2, and the conversion relationship formula can be as follows:

[0093] u' = u (1 + k1r 2 +k2r 4 +k3r 6 )+2p1v+p2(r 2 +2u 2 ) (4)

[0094] v' = v (1 + k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2v 2 )+2p2u (5)

[0095] wherein u' and v' are the x and y coordinates of the corrected image, u and v are the x and y coordinates of the original image, and r is the normalized distance of the pixel point from the image center.

[0096] Therefore, the matrix and the distortion coefficients can be found by using the exhaustive approximation method on the same horizontal plane, that is, the conversion relationship between any pixel point represented in the planar image data and the corresponding 3D coordinates of the corresponding point in the world coordinate system of the plane can be found by finding the matrix M.

[0097] Please refer to Figure 11 .

[0098] Figure 11 A flowchart of a conversion method of a planar image according to an embodiment of the present application is shown, as shown in Figure 11 the coordinate mapping relationship can be a conversion matrix, and the method of determining the coordinate mapping relationship corresponding to each sub-surface according to the feature point coordinate data corresponding to each sub-surface can include steps S1-S5:

[0099] S1: generating a random conversion matrix, performing S2.

[0100] S2: converting the feature point coordinate data of the target sub-curve in the second feature point set into two-dimensional coordinate data through the random conversion matrix, obtaining a predicted feature point set, and performing S3.

[0101] S3: calculating the average Euclidean distance value between corresponding feature points in the predicted feature point set and the first feature point set, and performing S4.

[0102] S4: if the average Euclidean distance value is greater than or equal to a preset distance value, performing S1, and if the average Euclidean distance value is less than the preset distance value, performing S5.

[0103] S5: outputting the inverse matrix of the random conversion matrix to determine the coordinate mapping relationship of the target sub-curve.

[0104] In the present application, the coordinate mapping relationship can be abstracted as a conversion matrix. By calculating the product of the second feature point set and the conversion matrix, the predicted feature point set can be obtained. Then, by comparing the coordinate data of each feature point in the predicted feature point set and the first feature point set, it can be determined whether the current conversion matrix meets the accuracy requirement of coordinate conversion.

[0105] In the present application, a suitable random conversion matrix can be obtained by exhaustive approximation. For example, an existing random conversion matrix M1 is multiplied by the second feature point set to obtain a predicted feature point set E1. Then, the average Euclidean distance between each feature point in E1 and the first feature point set is calculated. The Euclidean distance can represent the deviation between each feature point, and the average Euclidean distance can represent the overall deviation between the two point sets. If the calculated average Euclidean distance value is greater than or equal to a preset distance value, a new random conversion matrix is generated and subsequent calculations are performed. If the average Euclidean distance value is less than the preset distance value, it means that the current conversion matrix can meet the conversion accuracy requirement, and the inverse matrix of the current random conversion matrix can be output to determine the coordinate mapping relationship.

[0106] For example, the first feature point set corresponding to an existing planar image, the actual road surface form corresponding to the planar image is a completely horizontal plane:

[0107]

[0108] That is, in the first feature point set, there are A(1, 0), B(0, 0), C(1, 1), and D(0, 1).

[0109] The existing random conversion matrix M1 is multiplied by the second feature point set to obtain a predicted feature point set E1:

[0110]

[0111] That is, in the predicted feature point set E1, there are A1(2, 3), B1(5, 7), C1(4, 5), and D1(1, 2).

[0112] The Euclidean distance values between each corresponding feature point can be calculated respectively:

[0113] A-A2:

[0114] B-B2:

[0115] C-C2:

[0116] D-D2:

[0117] It is not difficult to calculate that the average Euclidean distance value L1 is:

[0118] The average Euclidean distance value L1 is greater than the preset distance value 0.5, so it does not meet the requirements, and a random conversion matrix M2 needs to be generated again.

[0119] After multiplying M2 and the second feature point set, the predicted feature point set E2 can be obtained:

[0120]

[0121] That is, in the predicted feature point set E2, there are A2(1.1, 0.1), B2(0.1, 0.1), C2(1.2, 1.3), and D2(0.1, 1.5).

[0122] The Euclidean distance values between each corresponding feature point can be calculated respectively:

[0123] A-A2:

[0124] B-B2:

[0125] C-C2:

[0126] D-D2:

[0127] It is not difficult to calculate that the average Euclidean distance value L2 is:

[0128] The average Euclidean distance value L2 is less than the preset distance value 0.5, so it meets the requirements, and the inverse matrix M2' of the random conversion matrix M2 can be output, which is used to represent the coordinate conversion relationship.

[0129] For example, the actual road surface corresponding to the planar image has ups and downs, and the corresponding fitting surface model has two sub-surfaces A and B:

[0130] For sub-surface A, the corresponding first feature point set is obtained:

[0131]

[0132] That is, in the first feature point set, there are A(1, 0), B(0, 0), C(1, 1), and D(0, 1).

[0133] The existing random conversion matrix M3 is multiplied by the second feature point set to obtain the predicted feature point set E3:

[0134]

[0135] That is, in the predicted feature point set E3, there are A3(1.2, 0.2), B3(0.05, 0), C3(1.1, 1.3), and D3(0.2, 2).

[0136] The Euclidean distance values between each corresponding feature point can be calculated respectively:

[0137] A-A3:

[0138] B-B3:

[0139] C-C3:

[0140] D-D3:

[0141] The average Euclidean distance value L3 is calculated as:

[0142] The average Euclidean distance value L3 is less than the preset distance value 0.6, so it meets the requirements, and the inverse matrix M3' of the random conversion matrix M3 can be output to represent the coordinate conversion relationship on the sub-surface A.

[0143] For sub-surface B, the corresponding first feature point set is obtained:

[0144]

[0145] That is, in the first feature point set, there are A(1, 0), B(0, 0), C(1, 1), and D(0, 1).

[0146] The existing random conversion matrix M4 is multiplied by the second feature point set to obtain the predicted feature point set E4:

[0147]

[0148] That is, in the predicted feature point set E4, there are A4(1.1, 1.1), B4(0.1, 1.2), C4(1.2, 2.3), and D4(0.1, 2.5).

[0149] The Euclidean distance values between each corresponding feature point can be calculated respectively:

[0150] A-A4:

[0151] B-B4:

[0152] C-C4:

[0153] D-D4:

[0154] It is not difficult to calculate that the average Euclidean distance value L3 is:

[0155] The average Euclidean distance value L3 is less than the preset distance value 0.6, so it meets the requirements, and the inverse matrix M4' of the random conversion matrix M4 can be output and can be used to represent the coordinate conversion relationship on the sub-surface B.

[0156] Therefore, the inverse matrices M3' and M4' can be output and can be used as the final coordinate conversion relationship. The pixel points at different positions on the plane image can correspond to different sub-surfaces, so that different inverse matrices are called to perform coordinate conversion.

[0157] In the embodiment of the present application, the coordinate mapping relationship between the two-dimensional coordinates and the three-dimensional coordinates can be determined by the feature points in the plane image and the feature points in the fitted surface model corresponding to the actual ground shape, the two-dimensional coordinates of each pixel point in the plane image are directly converted into three-dimensional coordinates, and the conversion process does not require the internal and external parameter data of the device, and the internal and external parameter data can be replaced by the coordinate mapping relationship. Moreover, in the embodiment of the present application, when the coordinate mapping relationship is constructed, the feature points in the three-dimensional space corresponding to the surfaces with different curvatures are selected to construct the coordinate mapping relationship in the space, so that the accuracy of the coordinate mapping relationship is improved, and the accuracy of the conversion of the image data into the point cloud data is improved.

[0158] Next, the device embodiment of the present application will be described in combination with the accompanying drawings.

[0159] Please refer to Figure 12 .

[0160] Figure 12A structural diagram of a planar image conversion apparatus according to an embodiment of the present application is shown. The planar image conversion apparatus 1200 can include an acquisition unit 1201, a first determination unit 1202, a second determination unit 1203, and a third determination unit 1204.

[0161] The apparatus 1200 can be configured as follows: the acquisition unit 1201 is configured to acquire planar image data and corresponding point cloud image data, determine a fitted curved surface model based on a ground shape of a current scene, the planar image data, and the point cloud image data, the fitted curved surface model including at least two sub-curved surfaces with different curvatures; the first determination unit 1202 is configured to determine a first set of feature points from the planar image data, and determine a second set of feature points based on the point cloud image data and the fitted curved surface model; the second determination unit 1203 is configured to determine a coordinate mapping relationship based on coordinate data of feature points in the first set of feature points and the second set of feature points, the coordinate mapping relationship being used to represent a corresponding relationship between two-dimensional coordinates of feature points in the planar image data and three-dimensional coordinates in a world coordinate system; and the third determination unit 1204 is configured to determine three-dimensional coordinates of each pixel point of the planar image data according to the coordinate mapping relationship, so as to complete conversion of the planar image data.

[0162] Next, refer to Figure 13 .

[0163] Figure 13 A structural diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown.

[0164] It should be noted that Figure 13 The computer system 1300 of the electronic device shown is only an example, and should not impose any limitation on the functions and use range of embodiments of the present application.

[0165] As Figure 13 shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1302 or programs loaded from a storage portion 1308 into a random access memory (RAM) 1303, such as performing the methods described in the above embodiments. In the RAM 1303, various programs and data required for system operation are also stored. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other through a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0166] The following components are connected to the I / O interface 1305: an input part 1306 including a keyboard, a mouse, etc.; an output part 1307 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage part 1308 including a hard disk, etc.; and a communication part 1309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as necessary. A removable media 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1310 as necessary, so that a computer program read out therefrom is installed in the storage part 1308 as necessary.

[0167] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 1309, and / or installed from the removable media 1311. When the computer program is executed by the central processing unit (CPU) 1301, various functions defined in the system of the present application are executed.

[0168] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carrying computer-readable program code in a baseband or as a part of a carrier wave. Such a propagated data signal can take on various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium that can send, propagate or transmit a program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0169] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0170] The units described in the embodiments of the present application can be implemented by software, or can be implemented by hardware, and the units described can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0171] As another aspect, the present application also provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the conversion method of the planar image described in the above embodiments.

[0172] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the conversion method of the planar image described in the above embodiments.

[0173] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into several modules or units.

[0174] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or network, and includes several instructions to make a computing device (which can be a personal computer, server, touch terminal, or network device, etc.) execute the methods according to the embodiments of the present application.

[0175] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the application following the general principles thereof and including such departures from the present disclosure as come within known use or custom in the art.

[0176] It is to be understood that the application is not limited to the precise construction already described above and shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should only be limited by the claims appended hereto.

Claims

1. A conversion method of a planar image, characterized by, The method comprises: acquiring planar image data and corresponding point cloud image data, determining a fitted curved surface model based on the ground shape of the current scene, the planar image data and the point cloud image data, the fitted curved surface model comprising at least two sub-curved surfaces with different curvatures; determining a first feature point set from the planar image data, and determining a second feature point set based on the point cloud image data and the fitted curved surface model; determining a coordinate mapping relationship based on the feature point coordinate data in the first feature point set and the second feature point set, the coordinate mapping relationship being used to represent the corresponding relationship between the two-dimensional coordinates of the feature points in the planar image data and the three-dimensional coordinates in the world coordinate system; determining the three-dimensional coordinates of each pixel point of the planar image data according to the coordinate mapping relationship, so as to complete the conversion of the planar image data; wherein the determination of the first feature point set from the planar image data and the determination of the second feature point set based on the point cloud image data and the fitted curved surface model comprise: selecting feature points from the planar image data, determining the two-dimensional coordinate data of each feature point, and constructing the first feature point set; correspondingly reading the three-dimensional coordinate data of at least one feature point based on the point cloud image data; calculating the three-dimensional coordinate data of the target feature point based on the two-dimensional coordinate data of the target feature point and the fitted curved surface model, the target feature point being a feature point whose three-dimensional coordinate data cannot be read from the point cloud image data; constructing the second feature point set according to the three-dimensional coordinate data of each feature point; the determination of the fitted curved surface model based on the ground shape of the current scene, the planar image data and the point cloud image data comprises: selecting feature points based on the planar image data, and determining a third feature point set corresponding to the planar image data; determining an initial fitted curved surface model according to the ground shape of the current scene, the initial fitted curved surface model comprising at least one fitting parameter; fitting and calculating each fitting parameter according to the third feature point set, the point cloud point set matched with the third feature point set in the point cloud image data and the initial fitted curved surface model, so as to determine the fitted curved surface model by using the least square method.

2. The method of claim 1, wherein, the calculation of the three-dimensional coordinate data of the target feature point based on the two-dimensional coordinate data of the target feature point and the fitted curved surface model comprises: determining the remote sensing coordinate data matched with the target feature point in the remote sensing satellite map according to the two-dimensional coordinate data of the target feature point; calculating the three-dimensional coordinate data of the target feature point based on the remote sensing coordinate data and the fitted curved surface model.

3. The method of claim 1, wherein, the selection of feature points based on the planar image data and the determination of a third feature point set corresponding to the planar image data comprise: selecting feature points based on the planar image data; determining the three-dimensional coordinate data corresponding to each feature point from the satellite map, and constructing a fourth feature point set; determining the coordinates of each feature point of the fourth feature point set in the world coordinate system through offsetting and latitude-longitude conversion based on the feature point coordinate data of the fourth feature point set; According to coordinates of each feature point in the world coordinate system, a third feature point set corresponding to the planar image data is constructed.

4. The method of claim 1, wherein, The determining of the coordinate mapping relationship based on the feature point coordinate data in the first feature point set and the second feature point set comprises: The feature point coordinate data corresponding to each sub-surface is determined based on the first feature point set and the second feature point set. The coordinate mapping relationship corresponding to each sub-surface is determined respectively according to the feature point coordinate data corresponding to each sub-surface.

5. The method of claim 4, wherein, The coordinate mapping relationship is a conversion matrix, and the coordinate mapping relationship corresponding to each sub-surface is determined respectively according to the feature point coordinate data corresponding to each sub-surface, comprising: For a target sub-surface, the target sub-surface is any one of the sub-surfaces; S1: generating a random conversion matrix, and performing S2; S2: converting the feature point coordinate data of the target sub-surface in the second feature point set into two-dimensional coordinate data through the random conversion matrix to obtain a predicted feature point set, and performing S3; S3: calculating an average Euclidean distance value between corresponding feature points in the predicted feature point set and the first feature point set, and performing S4; S4: if the average Euclidean distance value is greater than or equal to a preset distance value, performing S1, and if the average Euclidean distance value is less than the preset distance value, performing S5; S5: outputting an inverse matrix of the random conversion matrix to determine the coordinate mapping relationship of the target sub-surface.

6. A planar image conversion device, characterized by The device comprises: An acquisition unit is configured to acquire planar image data and corresponding point cloud image data, determine a fitted surface model based on a ground shape of a current scene, the planar image data and the point cloud image data, and the fitted surface model comprises at least two sub-surfaces with different curvatures; A first determination unit is configured to determine a first feature point set from the planar image data, and determine a second feature point set based on the point cloud image data and the fitted surface model; A second determination unit is configured to determine a coordinate mapping relationship based on feature point coordinate data in the first feature point set and the second feature point set, and the coordinate mapping relationship is used to represent a corresponding relationship between two-dimensional coordinates of a feature point in the planar image data and three-dimensional coordinates of the feature point in a world coordinate system; A third determination unit is configured to determine three-dimensional coordinates of each pixel point of the planar image data according to the coordinate mapping relationship, so as to complete conversion of the planar image data. The determination of the first feature point set from the planar image data and the determination of the second feature point set based on the point cloud image data and the fitted surface model comprise: Selecting feature points from the planar image data, determining two-dimensional coordinate data of each feature point, and constructing a first feature point set; Reading three-dimensional coordinate data of at least one feature point based on the point cloud image data; Calculating three-dimensional coordinate data of a target feature point based on two-dimensional coordinate data of the target feature point and the fitted surface model, the target feature point being a feature point whose three-dimensional coordinate data cannot be read from the point cloud image data; According to the three-dimensional coordinate data of each feature point, a second feature point set is constructed; The fitting curved surface model is determined based on the ground shape of the current scene, the planar image data and the point cloud image data, including: Based on the planar image data, a feature point is selected, and a third feature point set corresponding to the planar image data is determined; According to the ground shape of the current scene, an initial fitting curved surface model is determined, and the initial fitting curved surface model includes at least one fitting parameter; According to the third feature point set, a point cloud point set matched with the third feature point set in the point cloud image data, and the initial fitting curved surface model, each fitting parameter is fitted and calculated to determine the fitting curved surface model by using the least square method.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to realize the operations performed by the conversion method of the planar image according to any one of claims 1 to 5.

8. An electronic device, comprising: The electronic device includes one or more processors and one or more memories, and the one or more memories store at least one program code, and the at least one program code is loaded and executed by the one or more processors to realize the operations performed by the conversion method of the planar image according to any one of claims 1 to 5.

Images