Three-dimensional back projection method, apparatus and electronic device

Abstract

The application discloses a three-dimensional back projection method, device and electronic equipment. According to the embodiments provided in the application, the three-dimensional coordinates of other back projection points in a three-dimensional space are solved by using point data of known end points or vertexes, a new back projection method is provided for a scene in which points are marked on a two-dimensional image and point data needs to be determined in a corresponding three-dimensional space. The method provided in the application solves by using point data of known points, ensures the accuracy of back projection, can provide a high-precision three-dimensional back projection scheme in various application scenes requiring three-dimensional space reconstruction, and avoids errors caused by manual calculation.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to a three-dimensional back projection method, device, and electronic device. Background Technology

[0002] Projecting 3D space onto 2D images is a mature technology. In many scenarios, such as map navigation, vehicle navigation, and art exhibitions, points in 3D space can be projected onto 2D images using camera projection principles. However, in some scenarios, it is necessary to project from 2D images back into 3D space. 3D spatial reconstruction has been a hot research topic in computer vision and graphics in recent years. 3D reconstruction technology has promising applications in various fields, including game simulation, VR and AR, map navigation, environmental monitoring, and vehicle driving. However, the reconstruction process does not utilize all coordinate points of the 2D image; it often uses landmarks or boundary points of typical regions in the image. This results in insufficient spatial accuracy in the reconstructed map and a poor user experience. Summary of the Invention

[0003] Based on this, the present invention aims to propose a three-dimensional back-projection method, device and electronic device, which uses the already projected coordinate points to solve the three-dimensional coordinates of the other coordinate points in the same image in three-dimensional space back-projection, so as to improve the accuracy of three-dimensional space reconstruction and overcome the defects of the prior art.

[0004] In a first aspect, the present invention provides a three-dimensional back-projection method for solving the three-dimensional coordinates of a point to be back-projected that is collinear with two known endpoints. The two-dimensional coordinates of the point to be back-projected have already been obtained, including:

[0005] Endpoint initialization steps: Determine the first and second endpoints on the same straight line as the point to be back-projected. The two-dimensional and three-dimensional coordinates of the first and second endpoints have been obtained. The point to be back-projected is located between the first and second endpoints on the straight line.

[0006] Endpoint update steps: Calculate the endpoint distance using the 3D coordinates of the first endpoint and the second endpoint. If the endpoint distance is less than the first distance threshold, set the 3D coordinates of the first endpoint or the second endpoint as the 3D coordinates of the point to be back-projected. Otherwise, proceed to the following segment ternary trisection solution steps.

[0007] The steps for solving the segment trisection method are as follows: Determine the trisection points of the three-dimensional endpoint line segment formed by the first endpoint and the second endpoint. Compare the distances of the two-dimensional projection points of the trisection points with the point to be inverted. The trisection point with a larger distance from the point to be inverted is updated as the first endpoint or the second endpoint, and the process returns to the endpoint update steps described above.

[0008] Furthermore, the steps for solving the line segment ternary method include:

[0009] Determine two trisection points of the three-dimensional endpoint line segment, including the first trisection point closer to the first endpoint and the second trisection point closer to the second endpoint;

[0010] Obtain the first projection point of the first third point onto the two-dimensional image, and the second projection point of the second third point onto the two-dimensional image;

[0011] Calculate the first projection distance between the first projection point and the point to be reverse-projected, and the second projection distance between the second projection point and the point to be reverse-projected;

[0012] Compare the first projection distance and the second projection distance. If the first projection distance is larger, update the first third point as the first endpoint. If the second projection distance is larger, update the second third point as the second endpoint.

[0013] Return to the endpoint update step.

[0014] Furthermore, the endpoint update steps include:

[0015] When the distance between the endpoints is less than the first distance threshold, the three-dimensional coordinates of the first endpoint are set to the three-dimensional coordinates of the point to be back-projected.

[0016] Secondly, the present invention provides a three-dimensional back-projection method for solving the three-dimensional coordinates of a point to be back-projected located within a triangular region formed by three known vertices, wherein the two-dimensional coordinates of the point to be back-projected have already been obtained, including:

[0017] Choose any side of the triangular region, with the two endpoints of the side being the first vertex and the second vertex. Find the collinear points on the side, which are collinear with the point to be projected and the third vertex.

[0018] Define the collinear point and the third vertex as the first and second endpoints, and the point to be back-projected is located between the collinear point and the third vertex;

[0019] The three-dimensional coordinates of the point to be back-projected are obtained using the method described in the first aspect.

[0020] Furthermore, the steps for finding collinear points include:

[0021] Vertex update steps: Calculate the vertex distance using the 3D coordinates of the first vertex and the second vertex. If the vertex distance is less than the second distance threshold, set the 3D coordinates of the first vertex or the second vertex to the 3D coordinates of the collinear point; otherwise, proceed to the following point-line distance calculation steps.

[0022] Point-to-line distance calculation steps: Determine the trisection points of the three-dimensional vertex segment formed by the first vertex and the second vertex. Calculate the two-dimensional projection line segments of the two trisection points and the third vertex respectively. Compare the point-to-line distances of the two two-dimensional projection line segments and the point to be inverted. The trisection point with the larger point-to-line distance is updated as the first vertex or the second vertex, and return to the vertex update steps above.

[0023] Furthermore, the steps for calculating the distance between points and lines include:

[0024] Determine two trisection points of a 3D vertex segment, including the third trisection point closer to the first vertex and the fourth trisection point closer to the second vertex;

[0025] Obtain the third projection point of the third trisection point projected onto the two-dimensional image, and the fourth projection point of the fourth trisection point projected onto the two-dimensional image.

[0026] Find the first two-dimensional projection line segment between the third projection point and the third vertex, and the second two-dimensional projection line segment between the fourth projection point and the third vertex;

[0027] Compare the distance between the first point and the second point. The distance between the first point and the second point is the distance between the first two-dimensional projection line segment and the point to be inverted and projected. The distance between the second point and the second point is the distance between the second two-dimensional projection line segment and the point to be inverted and projected. If the distance between the first point and the second point is greater, then update the third third point as the first vertex. If the distance between the second point and the third point is greater, then update the fourth third point as the second vertex.

[0028] Return to the vertex update steps.

[0029] Furthermore, the vertex update steps include:

[0030] When the vertex distance is less than the second distance threshold, let the three-dimensional coordinates of the first vertex be the three-dimensional coordinates of the point to be back-projected.

[0031] Thirdly, the present invention provides a three-dimensional back-projection device for solving the three-dimensional coordinates of a point to be back-projected that is collinear with two known endpoints. The two-dimensional coordinates of the point to be back-projected have already been obtained, including:

[0032] The endpoint initialization unit is used to determine the first endpoint and the second endpoint that are on the same straight line as the point to be back-projected. The two-dimensional coordinates and three-dimensional coordinates of the first endpoint and the second endpoint have been obtained. The point to be back-projected is located between the first endpoint and the second endpoint on the straight line.

[0033] The endpoint update unit is used to calculate the endpoint distance using the three-dimensional coordinates of the first endpoint and the second endpoint. When the endpoint distance is less than the first distance threshold, the three-dimensional coordinates of the first endpoint or the second endpoint are set as the three-dimensional coordinates of the point to be back-projected.

[0034] The segment tripartite solution unit is used to determine the tripartite points of the three-dimensional endpoint line segment formed by the first endpoint and the second endpoint, and compare the distances of the two-dimensional projection points of the tripartite points with the points to be inverted;

[0035] The distance comparison result is fed back to the endpoint update unit, which updates the three points that are farther away from the point to be back-projected as the first endpoint or the second endpoint.

[0036] Furthermore, the segment ternary trisection solution unit includes performing the following steps:

[0037] Determine two trisection points of the three-dimensional endpoint line segment, including the first trisection point closer to the first endpoint and the second trisection point closer to the second endpoint;

[0038] Obtain the first projection point of the first third point onto the two-dimensional image, and the second projection point of the second third point onto the two-dimensional image;

[0039] Calculate the first projection distance between the first projection point and the point to be reverse-projected, and the second projection distance between the second projection point and the point to be reverse-projected;

[0040] The first projection distance and the second projection distance are compared, and the distance comparison result is fed back to the endpoint update unit so that the first third point is updated as the first endpoint when the first projection distance is larger, and the second third point is updated as the second endpoint otherwise.

[0041] Furthermore, the endpoint update unit includes performing the following steps:

[0042] When the distance between the endpoints is less than the first distance threshold, the three-dimensional coordinates of the first endpoint are set to the three-dimensional coordinates of the point to be back-projected.

[0043] Fourthly, the present invention provides a three-dimensional back-projection device for solving the three-dimensional coordinates of a point to be back-projected located within a triangular region formed by three known vertices, wherein the two-dimensional coordinates of the point to be back-projected have already been obtained, including:

[0044] The collinearity point finding unit is used to arbitrarily select one side of a triangular region, with the two endpoints of the side being the first vertex and the second vertex, and to find the collinear points located on that side. The collinear points are collinear with the point to be projected and the third vertex.

[0045] The point data of the collinear point and the third vertex are sent to the device provided by the third party, so that the collinear point and the third vertex are determined as the first endpoint and the second endpoint, and the point to be back-projected is located between the collinear point and the third vertex;

[0046] The three-dimensional coordinates of the point to be projected are obtained using the apparatus provided by the third party.

[0047] Furthermore, the collinearity point determination unit includes:

[0048] The vertex update module is used to calculate the vertex distance using the 3D coordinates of the first vertex and the second vertex. When the vertex distance is less than the second distance threshold, the 3D coordinates of the first vertex or the second vertex are set to the 3D coordinates of the collinear point.

[0049] The point-to-line distance calculation module is used to determine the trisection points of the three-dimensional vertex line segment formed by the first vertex and the second vertex. It calculates the two-dimensional projection line segments of the two trisection points and the third vertex, compares the point-to-line distances of the two two-dimensional projection line segments with the point to be inverted, and feeds the point-to-line distance comparison results back to the vertex update module, so that the trisection point with the larger point-to-line distance is updated as the first vertex or the second vertex.

[0050] Furthermore, the point-to-line distance calculation module includes performing the following steps:

[0051] Determine two trisection points of a 3D vertex segment, including the third trisection point closer to the first vertex and the fourth trisection point closer to the second vertex;

[0052] Obtain the third projection point of the third trisection point projected onto the two-dimensional image, and the fourth projection point of the fourth trisection point projected onto the two-dimensional image.

[0053] Find the first two-dimensional projection line segment between the third projection point and the third vertex, and the second two-dimensional projection line segment between the fourth projection point and the third vertex;

[0054] Compare the first point-line distance and the second point-line distance. The first point-line distance is the distance between the first two-dimensional projection line segment and the point to be inverted, and the second point-line distance is the distance between the second two-dimensional projection line segment and the point to be inverted. Feed the point-line distance comparison result back to the vertex update module, so that if the first point-line distance is larger, the third trisection point is updated as the first vertex, otherwise the fourth trisection point is updated as the second vertex.

[0055] Furthermore, the vertex update module includes performing the following steps:

[0056] When the vertex distance is less than the second distance threshold, let the three-dimensional coordinates of the first vertex be the three-dimensional coordinates of the point to be back-projected.

[0057] Fifthly, an electronic device including a memory and a processor;

[0058] The memory is used to store programs;

[0059] The processor is used to execute the program to implement the three-dimensional back-projection method of the first aspect and / or the second aspect.

[0060] In a sixth aspect, the present invention provides a vehicle including the electronic equipment of the fifth aspect.

[0061] In a seventh aspect, the present invention provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the three-dimensional back-projection method provided in the first aspect and / or the second aspect.

[0062] Compared with existing technologies, the 3D backprojection method provided by this invention utilizes known endpoints or vertices to solve for the 3D coordinates of other 2D points projected into 3D space. This provides a novel backprojection method for scenarios where marking is desired on 2D images and point data needs to be determined in the corresponding 3D space. The method of this invention uses known point data for solving, ensuring the accuracy of the backprojection. It can provide high-precision 3D backprojection solutions in various application scenarios requiring 3D space reconstruction, avoiding errors caused by manual calculation. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0064] Figure 1 A structural block diagram of a three-dimensional back projection device provided in an embodiment of this application;

[0065] Figure 2 An execution flow of the three-dimensional back projection method provided in the embodiments of this application;

[0066] Figure 3 A structural block diagram of another three-dimensional back projection device provided in the embodiments of this application;

[0067] Figure 4 Another execution flow of the three-dimensional back projection method provided in the embodiments of this application;

[0068] Figure 5 A structural block diagram of another three-dimensional back projection device provided in the embodiments of this application;

[0069] Figure 6 This is yet another execution flow of the three-dimensional back projection method provided in the embodiments of this application;

[0070] Figure 7 A structural block diagram of the collinearity point determination unit provided in the embodiments of this application;

[0071] Figure 8 An execution flow for finding collinear points in the three-dimensional back projection method provided in the embodiments of this application;

[0072] Figure 9 This application provides an example of a three-dimensional back projection implementation scenario.

[0073] Figure 10 This application provides an execution logic for a three-dimensional back projection implementation scene.

[0074] Figure 11 Another example of a three-dimensional back projection implementation scenario provided in this application embodiment;

[0075] Figure 12 This application provides another execution logic for a three-dimensional back projection implementation scenario.

[0076] Figure 13 and Figure 14 This provides another example of a three-dimensional back projection implementation scenario for embodiments of this application;

[0077] Figure 15 This is a hardware structure block diagram of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Figure 1 A three-dimensional back-projection device 100 provided in this application embodiment includes an endpoint initialization unit 110, an endpoint update unit 120, and a line segment ternary method solution unit 130, which is used to solve the three-dimensional coordinates of the point to be back-projected that is collinear with the three known endpoints. The two-dimensional coordinates of the point to be back-projected have been obtained.

[0080] The known endpoint point data includes two-dimensional coordinates and three-dimensional coordinates. The endpoint point data can be obtained through an initial two-dimensional projection. The two-dimensional projection can be achieved by using the camera projection principle to project a three-dimensional scene into a two-dimensional image, such as projecting a three-dimensional map onto a two-dimensional image, or by modeling a three-dimensional space to obtain a corresponding two-dimensional image. The known endpoint point data can also be obtained using the back projection method of this invention or other back projection methods.

[0081] This embodiment aims to use the point data of two known endpoints to obtain the three-dimensional coordinates of points on the line segment formed by the two known endpoints, so that the constructed three-dimensional space has more point data and improves the resolution of the three-dimensional space.

[0082] based on Figure 1 The device structure shown, Figure 2 This embodiment illustrates an execution flow of the three-dimensional back projection method provided, which includes:

[0083] Step S11. Select the first endpoint and the second endpoint.

[0084] Specifically, the endpoint initialization unit 110 determines the first endpoint and the second endpoint of the same straight line as the point to be back-projected in the two-dimensional image. The two-dimensional coordinates and three-dimensional coordinates of the first endpoint and the second endpoint have been obtained. The point to be back-projected is located between the first endpoint and the second endpoint on the straight line.

[0085] Geometrically, the first endpoint, the second endpoint, and the point to be projected are collinear. Therefore, the coordinate information of the point to be projected in three-dimensional space can be obtained using the point data of the first and second endpoints, given the known two-dimensional coordinates.

[0086] Step S12. Calculate endpoint distance and update endpoints.

[0087] Specifically, the endpoint update unit 120 calculates the endpoint distance using the three-dimensional coordinates of the first endpoint and the second endpoint. When the endpoint distance is less than the first distance threshold, the three-dimensional coordinates of the first endpoint or the second endpoint are set to the three-dimensional coordinates of the point to be back-projected.

[0088] It is understandable that when the distance between the first endpoint and the second endpoint is small enough, the point to be back-projected can be approximated as one of the two endpoints. Therefore, the three-dimensional coordinates of the first endpoint or the second endpoint can be directly determined as the three-dimensional coordinates of the point to be back-projected.

[0089] Step S13. Calculate the distance between the projection point of the trisection points of the line segment and the point to be inverted, and return to step S12.

[0090] Specifically, the segment trisection solution unit 130 determines the trisection points of the three-dimensional endpoint line segment formed by the first endpoint and the second endpoint, compares the distance between the two-dimensional projection point of the trisection points and the point to be inverted, and sends the distance comparison result to the endpoint update unit 120, so that the trisection point with a larger distance from the point to be inverted is updated as the first endpoint or the second endpoint.

[0091] When the endpoint distance does not meet the above threshold condition, this embodiment Figure 2 One information transmission scenario is illustrated, where the endpoint update unit 120 directly sends the point data of the two endpoints to the segment ternary method solution unit 130. In other examples, the endpoint update unit 120 may send a request to the endpoint selection unit 110, and the endpoint initialization unit 110 may then send the point data of the two endpoints to the segment ternary method solution unit 130. The device structure and information transmission logic of this example scenario are as follows: Figure 3 and Figure 4 As shown.

[0092] In a further embodiment, the steps performed by the segment ternary method solution unit 130 are as follows:

[0093] Determine two trisection points of the three-dimensional endpoint line segment, including the first trisection point closer to the first endpoint and the second trisection point closer to the second endpoint;

[0094] Obtain the first projection point of the first third point onto the two-dimensional image, and the second projection point of the second third point onto the two-dimensional image;

[0095] Calculate the first projection distance between the first projection point and the point to be reverse-projected, and the second projection distance between the second projection point and the point to be reverse-projected;

[0096] The first projection distance and the second projection distance are compared, and the distance comparison result is sent to the endpoint update unit 120. If the first projection distance is larger, the first third point is updated as the first endpoint; otherwise, the second third point is updated as the second endpoint.

[0097] It is understandable that when the first projection distance is larger, it means that the first trisection point is farther away from the point to be projected. The purpose of the trisection method is to use the known points to infinitely approximate the point to be projected, so the line segment needs to be shrunk. Therefore, when the first trisection point is farther away from the point to be projected, it also means that the first endpoint is farther away from the point to be projected. So the first trisection point is updated to the first endpoint to shrink the line segment formed by the first endpoint and the second endpoint.

[0098] Based on the execution logic of the segment ternary search unit 130 described above, in a further embodiment, the endpoint update unit 120 includes performing the following steps:

[0099] When the distance between the endpoints is less than the first distance threshold, the three-dimensional coordinates of the first endpoint are set to the three-dimensional coordinates of the point to be back-projected.

[0100] See Figure 5 , Figure 5 Another three-dimensional back projection device 200 provided in an embodiment of this application is shown, including a collinearity point determination unit 210.

[0101] The device 200 provided in this embodiment is used to solve the three-dimensional coordinates of the point to be back-projected located in the triangular region formed by three known vertices, and the two-dimensional coordinates of the point to be back-projected have been obtained.

[0102] based on Figure 5 and Figure 1 The device structure shown, Figure 6 This embodiment illustrates an execution flow of the three-dimensional back projection method provided, which includes:

[0103] Step S21. Find the collinear points.

[0104] Specifically, the collinear point finding unit 210 arbitrarily selects one side of the triangular region, with the two endpoints of the side being the first vertex and the second vertex, and finds the collinear points located on the side. The collinear points are collinear with the point to be projected and the third vertex.

[0105] In terms of geometric relationships, when collinear points are collinear with the point to be projected and the third vertex, the three-dimensional coordinates of the point to be projected can be solved using the method of solving the points on the line segment formed by the two known endpoints in the previous embodiment.

[0106] Step S22. Determine the first endpoint and the second endpoint.

[0107] Specifically, the collinearity point acquisition unit 210 sends the point data of the collinearity point and the third vertex to the device 100 of the previous embodiment, and the device 100 determines the collinearity point and the third vertex as the first endpoint and the second endpoint.

[0108] Step S23. Calculate endpoint distance and update endpoints.

[0109] Step S24. Calculate the distance between the projection point of the trisection points of the line segment and the point to be inverted, and return to step S23.

[0110] The steps S22-S24 described above correspond one-to-one with steps S11-S13 in the previous embodiment. Please refer to the foregoing description for details, which will not be repeated here.

[0111] Compared to the previous embodiment, in this embodiment, the point to be back-projected is not on the line segment formed by the known endpoints, but is located in the triangular region formed by the three known vertices. In this case, it is necessary to use the point data of two vertices of the triangle to find a collinear point, so that the point to be back-projected is located on the line segment formed by the collinear point and a vertex of the triangular region. Then, the three-dimensional coordinates of the point to be back-projected are solved using the method of the previous embodiment.

[0112] In a further embodiment, collinear points can be determined using point-to-line distance. In an example, the collinear point determination unit 210 includes a vertex update module 211 and a point-to-line distance calculation module 212, such as... Figure 7 As shown. Figure 8 As shown, step S21 above includes the following steps:

[0113] Step S211. Calculate vertex distance and update vertices.

[0114] Specifically, the vertex update module 211 calculates the vertex distance using the three-dimensional coordinates of the first vertex and the second vertex. When the vertex distance is less than the second distance threshold, the three-dimensional coordinates of the first vertex or the second vertex are set to the three-dimensional coordinates of the collinear point.

[0115] It is understandable that when the distance between the first vertex and the second vertex is small enough, the collinear point can be approximated as one of the two endpoints. Therefore, the three-dimensional coordinates of the first vertex or the second vertex can be directly determined as the three-dimensional coordinates of the collinear point.

[0116] Step S212. Calculate the distance between the point and the line, then return to step S211.

[0117] Specifically, the point-to-line distance calculation module 212 determines the trisection points of the three-dimensional vertex line segment formed by the first vertex and the second vertex, calculates the two-dimensional projection line segments of the two trisection points and the third vertex respectively, compares the point-to-line distances of the two two-dimensional projection line segments with the point to be back-projected, and feeds back the point-to-line distance comparison results to the vertex update unit 211, so that it updates the trisection point corresponding to the larger point-to-line distance as the first vertex or the second vertex.

[0118] In a further embodiment, the point-to-line distance calculation module 212 performs the following steps:

[0119] Determine two trisection points of a 3D vertex segment, including the third trisection point closer to the first vertex and the fourth trisection point closer to the second vertex;

[0120] Obtain the third projection point of the third trisection point projected onto the two-dimensional image, and the fourth projection point of the fourth trisection point projected onto the two-dimensional image.

[0121] Find the first two-dimensional projection line segment between the third projection point and the third vertex, and the second two-dimensional projection line segment between the fourth projection point and the third vertex;

[0122] The distance between the first point and the line is compared with the distance between the first point and the line. The first point and the line is the distance between the first two-dimensional projection line segment and the point to be reverse-projected. The second point and the line is the distance between the second two-dimensional projection line segment and the point to be reverse-projected. The result of the point and line distance comparison is fed back to the vertex update module 211, so that if the first point and the line is larger, the third third point is updated as the first vertex, otherwise the fourth third point is updated as the second vertex.

[0123] It is understandable that when the distance between the first point and the line is larger, it means that the first two-dimensional projection line segment is farther away from the point to be projected. The purpose of calculating the distance between the point and the line is to make the line segment formed by the third point and the third vertex infinitely close to the point to be projected. If the distance is small enough, it can be considered that the point to be projected is located on the line segment. Therefore, when the first two-dimensional projection line segment is farther away from the point to be projected, it means that the third third point is farther away from the point to be projected. The third third point is then updated as the first endpoint.

[0124] Based on the execution logic of the point-to-line distance calculation module 212 described above, in a further embodiment, the vertex update module 211 includes performing the following steps:

[0125] When the vertex distance is less than the second distance threshold, let the three-dimensional coordinates of the first vertex be the three-dimensional coordinates of the point to be back-projected.

[0126] Next, we will introduce the three-dimensional back projection method of this application through two practical application scenarios.

[0127] See Figure 9 and Figure 10 This example illustrates one implementation scenario of the three-dimensional back-projection method of this application, where the point to be back-projected is located on a line segment formed by two known endpoints.

[0128] Points A1 and B1 are two known points on a two-dimensional image, and their coordinates in both the two-dimensional image and the three-dimensional map are known. Point N1 is the point to be back-projected in this embodiment. Points A1, B1, and N1 are collinear, and point N1 lies on the line segment between points A1 and B1. The two-dimensional coordinates of point N1 in the two-dimensional image are known. Now, we need to find the three-dimensional coordinates of point N1 in the three-dimensional map.

[0129] like Figure 10 As shown, the process for back-projection point N1 includes the following steps:

[0130] Step S31. Let point A1 be the left endpoint L1 and point B1 be the right endpoint R1;

[0131] Step S32. Calculate the distance L1 between point L1 and point R1.

[0132] Step S33. Determine whether the distance L1 meets the preset conditions. If it does, determine the three-dimensional coordinates of point L1 as the three-dimensional coordinates of point N1. If it does not meet the conditions, proceed to step S34.

[0133] The preset condition is that the distance L1 is less than the first distance threshold L. p1 .

[0134] Step S34. Determine the third point Lmid1 on line segment L1R1 that is closer to point L1, and the third point Rmid1 that is closer to point R1.

[0135] Step S35. Calculate the distance disL1 between the projection point of point Lmid1 on the two-dimensional image and point N1, and the distance disR1 between the projection point of point Rmid1 on the two-dimensional image and point N1.

[0136] Step S36. Compare the sizes of disL1 and disR1. If disL1 > disR1, update point Lmid1 to point L1; otherwise, update point Rmid1 to point R1. Return to step S32.

[0137] Obviously, the solution of this invention can use two known endpoints to solve the three-dimensional coordinates of a point on the endpoint line segment projected back onto the three-dimensional space, so that the three-dimensional space has more point data and improves the resolution and spatial reconstruction accuracy of the three-dimensional space.

[0138] See Figure 11 and Figure 12 This example illustrates another implementation scenario of the three-dimensional back-projection method of this application, where the point to be back-projected is located within a triangular region formed by three known vertices.

[0139] Points A2, C, and B2 are three known points on a two-dimensional image. Their coordinates in both the two-dimensional image and the three-dimensional map are known. The three points form a triangular region. Point N2 is the point to be back-projected in this embodiment and is not on any side of the triangle. The two-dimensional coordinates of point N2 in the two-dimensional image are known. Now, we need to find the three-dimensional coordinates of point N2 in the three-dimensional map.

[0140] like Figure 12 As shown, the process for back-projection point N2 includes the following steps:

[0141] Step S401. Arbitrarily select one side of the triangular region. In this embodiment, select the side where point A2 and point C are located, and let point A2 be the first vertex L2 and point C be the second vertex R2.

[0142] Step S402. Calculate the distance L2 between point L2 and point R2.

[0143] Step S403. Determine whether the distance L2 meets the preset conditions. If it does, determine that the three-dimensional coordinates of point L2 are the three-dimensional coordinates of collinear point M, and proceed to step S407; otherwise, proceed to step S404.

[0144] The preset condition is that the distance L2 is less than the first distance threshold L. p2 .

[0145] Step S404. Take the third point Lmid2 closest to point L2 and the third point Rmid2 closest to point R2 on line segment L2R2. Determine the projection point of point Lmid2 onto the two-dimensional image as point O and the projection point of point Rmid2 onto the two-dimensional image as point P.

[0146] Step S405. Calculate the distance disL2 between line segment OB2 and point N2, and the distance disR2 between line segment PB2 and point N2.

[0147] Step S406. Compare the sizes of disL2 and disR2. If disL2 > disR2, update point Lmid2 to point L2; otherwise, update point Rmid2 to point R2. Return to step S402.

[0148] Step S407. Calculate the distance L3 between point M and point B2.

[0149] Step S408. Determine whether the distance L3 meets the preset conditions. If it does, determine the three-dimensional coordinates of point M as the three-dimensional coordinates of point N2. If it does not meet the conditions, proceed to step S409.

[0150] The preset condition is that the distance L3 is less than the third distance threshold L. p3 .

[0151] Step S409. Determine the third point Lmid3 on line segment MB2 that is closer to point M and the third point Rmid3 that is closer to point B2.

[0152] Step S410. Calculate the distance disL3 between the projection point of point Lmid3 on the two-dimensional image and point N2, and the distance disR3 between the projection point of point Rmid3 on the two-dimensional image and point N2.

[0153] Step S411. Compare the sizes of disL3 and disR3. If disL3 > disR3, update point Lmid3 to point M; otherwise, update point Rmid3 to point B2. Return to step S407.

[0154] Obviously, the solution of this invention can use three known vertices to solve the three-dimensional coordinates of a point in the triangular region formed by the vertices and project it into the three-dimensional space, so that the three-dimensional space has more point data and improves the resolution and spatial reconstruction accuracy of the three-dimensional space.

[0155] See Figure 13 and Figure 14 This illustrates yet another implementation scenario of the three-dimensional back projection method of the present invention.

[0156] In this embodiment, the three-dimensional back-projection scheme of the present invention can be applied to vehicles. Typically, the vehicle can be equipped with an intelligent real-scene navigation system and / or a driving assistance system, which includes an onboard camera and map software. The onboard camera can capture the environment around the vehicle. For example, when parking, the driving assistance system can provide real-scene assisted parking services, displaying real environmental information on the vehicle's display screen. After the system acquires relevant environmental information through the onboard camera, it first projects the three-dimensional environmental information into a two-dimensional image for processing. After processing, the back-projection scheme of this application is used to back-project the points of the image onto the real-scene map to help the driver park more safely and efficiently. Alternatively, during driving, the onboard camera acquires road condition information in front of the vehicle, and the back-projection scheme of this application is used to reflect the road condition information on the real-scene navigation map.

[0157] Figure 13 This illustrates the scenario where a point is clicked on a line segment in a 2D image, and that point can then be back-projected onto... Figure 14 As shown in the map.

[0158] The three-dimensional back projection device provided in the above embodiments of this application can be applied to electronic devices. The electronic device may include only device 100 or device 200, or it may include both device 100 and device 200. Figure 15 A hardware block diagram of the electronic device is shown, with reference to... Figure 15 The hardware structure of an electronic device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0159] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0160] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0161] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0162] The memory stores a program, which the processor can call. The program is used to implement the various processing steps of the three-dimensional back projection method in the foregoing embodiments.

[0163] This application also provides a computer-readable storage medium that stores a program suitable for execution by a processor, the program being used to implement the various processing flows of the three-dimensional back-projection method in the foregoing embodiments.

[0164] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0165] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional back projection method, characterized in that, This is used to solve for the 3D coordinates of the point to be back-projected, which is collinear with the two known endpoints. The 2D coordinates of the point to be back-projected have already been obtained, including: Endpoint initialization steps: Determine the first endpoint and the second endpoint that are on the same straight line as the point to be back-projected. The two-dimensional coordinates and three-dimensional coordinates of the first endpoint and the second endpoint have been obtained. The point to be back-projected is located between the first endpoint and the second endpoint on the straight line. Endpoint update step: Calculate the endpoint distance using the three-dimensional coordinates of the first endpoint and the second endpoint. If the endpoint distance is less than a first distance threshold, determine the three-dimensional coordinates of the first endpoint or the second endpoint as the three-dimensional coordinates of the point to be back-projected; otherwise, proceed to the segment ternary method solution step. The steps for solving the segment trisection method are as follows: Determine the trisection points of the three-dimensional endpoint line segment formed by the first endpoint and the second endpoint; compare the distances of the two-dimensional projection points of the trisection points with the point to be inverted; update the trisection point with a larger distance from the point to be inverted as the first endpoint or the second endpoint; and return to the endpoint update step. The steps for solving the line segment trisection method include: Determine two trisection points of the three-dimensional endpoint line segment, including a first trisection point closer to the first endpoint and a second trisection point closer to the second endpoint; Obtain the first projection point of the first three points onto the two-dimensional image, and the second projection point of the second three points onto the two-dimensional image; Calculate the first projection distance between the first projection point and the point to be projected, and the second projection distance between the second projection point and the point to be projected; Compare the first projection distance and the second projection distance. If the first projection distance is larger, update the first third point as the first endpoint; otherwise, update the second third point as the second endpoint. Return to the endpoint update step; wherein, the point data of the endpoint includes two-dimensional coordinates and three-dimensional coordinates. The two-dimensional coordinates are obtained through two-dimensional projection, which refers to projecting a three-dimensional scene into a two-dimensional image using the principle of camera projection.

2. The three-dimensional back projection method according to claim 1, characterized in that, The endpoint update step includes: When the distance between the endpoints is less than the first distance threshold, the three-dimensional coordinates of the first endpoint are determined as the three-dimensional coordinates of the point to be back-projected.

3. A three-dimensional back projection method, characterized in that, This is used to solve for the 3D coordinates of the point to be back-projected, located within a triangular region formed by three known vertices. The 2D coordinates of the point to be back-projected have already been obtained, including: Choose any side of the triangular region, with the two endpoints of the side being the first vertex and the second vertex. Find the collinear point located on the side, where the collinear point is collinear with the third vertex and the point to be projected. The collinear point and the third vertex are defined as the first endpoint and the second endpoint, and the point to be projected is located on the line segment between the collinear point and the third vertex. The three-dimensional coordinates of the point to be back-projected are obtained using the three-dimensional back-projection method according to any one of claims 1 to 2.

4. The three-dimensional back projection method according to claim 3, characterized in that, The determination of the collinear points includes: Vertex update step: Calculate the vertex distance using the three-dimensional coordinates of the first vertex and the second vertex. If the vertex distance is less than the second distance threshold, determine that the three-dimensional coordinates of the first vertex or the second vertex are the three-dimensional coordinates of a collinear point; otherwise, proceed to the point-line distance calculation step. Point-to-line distance calculation steps: Determine the trisection points of the three-dimensional vertex line segment formed by the first vertex and the second vertex, respectively calculate the two-dimensional projection line segments of the two trisection points and the third vertex, compare the point-to-line distances of the two two-dimensional projection line segments and the point to be inverted, update the trisection point corresponding to the larger point-to-line distance to the first vertex or the second vertex, and return to the vertex update step.

5. The three-dimensional back projection method according to claim 4, characterized in that, The point-to-line distance calculation steps include: Determine two trisection points of the three-dimensional vertex segment, including a third trisection point closer to the first vertex and a fourth trisection point closer to the second vertex; Obtain the third projection point of the third third point onto the two-dimensional image, and the fourth projection point of the fourth third point onto the two-dimensional image; Find the first two-dimensional projection line segment between the third projection point and the third vertex, and the second two-dimensional projection line segment between the fourth projection point and the third vertex; Compare the distance between the first point and the second point. The first point is the distance between the first two-dimensional projection line segment and the point to be projected. The second point is the distance between the second two-dimensional projection line segment and the point to be projected. If the first point is greater, update the third third point to the first vertex; otherwise, update the fourth third point to the second vertex. Return to the vertex update step.

6. The three-dimensional back projection method according to claim 5, characterized in that, The vertex update steps include: When the vertex distance is less than the second distance threshold, the three-dimensional coordinates of the first vertex are determined as the three-dimensional coordinates of the point to be back-projected.

7. A three-dimensional back projection device, characterized in that, This is used to solve for the 3D coordinates of the point to be back-projected, which is collinear with the two known endpoints. The 2D coordinates of the point to be back-projected have already been obtained, including: An endpoint initialization unit is used to determine the first endpoint and the second endpoint that are on the same straight line as the point to be back-projected. The two-dimensional coordinates and three-dimensional coordinates of the first endpoint and the second endpoint have been obtained. The point to be back-projected is located between the first endpoint and the second endpoint on the straight line. The endpoint update unit calculates the endpoint distance using the three-dimensional coordinates of the first endpoint and the second endpoint. When the endpoint distance is less than a first distance threshold, the three-dimensional coordinates of the first endpoint or the second endpoint are set to the three-dimensional coordinates of the point to be back-projected. The segment ternary method solution unit is used to determine the ternary points of the three-dimensional endpoint line segment formed by the first endpoint and the second endpoint, and compare the distance between the two-dimensional projection point of the ternary point and the point to be inverted; The distance comparison result is fed back to the endpoint update unit, which updates the three points that are farther away from the point to be back-projected as the first endpoint or the second endpoint. The segment tripping solution unit includes the following steps: Determine two trisection points of the three-dimensional endpoint line segment, including a first trisection point closer to the first endpoint and a second trisection point closer to the second endpoint; Obtain the first projection point of the first three points onto the two-dimensional image, and the second projection point of the second three points onto the two-dimensional image; Calculate the first projection distance between the first projection point and the point to be projected, and the second projection distance between the second projection point and the point to be projected. The first projection distance and the second projection distance are compared, and the distance comparison result is fed back to the endpoint update unit so that when the first projection distance is larger, the first third point is updated as the first endpoint, otherwise the second third point is updated as the second endpoint; wherein, the point data of the endpoint includes two-dimensional coordinates and three-dimensional coordinates. The two-dimensional coordinates are obtained through two-dimensional projection, which refers to projecting a three-dimensional scene into a two-dimensional image using the camera projection principle.

8. The three-dimensional back projection device according to claim 7, characterized in that, The endpoint update unit includes performing the following steps: When the distance between the endpoints is less than a first distance threshold, the three-dimensional coordinates of the first endpoint are determined as the three-dimensional coordinates of the point to be back-projected.

9. A three-dimensional back projection device, characterized in that, This is used to solve for the 3D coordinates of the point to be back-projected, located within a triangular region formed by three known vertices. The 2D coordinates of the point to be back-projected have already been obtained, including: The collinearity point finding unit is used to arbitrarily select one side of a triangular region, with the two endpoints of the side being the first vertex and the second vertex, and to find the collinear points located on the side. The collinear points are collinear with the point to be projected and the third vertex. The point data of the collinear point and the third vertex are sent to the three-dimensional back projection device according to any one of claims 7 to 8, so that the collinear point and the third vertex are determined as the first endpoint and the second endpoint, and the point to be back projected is located between the collinear point and the third vertex; The three-dimensional coordinates of the point to be back-projected are obtained using the three-dimensional back-projection device according to any one of claims 7 to 8.

10. The three-dimensional back projection device according to claim 9, characterized in that, The collinearity point determination unit includes: The vertex update module is used to calculate the vertex distance using the three-dimensional coordinates of the first vertex and the second vertex, and when the vertex distance is less than a second distance threshold, the three-dimensional coordinates of the first vertex or the second vertex are set to the three-dimensional coordinates of a collinear point. The point-to-line distance calculation module is used to determine the trisection points of the three-dimensional vertex line segment formed by the first vertex and the second vertex, calculate the two-dimensional projection line segments of the two trisection points and the third vertex respectively, compare the point-to-line distances of the two two-dimensional projection line segments with the point to be back-projected, and feed back the point-to-line distance comparison results to the vertex update module, so that the trisection point corresponding to the larger point-to-line distance is updated as the first vertex or the second vertex.

11. The three-dimensional back projection device according to claim 10, characterized in that, The point-to-line distance calculation module includes the following steps: Determine two trisection points of the three-dimensional vertex segment, including a third trisection point closer to the first vertex and a fourth trisection point closer to the second vertex; Obtain the third projection point of the third third point onto the two-dimensional image, and the fourth projection point of the fourth third point onto the two-dimensional image; Find the first two-dimensional projection line segment between the third projection point and the third vertex, and the second two-dimensional projection line segment between the fourth projection point and the third vertex; The first point-line distance and the second point-line distance are compared. The first point-line distance is the distance between the first two-dimensional projection line segment and the point to be reverse-projected, and the second point-line distance is the distance between the second two-dimensional projection line segment and the point to be reverse-projected. The point-line distance comparison result is fed back to the vertex update module, so that if the first point-line distance is larger, the third trisection point is updated to the first vertex; otherwise, the fourth trisection point is updated to the second vertex.

12. The three-dimensional back projection device according to claim 11, characterized in that, The vertex update module includes performing the following steps: When the vertex distance is less than the second distance threshold, the three-dimensional coordinates of the first vertex are determined as the three-dimensional coordinates of the point to be back-projected.

13. An electronic device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the three-dimensional back projection method according to any one of claims 1 to 6.

14. A vehicle, characterized in that, Including the electronic device as described in claim 13.

15. A computer-readable medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the three-dimensional back projection method according to any one of claims 1 to 6.

Images